Road floating object sensing and processing method and device, electronic equipment, medium and vehicle
By sensing and identifying road floating objects, image and radar data are used to identify floating objects states and adsorption levels, and protection control instructions are generated, which solves the problem of vehicle floating objects blocking the line of sight and improves driving safety.
Patent Information
- Application Number
- CN202410900508.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-01
AI Technical Summary
Vehicles are easily blocked by road floating objects during driving, resulting in driving safety hazards. It is difficult for the existing technology to effectively identify and deal with road floating objects, increasing the risk of traffic accidents.
A method of perception and processing of floating objects on the road is adopted. By obtaining floating objects sample images and radar data, the perception model and recognition model are trained, the environmental data of the vehicle's driving road surface is dynamically collected, the state and adsorption level of floating objects are identified, and the vehicle protection plan is used. Generate protection control instructions and automatically carry out floating objects protection.
Automatic perception and identification of floating objects on the road is realized, the adsorption level is determined, and corresponding protection is carried out, which improves driving safety, avoids floating objects being absorbed on the vehicle, and enhances the clarity of driving line of sight.
Smart Images

Figure CN118887635A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle control technology, and in particular to a method, device, electronic equipment, medium and vehicle for sensing and processing road floating objects. Background Art
[0002] When a vehicle is driving on the road, its vision is easily blocked by floating objects on the road. For example, when a vehicle is driving on the road and a truck in front of it drops dusty materials such as lime, when vehicles pass by quickly on the road, they will stir up dust. Some dust seriously affects the vision. The burning of crops by farmers on the side of the road can also cause a lot of thick smoke to block the vision. Some forest fires (such as the Chongqing forest fire) can also cause a lot of thick smoke to make the road invisible.
[0003] When floating objects on the road come into contact with water and are adsorbed onto vehicles, they will seriously affect the driver's vision, increase driving safety risks, and easily lead to traffic accidents. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a method, device, electronic device, medium and vehicle for sensing and processing road floating objects, which can automatically sense and identify road floating objects and perform vehicle protection processing to improve driving safety.
[0005] An aspect of an embodiment of the present application provides a method for sensing and processing floating objects on a road, the method comprising the following steps:
[0006] Obtaining a floating object sample image dataset and a corresponding floating object sample radar dataset;
[0007] Acquire a floating object perception model, train the floating object perception model using the floating object sample image data set, and obtain a model training output result;
[0008] Training a floating object recognition model using the model training output result and the floating object sample radar data set;
[0009] Dynamically collect environmental image data and radar data of the road surface on which the vehicle is traveling;
[0010] Inputting the environmental image data and the radar data into a trained floating object perception model, performing floating object perception and recognition using the trained floating object perception model, and outputting corresponding floating object status information; the floating object status information at least includes a floating object category and a floating object area;
[0011] Inputting the floating object state information and the radar data into a trained floating object recognition model, using the trained floating object recognition model to identify the nature of the floating objects, and outputting a corresponding target floating object adsorption level;
[0012] A vehicle protection plan is obtained, a target vehicle protection strategy corresponding to the target floating object adsorption level is determined from the vehicle protection plan, a vehicle protection control instruction is generated according to the target vehicle protection strategy, and the vehicle is controlled to perform floating object protection processing according to the vehicle protection control instruction.
[0013] In some embodiments, the obtaining of the floating object sample image dataset and the corresponding floating object sample radar dataset specifically includes:
[0014] Collecting a plurality of floating object sample images with floating object status labels and sample radar data with floating object water adsorption level labels corresponding to each of the floating object sample images; the floating object status labels at least include the floating object type and the floating object position;
[0015] All of the floating object sample images are combined to obtain the floating object sample image data set, and all of the sample radar data are combined to obtain the floating object sample radar data set.
[0016] In some embodiments, the acquiring of the floating object perception model, training the floating object perception model using the floating object sample image data set, and obtaining the model training output result specifically includes:
[0017] Using each of the floating object sample images to train the floating object perception model, and outputting floating object sample state information corresponding to each of the floating object sample images;
[0018] According to the floating object state label corresponding to each of the floating object sample images and the floating object sample state information corresponding to each of the floating object sample images, a model evaluation is performed on the floating object perception model to determine a first model evaluation result, and according to the first model evaluation result, it is determined whether to continue training the floating object perception model.
[0019] In some embodiments, the using the model training output result and the floating object sample radar data set to train the floating object recognition model specifically includes:
[0020] Acquire the floating object sample state information and the sample radar data corresponding to each of the floating object sample images;
[0021] Using the floating object sample state information and the sample radar data corresponding to each of the floating object sample images, the floating object recognition model is trained to output the floating object sample adsorption level corresponding to each of the sample radar data;
[0022] According to the floating object sample adsorption level and the floating object water adsorption level label corresponding to each sample radar data, the floating object recognition model is evaluated to determine a second model evaluation result, and according to the second model evaluation result, it is determined whether to continue training the floating object recognition model.
[0023] In some embodiments, the vehicle protection scheme includes a plurality of vehicle protection strategies and floating object absorption levels corresponding to the vehicle protection strategies. The obtaining of the vehicle protection scheme includes determining a target vehicle protection strategy corresponding to the target floating object absorption level from the vehicle protection scheme, generating a vehicle protection control instruction according to the target vehicle protection strategy, and controlling the vehicle to perform floating object protection processing according to the vehicle protection control instruction, specifically including:
[0024] Determining the target vehicle protection strategy corresponding to the target floating object adsorption level from the plurality of vehicle protection strategies according to the floating object adsorption level threshold interval corresponding to each of the vehicle protection strategies, wherein the floating object adsorption level corresponding to the target vehicle protection strategy matches the target floating object adsorption level;
[0025] The vehicle protection control instruction is generated according to the target vehicle protection strategy, and the vehicle is controlled to perform floating object protection processing according to the vehicle protection control instruction.
[0026] In some embodiments, the performing model evaluation on the floating object perception model according to the floating object status label corresponding to each of the floating object sample images and the floating object sample status information corresponding to each of the floating object sample images, determining a first model evaluation result, and determining whether to continue training the floating object perception model according to the first model evaluation result specifically includes:
[0027] Get multiple model evaluation metrics;
[0028] Determining a model evaluation value corresponding to each of the model evaluation indicators according to the floating object state label corresponding to each of the floating object sample images and the floating object sample state information corresponding to each of the floating object sample images;
[0029] When the model evaluation value corresponding to each of the model evaluation indicators exceeds the model evaluation indicator threshold given by each of the model evaluation indicators, the first model evaluation result is determined to be model evaluation passed and the training of the floating object perception model is stopped; otherwise, the training of the floating object perception model is continued.
[0030] Another aspect of the present application provides a road floating object sensing and processing device, the device comprising:
[0031] The first module is used to obtain a floating object sample image dataset and a corresponding floating object sample radar dataset;
[0032] The second module is used to obtain a floating object perception model, train the floating object perception model using the floating object sample image data set, and obtain a model training output result;
[0033] A third module is used to train a floating object recognition model using the model training output result and the floating object sample radar data set;
[0034] The fourth module is used to dynamically collect environmental image data and radar data of the road surface on which the vehicle is traveling;
[0035] A fifth module is used to input the environmental image data and the radar data into a trained floating object perception model, use the trained floating object perception model to perform floating object perception and recognition, and output corresponding floating object status information; the floating object status information at least includes a floating object category and a floating object area;
[0036] A sixth module is used to input the floating object state information and the radar data into the trained floating object recognition model, use the trained floating object recognition model to identify the nature of the floating objects, and output the corresponding target floating object adsorption level;
[0037] The seventh module is used to obtain a vehicle protection plan, determine a target vehicle protection strategy corresponding to the target floating object adsorption level from the vehicle protection plan, generate a vehicle protection control instruction according to the target vehicle protection strategy, and control the vehicle to perform floating object protection processing according to the vehicle protection control instruction.
[0038] Another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned road floating object perception and processing method when executing the computer program.
[0039] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for sensing and processing floating objects on the road is implemented.
[0040] Another aspect of an embodiment of the present application provides a vehicle, wherein the vehicle includes the road floating object sensing and processing device or the electronic device as described above.
[0041] The embodiments of the present application include at least the following beneficial effects: the present application provides a method, device, electronic device, medium and vehicle for sensing and processing road floating objects, which dynamically collects environmental image data and radar data of the road surface on which the vehicle is traveling, uses a floating object sensing model to sense and identify floating objects according to the environmental image data, outputs corresponding floating object state information, uses a floating object recognition model to identify the nature of floating objects according to radar data and floating object state information, outputs the corresponding target floating object adsorption level, determines the target vehicle protection strategy corresponding to the target floating object adsorption level from the vehicle protection plan, generates a vehicle protection control instruction, and controls the vehicle to perform floating object protection processing according to the vehicle protection control instruction. The present application can automatically sense and identify road floating objects, determine the target floating object adsorption level of road floating objects, and perform vehicle protection processing according to the target floating object adsorption level, so that road floating objects will not be adsorbed on the vehicle, thereby improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 is a flow chart of a method for sensing and processing floating objects on a road provided by an embodiment of the present application;
[0043] Figure 2 It is a structural schematic diagram of a road floating object sensing and processing device provided in an embodiment of the present application;
[0044] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0046] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0048] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0049] Reference Figure 1 , Figure 1 This is an optional flow chart of a method for sensing and processing road floating objects provided in an embodiment of the present application. The method may include but is not limited to steps S101 to S107:
[0050] Step S101, obtaining a floating object sample image data set and a corresponding floating object sample radar data set;
[0051] Step S102, obtaining a floating object perception model, training the floating object perception model using a floating object sample image data set, and obtaining a model training output result;
[0052] Step S103, training a floating object recognition model using the model training output result and the floating object sample radar data set;
[0053] Step S104, dynamically collecting environmental image data and radar data of the road surface on which the vehicle is traveling;
[0054] Step S105, inputting the environmental image data and the radar data into the trained floating object perception model, using the trained floating object perception model to perform floating object perception and recognition, and outputting corresponding floating object status information, wherein the floating object status information at least includes the floating object category and the floating object area;
[0055] Step S106, inputting the floating object state information and radar data into the trained floating object recognition model, using the trained floating object recognition model to identify the nature of the floating objects, and outputting the corresponding target floating object adsorption level;
[0056] Step S107, obtaining a vehicle protection plan, determining a target vehicle protection strategy corresponding to a target floating object adsorption level from the vehicle protection plan, generating a vehicle protection control instruction according to the target vehicle protection strategy, and controlling the vehicle to perform floating object protection processing according to the vehicle protection control instruction.
[0057] In some embodiments, step S101 may include but is not limited to steps S201 to S202:
[0058] Step S201, collecting a plurality of floating object sample images with floating object status labels and sample radar data with floating object water adsorption level labels corresponding to each floating object sample image, wherein the floating object status label at least includes the floating object type and the floating object position;
[0059] Step S202: All floating object sample images are combined to obtain a floating object sample image data set, and all sample radar data are combined to obtain a floating object sample radar data set.
[0060] In some embodiments, the types of floating objects may include but are not limited to fog, rain, automobile exhaust smoke, roadside straw, sandstorms, lime, snow, mountain fire ash, fire smoke, yellow mud dust and sand and stone dust, etc. The radar data may include echo, intensity, angle, reflectivity, waveform, polarization and 3D size data, etc. Different types of floating objects correspond to different radar data.
[0061] In some embodiments, step S102 may include but is not limited to steps S301 to S302:
[0062] Step S301, using each floating object sample image to train a floating object perception model, and outputting floating object sample state information corresponding to each floating object sample image;
[0063] Step S302, performing model evaluation on the floating object perception model according to the floating object state label corresponding to each floating object sample image and the floating object sample state information corresponding to each floating object sample image, determining a first model evaluation result, and determining whether to continue training the floating object perception model according to the first model evaluation result.
[0064] In some embodiments, a time series image model and an image detection model are introduced, and a floating object perception model is constructed by combining the image detection model and the time series image model. Optionally, the image detection model can be a target detection model or an image segmentation model or other model constructed by an image neural network. The time series image model aims to extract relevant features of floating objects from an image sequence, and then identify the type of floating objects in the image.
[0065] There are many available time series algorithms corresponding to the time series image model. The following are some common algorithm examples for identifying the movement of floating objects in the air and determining the type of floating objects based on the movement of floating objects in the air.
[0066] Some relevant examples of timing algorithms are as follows:
[0067] GB+Optical Flow (two-stream, TSN): combines optical flow and convolutional neural network (CNN), where optical flow is used to capture motion information in the video, and CNN is used to extract spatial features. The two-stream model divides the video into two streams, spatial and temporal, for processing to capture the spatiotemporal information in the video;
[0068] i3D, 3D, (2+1D) convolution: 3D convolution kernels are used to extract features directly in the spatiotemporal dimension of the video, so that spatial and temporal information can be processed simultaneously. i3D is the abbreviation of Inflated 3D ConvNets, which processes video data by inflating the pre-trained 2D model into a 3D model;
[0069] RNN (LSTM / GPU): Recurrent neural networks (RNN) and their variants, long short-term memory networks (LSTM), are used to process temporal dependencies in videos. This method can capture long-term dependencies in video sequences and is particularly useful for action recognition tasks.
[0070] Self-supervised learning methods: Self-supervised learning has also made significant progress in the field of video action recognition. For example, the Video MAE (Masked Autoencoders As Spatiotemporal Learners) method masks video frames through a masking strategy, and then uses the encoder-decoder structure for feature learning and image restoration. This method can significantly improve the performance of the model without the need for a large amount of labeled data for training.
[0071] In step S302 of some embodiments, the model evaluation indicators may include but are not limited to accuracy (ACC), intersection over union (IoU), precision, recall, etc.
[0072] Specifically, according to the floating object state label corresponding to each floating object sample image and the floating object sample state information corresponding to each floating object sample image, the model evaluation value corresponding to each model evaluation indicator is determined; when the model evaluation value corresponding to each model evaluation indicator exceeds the model evaluation indicator threshold given by each model evaluation indicator, the first model evaluation result is determined to be the model evaluation passed and the training of the floating object perception model is stopped; otherwise, the training of the floating object perception model continues.
[0073] In some embodiments, step S103 may include but is not limited to steps S401 to S403:
[0074] Step S401, obtaining floating object sample state information and sample radar data corresponding to each floating object sample image;
[0075] Step S402, using the floating object sample state information and sample radar data corresponding to each floating object sample image, training a floating object recognition model, and outputting the floating object sample adsorption level corresponding to each sample radar data;
[0076] Step S403, based on the floating object sample adsorption level and the floating object water adsorption level label corresponding to each sample radar data, the floating object recognition model is evaluated to determine a second model evaluation result, and based on the second model evaluation result, it is determined whether to continue training the floating object recognition model.
[0077] In step S403 of some embodiments, the model evaluation process of the floating object recognition model is substantially consistent with the model evaluation process of the floating object perception model described above.
[0078] Optionally, the model evaluation value corresponding to each model evaluation indicator is determined according to the floating object sample adsorption level and the floating object water adsorption level label corresponding to each sample radar data; when the model evaluation value corresponding to each model evaluation indicator exceeds the model evaluation indicator threshold given by each model evaluation indicator, the second model evaluation result is determined to be the model evaluation passed and the training of the floating object recognition model is stopped; otherwise, the training of the floating object recognition model continues.
[0079] In step S106 of some embodiments, optionally, in combination with the vehicle's current state information, including but not limited to the vehicle's current driving path, the vehicle's current position and the vehicle's current speed, the vehicle's current state information, the floating object state information and the radar data are input into a trained floating object recognition model, and the trained floating object recognition model is used to identify the nature of the floating objects to determine whether the vehicle will enter the floating object area. If it is determined that the vehicle will not enter the floating object area, the target floating object adsorption level is set to the lowest level, and no vehicle protection processing is required at this time. If it is determined that the vehicle enters the floating object area, the floating object recognition model is used to determine the distribution of the floating objects per unit area of the air, and the air density of the floating objects is determined based on the distribution of the floating objects per unit area. Then, the corresponding target floating object adsorption level is determined based on the air density of the floating objects, the floating object type and a preset conditional function.
[0080] In some embodiments, according to the type of float and the air density of the float, the water absorption of different types of floats at different air densities of floats is determined, and according to the water absorption of different types of floats at different air densities of floats, different float adsorption levels are preset to obtain a float adsorption level reference table, and according to the float adsorption level reference table, a corresponding condition function is set. Exemplarily, the condition function is f(p,x), which is specifically shown as follows:
[0081]
[0082] Among them, f(p,x) is the conditional function, p is the air density of the floating object, x is the type of floating object, Y1, Y2, ..., Y N is the target floating object adsorption level, p1, p2, ..., p N-1 、p N is the floating object control air density preset in the conditional function f(p,x), x Q It is the default float control type in the conditional function f(p,x).
[0083] In some embodiments, referring to Table 1, Table 1 is an example of a floating object adsorption level reference table, and Table 1 is as follows:
[0084] Table 1 Reference table of floating matter adsorption levels
[0085] Type of floating objects Air density of floating objects Floating matter adsorption level fog Greater than 0 0 rainwater Greater than 0 0 A small amount of yellow mud dust <![CDATA[Greater than p 1x Less than p 2x > 2 A large amount of yellow mud dust <![CDATA[greater than p 2x > 9 Sandstorm <![CDATA[Greater than p 3x > 7 lime <![CDATA[Greater than p 4x > 8 Volcanic ash <![CDATA[Greater than p 5x > 5
[0086] Among them, p 1x 、p 2x 、p 3x 、p 4x and p 5x All of them are preset floating object reference air densities, which can be set according to actual needs and are not limited here.
[0087] In some embodiments, step S107 may include but is not limited to steps S501 to S502:
[0088] Step S501, determining a target vehicle protection strategy corresponding to a target floating object adsorption level from a plurality of vehicle protection strategies according to the floating object adsorption level threshold interval corresponding to each vehicle protection strategy, wherein the floating object adsorption level corresponding to the target vehicle protection strategy matches the target floating object adsorption level;
[0089] Step S502: generating a vehicle protection control instruction according to the target vehicle protection strategy, and controlling the vehicle to perform floating object protection processing according to the vehicle protection control instruction.
[0090] In some embodiments, the vehicle protection strategy may include, but is not limited to, a windshield cleaning strategy and a general protection strategy.
[0091] Exemplarily, multiple floating object adsorption level thresholds are set, and multiple adsorption level threshold intervals are determined according to each floating object adsorption level threshold. A target adsorption level threshold interval corresponding to the target floating object adsorption level is determined from the multiple adsorption level threshold intervals, and each target adsorption level threshold interval corresponds to a different vehicle protection strategy.
[0092] Optionally, the vehicle protection strategy is as follows:
[0093] When the target floating object adsorption level is in the first adsorption level threshold range, it is determined that the current floating object will not affect the front windshield line of sight, and the wiper will not work, such as the smoke produced by burning plants on the roadside, and the general protection strategy will not be implemented;.
[0094] When the target floating object adsorption level is within the second adsorption level threshold range, it is determined that the current floating object will be adsorbed on the glass and will not be more easily adsorbed after encountering water, then the windshield wipers are automatically turned on, and water is automatically sprayed to clean the front windshield, and a general protection strategy is executed;
[0095] When the target floating object adsorption level is in the third adsorption level threshold range, it is determined that the floating objects will be adsorbed on the glass and will be more easily adsorbed after meeting water, such as lime left behind by a truck, a large amount of dust and mud, etc. In this case, only the wipers will be turned on, the general protection strategy will be executed, and the owner will be prompted by voice with the name of the substance ahead, warning the owner to try not to enter the front gear for spraying water, and to choose a safe area to park when the opportunity arises.
[0096] Optionally, the general protection strategy may include, but is not limited to: automatically closing windows and sunroofs; turning off the air conditioning external circulation and adjusting it to internal circulation to prevent external air pollution; automatically activating air purifiers; pausing the audio and video entertainment system; automatically turning on the double flash to warn the following vehicles.
[0097] Reference Figure 2 , Figure 2 : is an optional structural diagram of a road floating object sensing and processing device provided in an embodiment of the present application. The device can be used to implement the above-mentioned road floating object sensing and processing method. The device may include:
[0098] The first module is used to obtain a floating object sample image dataset and a corresponding floating object sample radar dataset;
[0099] The second module is used to obtain a floating object perception model, train the floating object perception model using a floating object sample image data set, and obtain a model training output result;
[0100] The third module is used to train the floating object recognition model using the model training output results and the floating object sample radar data set;
[0101] The fourth module is used to dynamically collect environmental image data and radar data of the road surface on which the vehicle is traveling;
[0102] A fifth module is used to input the environmental image data and the radar data into the trained floating object perception model, use the trained floating object perception model to perform floating object perception and recognition, and output corresponding floating object status information, wherein the floating object status information at least includes the floating object category and the floating object area;
[0103] The sixth module is used to input the floating object state information and radar data into the trained floating object recognition model, use the trained floating object recognition model to identify the nature of the floating objects, and output the corresponding target floating object adsorption level;
[0104] The seventh module is used to obtain a vehicle protection plan, determine a target vehicle protection strategy corresponding to a target floating object adsorption level from the vehicle protection plan, generate a vehicle protection control instruction according to the target vehicle protection strategy, and control the vehicle to perform floating object protection processing according to the vehicle protection control instruction.
[0105] The specific implementation of the road floating object sensing and processing device is basically the same as the specific implementation of the road floating object sensing and processing method described above, and will not be repeated here.
[0106] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned road floating object sensing and processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.
[0107] See also Figure 3 , Figure 3 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0108] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0109] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the road floating object perception and processing method of the embodiment of this application;
[0110] Input / output interface 903, used to implement information input and output;
[0111] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0112] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0113] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0114] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned road floating object perception and processing method is implemented.
[0115] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0116] The embodiment of the present application also provides a vehicle, which includes the electric drive assembly of the above-mentioned road floating object sensing and processing device or electronic device. Specifically, the vehicle can be a private car, such as a sedan, SUV, MPV or pickup truck. The vehicle can also be an operating vehicle, such as a van, bus, small truck or large trailer. The vehicle can be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it can be a hybrid vehicle.
[0117] The embodiments of the present application provide a method, device, electronic device, medium and vehicle for sensing and processing road floating objects, which dynamically collects environmental image data and radar data of the road surface on which the vehicle is traveling, uses a floating object sensing model to sense and identify floating objects based on the environmental image data, outputs corresponding floating object state information, uses a floating object recognition model to recognize the nature of floating objects based on radar data and floating object state information, outputs the corresponding target floating object adsorption level, determines a target vehicle protection strategy corresponding to the target floating object adsorption level from the vehicle protection plan, generates a vehicle protection control instruction, and controls the vehicle to perform floating object protection processing according to the vehicle protection control instruction. The embodiments of the present application can automatically sense and identify road floating objects, determine the target floating object adsorption level of road floating objects, and perform vehicle protection processing according to the target floating object adsorption level, so that road floating objects will not be adsorbed on the vehicle, thereby improving driving safety.
[0118] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0119] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0120] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0121] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0122] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0123] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0124] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0125] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0127] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.
[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0129] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A method for sensing and processing floating objects on the road, It is characterized in that The method comprises the following steps: Obtaining a floating object sample image dataset and a corresponding floating object sample radar dataset; Acquire a floating object perception model, train the floating object perception model using the floating object sample image data set, and obtain a model training output result; Training a floating object recognition model using the model training output result and the floating object sample radar data set; Dynamically collect environmental image data and radar data of the road surface on which the vehicle is traveling; Inputting the environmental image data and the radar data into a trained floating object perception model, performing floating object perception and recognition using the trained floating object perception model, and outputting corresponding floating object status information; the floating object status information at least includes a floating object category and a floating object area; Inputting the floating object state information and the radar data into a trained floating object recognition model, using the trained floating object recognition model to identify the nature of the floating objects, and outputting a corresponding target floating object adsorption level; A vehicle protection plan is obtained, a target vehicle protection strategy corresponding to the target floating object adsorption level is determined from the vehicle protection plan, a vehicle protection control instruction is generated according to the target vehicle protection strategy, and the vehicle is controlled to perform floating object protection processing according to the vehicle protection control instruction.
2. According to the method for sensing and processing floating objects on the road as described in claim 1, It is characterized in that The obtaining of the floating object sample image data set and the corresponding floating object sample radar data set specifically includes: Collecting a plurality of floating object sample images with floating object status labels and sample radar data with floating object water adsorption level labels corresponding to each of the floating object sample images; the floating object status labels at least include the floating object type and the floating object position; All of the floating object sample images are combined to obtain the floating object sample image data set, and all of the sample radar data are combined to obtain the floating object sample radar data set.
3. According to the method for sensing and processing floating objects on the road as described in claim 2, It is characterized in that The method of acquiring a floating object perception model and training the floating object perception model using the floating object sample image data set to obtain a model training output result specifically includes: Using each of the floating object sample images to train the floating object perception model, and outputting floating object sample state information corresponding to each of the floating object sample images; According to the floating object state label corresponding to each of the floating object sample images and the floating object sample state information corresponding to each of the floating object sample images, a model evaluation is performed on the floating object perception model to determine a first model evaluation result, and according to the first model evaluation result, it is determined whether to continue training the floating object perception model.
4. The method for sensing and processing floating objects on the road according to claim 3, It is characterized in that The method of training a floating object recognition model using the model training output result and the floating object sample radar data set specifically includes: Acquire the floating object sample state information and the sample radar data corresponding to each of the floating object sample images; Using the floating object sample state information and the sample radar data corresponding to each of the floating object sample images, the floating object recognition model is trained to output the floating object sample adsorption level corresponding to each of the sample radar data; According to the floating object sample adsorption level and the floating object water adsorption level label corresponding to each sample radar data, the floating object recognition model is evaluated to determine a second model evaluation result, and according to the second model evaluation result, it is determined whether to continue training the floating object recognition model.
5. The method for sensing and processing floating objects on the road according to claim 1, It is characterized in that The vehicle protection scheme includes a plurality of vehicle protection strategies and floating object absorption levels corresponding to the vehicle protection strategies. The obtaining of the vehicle protection scheme includes determining a target vehicle protection strategy corresponding to the target floating object absorption level from the vehicle protection scheme, generating a vehicle protection control instruction according to the target vehicle protection strategy, and controlling the vehicle to perform floating object protection processing according to the vehicle protection control instruction, specifically including: Determining the target vehicle protection strategy corresponding to the target floating object adsorption level from the plurality of vehicle protection strategies according to the floating object adsorption level threshold interval corresponding to each of the vehicle protection strategies, wherein the floating object adsorption level corresponding to the target vehicle protection strategy matches the target floating object adsorption level; The vehicle protection control instruction is generated according to the target vehicle protection strategy, and the vehicle is controlled to perform floating object protection processing according to the vehicle protection control instruction.
6. The method for sensing and processing floating objects on the road according to claim 3, It is characterized in that The method further comprises: performing a model evaluation on the floating object perception model according to the floating object state label corresponding to each of the floating object sample images and the floating object sample state information corresponding to each of the floating object sample images, determining a first model evaluation result, and determining whether to continue training the floating object perception model according to the first model evaluation result. Get multiple model evaluation metrics; Determining a model evaluation value corresponding to each of the model evaluation indicators according to the floating object state label corresponding to each of the floating object sample images and the floating object sample state information corresponding to each of the floating object sample images; When the model evaluation value corresponding to each of the model evaluation indicators exceeds the model evaluation indicator threshold given by each of the model evaluation indicators, the first model evaluation result is determined to be model evaluation passed and the training of the floating object perception model is stopped; otherwise, the training of the floating object perception model is continued.
7. A device for sensing and processing floating objects on the road, It is characterized in that The device comprises: The first module is used to obtain a floating object sample image dataset and a corresponding floating object sample radar dataset; The second module is used to obtain a floating object perception model, train the floating object perception model using the floating object sample image data set, and obtain a model training output result; A third module is used to train a floating object recognition model using the model training output result and the floating object sample radar data set; The fourth module is used to dynamically collect environmental image data and radar data of the road surface on which the vehicle is traveling; A fifth module is used to input the environmental image data and the radar data into a trained floating object perception model, use the trained floating object perception model to perform floating object perception and recognition, and output corresponding floating object status information; the floating object status information at least includes a floating object category and a floating object area; A sixth module is used to input the floating object state information and the radar data into the trained floating object recognition model, use the trained floating object recognition model to identify the nature of the floating objects, and output the corresponding target floating object adsorption level; The seventh module is used to obtain a vehicle protection plan, determine a target vehicle protection strategy corresponding to the target floating object adsorption level from the vehicle protection plan, generate a vehicle protection control instruction according to the target vehicle protection strategy, and control the vehicle to perform floating object protection processing according to the vehicle protection control instruction.
8. An electronic device, It is characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the road floating object sensing and processing method according to any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method for sensing and processing road floating objects as claimed in any one of claims 1 to 6 is implemented.
10. A vehicle, It is characterized in that The vehicle includes the road floating object sensing and processing device as described in claim 7 or the electronic device as described in claim 8.
Citation Information
Patent Citations
Running device and strategy of unmanned vehicle in smog, sand and dust weather
CN106740705A
Unmanned automobile collision avoidance method and system based on artificial intelligence
CN114415683A
Vehicle sensor cleaning method and system and vehicle
CN117207917A
Control method and apparatus for wiper
KR1020180051745A
Vehicle driving method, apparatus, and system
WO2022228251A1
Cited By
Floating object identification method and device, computer program product and electronic equipment
CN120318504A
Floating object identification method, device, computer program product and electronic equipment
CN120318504B
