Vehicle sensing method and device, electronic equipment and storage medium

Abstract

The invention discloses a vehicle sensing method and device, electronic equipment and a storage medium, and relates to the field of unmanned driving and artificial intelligence. The method comprises the steps that original data, sensed by an operation vehicle, of multiple modes are obtained, and the original data of the multiple modes are obtained by collecting the surrounding environment of the operation vehicle through multiple types of sensors installed on the operation vehicle; determining a data enhancement strategy and a corresponding data enhancement probability corresponding to the original data of the corresponding modality according to the modality types of the original data of the different modalities; based on the data enhancement strategies corresponding to the original data of the different modalities, performing data enhancement on the original data of the corresponding modalities according to the corresponding data enhancement probabilities to obtain enhanced data of the different modalities; and performing feature fusion on the enhanced data of different modes to obtain a sensing result. According to the invention, the technical problem that the perception accuracy of the vehicle to the obstacle is low due to the interference of weather factors in severe weather is solved.

Description

Technical Field

[0001] The present invention relates to the fields of unmanned driving and artificial intelligence, and in particular to a vehicle perception method, device, electronic equipment and storage medium. Background Art

[0002] The field of unmanned driving in mines requires high-efficiency operations around the clock. Due to the complex mining environment and the presence of many unsafe factors, such as ores, equipment, vehicles, etc., obstacles need to be identified to avoid collisions and accidents.

[0003] Usually, unmanned mining vehicles use different sensors to collect perception data and identify obstacles based on the collected perception data. Due to the scene characteristics of mine autonomous driving, vehicles should have the ability to operate in bad weather. However, under bad weather conditions, due to the influence of particles in rain, snow, fog, and dust, when identifying obstacles, there is a problem of misidentifying noise as a target object or missing the target object in the noise, which leads to a low accuracy rate of the vehicle's perception of obstacles and cannot guarantee the safety and efficiency of unmanned driving.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a vehicle perception method, device, electronic device and storage medium to at least solve the technical problem of low accuracy of vehicle perception of obstacles due to interference from weather factors in bad weather.

[0006] According to one aspect of an embodiment of the present invention, a vehicle perception method is provided, comprising: acquiring raw data of multiple modes perceived by a working vehicle, wherein the raw data of multiple modes are respectively collected by multiple types of sensors installed on the working vehicle on the surrounding environment of the working vehicle, and the multiple types of sensors include at least two of the following: a visual sensor, a lidar sensor, and a millimeter-wave radar sensor; determining a data enhancement strategy and a corresponding data enhancement probability corresponding to the raw data of a corresponding modality according to the modality type of the raw data of different modalities, wherein the data enhancement probability is used to characterize the probability of performing data enhancement by using the corresponding data enhancement strategy for the raw data of the corresponding modality; based on the data enhancement strategy corresponding to the raw data of different modalities, performing data enhancement on the raw data of the corresponding modality according to the corresponding data enhancement probability to obtain enhanced data of different modalities; and performing feature fusion on the enhanced data of different modalities to obtain a perception result.

[0007] According to another aspect of an embodiment of the present invention, a perception device for a vehicle is also provided, including: a data acquisition module, used to acquire raw data of multiple modes perceived by a working vehicle, wherein the raw data of multiple modes are obtained by collecting the surrounding environment of the working vehicle by multiple types of sensors installed on the working vehicle, and the multiple types of sensors include at least two of the following: a visual sensor, a lidar sensor, and a millimeter-wave radar sensor; a strategy determination module, used to determine the data enhancement strategy and the corresponding data enhancement probability corresponding to the raw data of the corresponding modality according to the modality type of the raw data of different modalities, wherein the data enhancement probability is used to characterize the probability of using the corresponding data enhancement strategy for data enhancement for the raw data of the corresponding modality; a data enhancement module, used to perform data enhancement on the raw data of the corresponding modality according to the corresponding data enhancement probability based on the data enhancement strategy corresponding to the raw data of different modalities, to obtain enhanced data of different modalities; a feature fusion module, used to perform feature fusion on the enhanced data of different modalities to obtain a perception result.

[0008] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0009] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0010] In an embodiment of the present invention, the raw data of multiple modes perceived by the working vehicle are obtained; according to the modal type of the raw data of different modes, the data enhancement strategy and the corresponding data enhancement probability corresponding to the raw data of the corresponding mode are determined; based on the data enhancement strategy corresponding to the raw data of different modes, the raw data of the corresponding mode is data enhanced according to the corresponding data enhancement probability to obtain enhanced data of different modes; the enhanced data of different modes are feature fused to obtain the perception result. It is easy to notice that according to the modal type of the raw data, the data enhancement strategy and the corresponding data enhancement probability corresponding to the raw data of the corresponding mode are determined, the raw data of multiple modes perceived by the vehicle are data enhanced, and the feature fusion is performed using the data after data enhancement, so that the data used for feature fusion has stronger authenticity, covers a variety of interference factors in bad weather, and more realistically simulates the sensor anomalies in bad weather and the impact of particles in rain, snow, fog, and dust on vehicle perception, improves the accuracy of vehicle perception, and thus solves the technical problem of low accuracy of vehicle perception of obstacles due to interference from weather factors in bad weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1 is a flow chart of a vehicle perception method according to an embodiment of the present invention;

[0013] Figure 2 is a schematic diagram of an optional feature fusion model for performing feature fusion according to an embodiment of the present invention;

[0014] Figure 3 is a flow chart of an optional training feature fusion model according to an embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of an optional acquisition range covered by a camera according to an embodiment of the present invention;

[0016] Figure 5 is a schematic diagram of a collection range covered by an optional laser radar sensor according to an embodiment of the present invention;

[0017] Figure 6 is a schematic diagram of a vehicle sensing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] According to an embodiment of the present invention, an embodiment of a vehicle perception method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Figure 1 is a flow chart of a vehicle perception method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0022] Step S102, obtaining raw data of multiple modes perceived by the working vehicle, wherein the raw data of multiple modes are obtained by collecting the surrounding environment of the working vehicle by multiple types of sensors installed on the working vehicle, and the multiple types of sensors include at least two of the following: visual sensors, lidar sensors, and millimeter wave radar sensors.

[0023] The operating vehicle in the above steps may be a mining truck that performs operations in adverse weather conditions, and may be an unmanned vehicle or a manually driven vehicle.

[0024] The visual sensor in the above steps is a sensor that can capture image data or video data, and can be a device that captures image data such as a telephoto camera, a mid-focus camera, a fisheye camera, or a device that captures image data such as a camera, but is not limited to this.

[0025] The lidar sensor in the above steps is a sensor that uses lasers to measure the distance and speed of objects. It measures the distance and angle between the working vehicle and objects in the surrounding environment by emitting laser beams and receiving reflected beams. The data generated by the lidar sensor is usually point cloud data, which contains a large number of three-dimensional spatial coordinate points that reflect the geometric structure of the surrounding environment.

[0026] The millimeter wave radar sensor in the above steps is a radar system that uses millimeter wave band radio waves for detection and measurement, and detects the distance and angle between the working vehicle and objects in the surrounding environment by transmitting and receiving millimeter wave band electromagnetic waves.

[0027] The raw data of multiple modes in the above steps are data collected by various types of sensors on the surrounding environment of the working vehicle, and may include point cloud data collected by a lidar sensor, image data or video data collected by a visual sensor, point cloud data collected by a lidar sensor, millimeter-wave radar data collected by a millimeter-wave radar sensor, image data or video data collected by a visual sensor, millimeter-wave radar data collected by a millimeter-wave radar sensor, point cloud data collected by a lidar sensor, image data or video data collected by a visual sensor, and millimeter-wave radar data collected by a millimeter-wave radar sensor.

[0028] In an optional embodiment, since visual sensors, lidar sensors and millimeter-wave radar sensors all have their own limitations, for example: visual sensors may not be able to accurately identify objects under low light or bad weather conditions, lidar sensors may be disturbed under bad weather conditions, resulting in reduced point cloud quality, and although millimeter-wave radars have good robustness to bad weather, their resolution is relatively low and it is difficult to identify small objects or complex scenes, etc., multiple types of sensors are installed on the work vehicle to obtain raw data of multiple modes, so that the raw data of multiple modes complement each other's deficiencies and improve the overall perception ability. A preferred embodiment is to install lidar sensors, visual sensors and millimeter-wave radar sensors on the vehicle at the same time.

[0029] The sensor for acquiring raw data may be a visual sensor. A camera may be selected as a visual sensor. The forward camera is usually installed at the front of the vehicle so that the collected image data can cover the road ahead. The side cameras are installed on both sides of the vehicle. The rear camera is installed at the rear of the vehicle to obtain more comprehensive image data of the surrounding environment of the working vehicle. Connect the power cord of the camera to the power system of the working vehicle to ensure that the power system can continuously power the camera. Connect the data transmission line of the camera to the data processing unit to transmit image data to the data processing unit. Configure the camera's resolution, frame rate, exposure and other parameters, turn on the camera, start collecting image data of the surrounding environment of the working vehicle, and obtain raw data. Using a visual sensor to collect image data can reduce the probability of misdetecting noise such as rain, snow, fog, and dust as targets during the perception process, and can improve the ability to recognize low obstacles and pedestrians at a distance.

[0030] The sensor for obtaining raw data can also be a lidar sensor. A lidar sensor can be installed at the front of the vehicle, or a high-beam lidar can be used as the main radar and a low-beam lidar can be used as a blind radar. A main radar and multiple blind radars can be installed on the body of the working vehicle to obtain more comprehensive point cloud data of the surrounding environment of the working vehicle. Set the parameters of the lidar sensor such as laser emission frequency, resolution, and field of view, and adjust the horizontal and vertical angles of the lidar sensor to calibrate the lidar sensor to ensure the accuracy of the measurement. Then start the lidar sensor so that the lidar sensor starts to emit laser pulses and receive reflected signals to collect distance information of the surrounding environment. The received reflected signals are converted into point cloud data to obtain raw data. The raw data collected by the lidar sensor can provide three-dimensional information about obstacles in the environment, including size, position, and shape, which is conducive to the working vehicle's perception of the surrounding environment.

[0031] The sensor for obtaining raw data can also be a millimeter-wave radar sensor. The millimeter-wave radar sensor can be installed in front and behind the working vehicle, or in front, on the side and behind the working vehicle to ensure that the millimeter-wave radar sensor has a good field of view and covers the surrounding environment of the working vehicle. The millimeter-wave radar sensor is calibrated by adjusting the antenna direction and setting the signal sensitivity to ensure the measurement accuracy of the millimeter-wave radar sensor. Then the millimeter-wave radar sensor is turned on for data collection, and the distance, speed, angle and other information of objects in the surrounding environment of the working vehicle are detected by transmitting and receiving reflected electromagnetic waves to obtain raw data. Using millimeter-wave radar to collect raw data can reduce the probability of perception missed detection caused by point cloud and image occlusion in bad weather.

[0032] Step S104, according to the modality type of the original data of different modalities, determine the data enhancement strategy and the corresponding data enhancement probability corresponding to the original data of the corresponding modality, wherein the data enhancement probability is used to characterize the probability of using the corresponding data enhancement strategy for data enhancement for the original data of the corresponding modality.

[0033] The modal type of the raw data in the above steps can be divided according to the type of sensor that obtains the raw data, or according to the type of raw data, but the division dimension may not be limited to this. When divided according to the type of sensor that obtains the raw data, the modal type of the raw data may include visual sensors, lidar sensors, visual sensors, millimeter-wave radar sensors, lidar sensors, millimeter-wave radar sensors, and visual sensors, lidar sensors, and millimeter-wave radar sensors, wherein the visual sensor collects image data, the lidar sensor collects lidar data, and the millimeter-wave radar sensor collects millimeter-wave radar data. The modal type of raw data can also be divided into image data and point cloud data according to the type of raw data, wherein the image data is collected by a visual sensor, and the point cloud data is collected by a lidar sensor or a millimeter-wave radar sensor.

[0034] The data enhancement strategy in the above steps is a strategy for enhancing the original data of different modal types. Data enhancement is a way to increase data diversity. The data enhancement strategy can be geometric transformation, color transformation, occlusion and masking, noise injection, random erasure, conditional variation, etc., but is not limited to these.

[0035] In an optional embodiment, the modal type of the raw data can be determined according to the source of the raw data. In the actual operating environment, the raw data collected by the sensor will be interfered by various factors, such as the obstruction of rain, snow, mud, dust, and leaves in bad weather, partial loss caused by sensor failure, image distortion, point cloud noise, abnormal sensor data access, etc.

[0036] The modality type can be divided into three types according to the type of sensor that collects raw data: visual sensor, lidar sensor, and millimeter wave radar sensor. Different types of sensors encounter different interferences when collecting raw data, so data enhancement strategies can be determined based on the different interferences encountered by different types of sensors. For example, when collecting raw data, the visual sensor encounters interferences including blur in dynamic scenes, objects being occluded, etc. Therefore, the data enhancement strategy corresponding to the image data can be determined as motion module, occlusion, etc.

[0037] When determining the data enhancement probability, the probability of different types of sensors encountering interference in actual applications can be counted, and the result can be used as the data enhancement probability. The probability of different types of sensors encountering interference in multiple actual applications can also be counted, and the average can be taken as the data enhancement probability. A value range can also be set based on the probability of different types of sensors encountering interference, and then a random number within the value range can be generated and used as the data enhancement probability, but the method for determining the data enhancement probability may not be limited to this.

[0038] In another optional embodiment, if the modality type can be divided into image data and point cloud data, the data enhancement strategy can be determined based on the differences in data characteristics. Specifically, image data usually exists in the form of a pixel matrix, and the inaccuracy of the pixel matrix is ​​usually due to the presence of noise, so the enhancement strategy for image data can be random noise. Point cloud data exists in the form of a set of points in three-dimensional space, and each point contains position information and possible reflection intensity information. Therefore, the enhancement strategy for point cloud data needs to take into account the characteristics of three-dimensional space, such as random point cloud erasure, intensity reduction, etc.

[0039] Step S106, based on the data enhancement strategies corresponding to the original data of different modalities, the original data of the corresponding modality is enhanced according to the corresponding data enhancement probability to obtain enhanced data of different modalities.

[0040] The enhanced data of different modes in the above steps are data obtained after data enhancement is performed on the original data of different modes.

[0041] In an optional embodiment, when there is a data enhancement strategy corresponding to the original data of different modalities, it is determined whether the original data of different modalities need to be modally enhanced according to the data enhancement probability. If necessary, the data enhancement strategy corresponding to the original data of different modalities is used to enhance the original data of different modalities to obtain enhanced data.

[0042] In another optional embodiment, when there are multiple data enhancement strategies corresponding to the original data of different modalities, for each data enhancement strategy, the data enhancement probability is used to determine whether to enhance the original data of different modalities with the data enhancement strategy, and if necessary, the data enhancement strategy is used to enhance the original data of different modalities to obtain enhanced data. If not, the next data enhancement strategy is judged, and the data enhancement probability is used to determine whether to enhance the original data of different modalities with the next data enhancement strategy.

[0043] In another optional embodiment, when there are multiple data enhancement strategies corresponding to original data of different modalities, it is determined whether the original data of different modalities need to be modally enhanced according to the data enhancement probability. If necessary, the original data of different modalities are enhanced using multiple data enhancement strategies to obtain enhanced data; if not necessary, the original data of different modalities are not enhanced.

[0044] Step S108, performing feature fusion on enhanced data of different modalities to obtain a perception result.

[0045] The feature fusion in the above steps is a data processing technology that integrates data from different sources or different types of sensors to improve the richness and accuracy of information.

[0046] The perception results in the above steps are the results of the working vehicle's surrounding environment, including the identification, classification and positioning of obstacles. Perceiving the working vehicle's surrounding environment can enable the working vehicle to accurately understand the surrounding environment and maintain this perception capability in complex environments and adverse weather conditions, providing safe and reliable decision support for mine autonomous driving.

[0047] In an optional embodiment, after data enhancement is performed on original data of different modalities to obtain enhanced data, a feature fusion model can be used to directly perform feature fusion processing on the enhanced data of different modalities to obtain a perception result.

[0048] In another optional embodiment, a special extraction and feature conversion method can be used for feature fusion. Specifically, image processing and computer vision technology can be used to extract image features such as edges, corners, and textures from the enhanced image data, and to extract features such as point cloud density, height, and normal from the enhanced laser radar data, and to extract features such as distance, speed, and angle from the millimeter wave radar data. Through normalization or dimensionality reduction operations, the extracted features are converted into a unified form, and the unified features are combined to form a new feature vector, thereby obtaining a perception result.

[0049] In extremely bad weather scenarios, noise points such as rain, snow, fog, and dust have uncertain shapes. It is difficult to distinguish between noise and target objects in the surrounding environment through point cloud data alone, resulting in false detection. Although the introduction of time series information or the tracking fusion module can eliminate some false detections, it cannot be completely solved. Therefore, the enhanced data of the visual modality is introduced, that is, the data after the image data is enhanced. The enhanced data of the visual modality brings richer semantic category information, which can help the model easily distinguish the target, noise and background, making the model more robust to the noise introduced by bad weather; at the same time, the telephoto camera as a visual sensor can observe small obstacles at a long distance, such as stones, pedestrians, etc., which can improve the accuracy of perception. In addition, in bad weather scenarios, both lidar data and image data will be severely interfered with, and there will be problems such as signal loss and noise interference that make the target invisible. Therefore, it is necessary to fuse the enhanced data corresponding to the millimeter-wave radar data, take advantage of its stability in bad weather, and provide a bottom line for the overall perception solution in extreme weather conditions. In summary, feature fusion of enhanced data of different modalities can make the enhanced data of different modalities complement each other's shortcomings, improve the perception ability in bad weather, and make the perception results more accurate.

[0050] In an embodiment of the present invention, the raw data of multiple modes perceived by the working vehicle are obtained; according to the modal type of the raw data of different modes, the data enhancement strategy and the corresponding data enhancement probability corresponding to the raw data of the corresponding mode are determined; based on the data enhancement strategy corresponding to the raw data of different modes, the raw data of the corresponding mode is data enhanced according to the corresponding data enhancement probability to obtain enhanced data of different modes; the enhanced data of different modes are feature fused to obtain the perception result. It is easy to notice that the data enhancement strategy and the corresponding data enhancement probability corresponding to the raw data of the corresponding mode are determined according to the modal type of the raw data, and the raw data of multiple modes perceived by the vehicle are data enhanced to overcome the influence of sensor abnormality and particles in rain, snow, fog and dust on vehicle perception in bad weather. Feature fusion is performed on the enhanced data, which can improve the accuracy of vehicle perception, thereby solving the technical problem of low accuracy of vehicle perception of obstacles caused by interference of weather factors in bad weather.

[0051] Optionally, based on the data enhancement strategies corresponding to the original data of different modalities, data enhancement is performed on the original data of the corresponding modality according to the corresponding data enhancement probability to obtain enhanced data of different modalities, including: randomly generating a first probability corresponding to the original data of the corresponding modality; in response to the first probability being greater than or equal to the corresponding data enhancement probability, based on the target data enhancement strategy corresponding to the original data of the corresponding modality, data enhancement is performed on the original data of the corresponding modality to obtain enhanced data of the corresponding modality; in response to the first probability being less than the corresponding data enhancement probability, prohibiting the target data enhancement strategy corresponding to the original data of the corresponding modality from being used to enhance the original data of the corresponding modality.

[0052] The first probability in the above steps is a randomly generated probability, which corresponds to the original data of different modalities. The original data of different modalities correspond to different first probabilities, and the first probability can be used to determine whether the original data of the corresponding modality needs to be enhanced.

[0053] In an optional embodiment, for raw data of different modalities, a random number generator may be used to generate a value between 0 and 1 as a first probability. Then, the first probability is compared with the data enhancement probability corresponding to the raw data of the corresponding modality. If the first probability is greater than the data enhancement probability, all data enhancement strategies corresponding to the raw data of the corresponding modality are used to perform data enhancement on the raw data of the corresponding modality to obtain enhanced data. If the first probability is less than the data enhancement probability, the raw data of the corresponding modality is not enhanced, and the raw data of the corresponding modality is used for feature fusion.

[0054] In another optional embodiment, an interval of randomly generated first probabilities may be set between 0 and 1 according to industry experience, reference materials or actual conditions, and then a random number generator may be used to generate a random number within the interval as the first probability. Then, by comparing the first probability with the data enhancement probability corresponding to the original data of the corresponding modality, it is determined whether to perform data enhancement on the original data of the corresponding modality.

[0055] Optionally, in the case where the data enhancement strategy corresponding to the original data of the corresponding modality includes multiple data enhancement strategies, the corresponding data enhancement probability includes: the probabilities corresponding to the multiple data enhancement strategies; based on the data enhancement strategies corresponding to the original data of different modalities, the original data of the corresponding modality is data enhanced according to the corresponding data enhancement probabilities to obtain enhanced data of different modalities, including: randomly generating second probabilities corresponding to multiple data enhancement strategies; in response to the second probability corresponding to any data enhancement strategy being greater than or equal to the probability corresponding to the preset data enhancement strategy, based on the data enhancement strategy, the original data of the corresponding modality is data enhanced to obtain enhanced data of the corresponding modality; in response to the second probabilities corresponding to multiple data enhancement strategies being less than the probabilities corresponding to the multiple data enhancement strategies, it is prohibited to perform data enhancement on the original data of the corresponding modality based on multiple data enhancement strategies.

[0056] The second probability in the above steps is a randomly generated probability, which has a corresponding relationship with the data enhancement strategy. Different data enhancement strategies correspond to different second probabilities. The second probability can be used to determine whether the original data of the corresponding modality should be enhanced according to the data enhancement strategy corresponding to the second probability.

[0057] In an optional embodiment, the original data of the corresponding modality corresponds to multiple data enhancement strategies. For each of the multiple data enhancement strategies, a random number generator is used to generate a value between 0 and 1 as the second probability corresponding to the data enhancement strategy. Compare the second probability corresponding to the data enhancement strategy and the data enhancement probability corresponding to the data enhancement strategy. If the second probability is greater than the data enhancement probability, the data enhancement measurement is used to perform data enhancement on the original data of the corresponding modality; if the second probability is less than the data enhancement probability, the data enhancement measurement is not used to perform data enhancement on the original data of the corresponding modality, but the relationship between the second probability corresponding to the next data enhancement strategy and the data enhancement probability corresponding to the next data enhancement strategy is continued to be determined. If the original data of the corresponding modality corresponds to multiple data enhancement strategies, and the second probability corresponding to each data enhancement strategy is less than the data enhancement probability corresponding to the data enhancement strategy, the original data of the corresponding modality is not data enhanced. Exemplarily, when the modality type of the original data of the corresponding modality is a visual sensor, and the corresponding data enhancement strategy includes motion blur, color change, and random occlusion, it is determined that the data enhancement probabilities corresponding to motion blur, color change, and random occlusion are 0.5, 0.3, and 0.7, respectively. If the second probabilities of motion blur, color change, and random occlusion are randomly generated by 0.4, 0.9, and 0.8 respectively, it can be seen that the second probability of color change is greater than the data enhancement probability, and the second probability of random occlusion is also greater than the data enhancement probability. For the original data collected by the visual sensor, the data enhancement strategies used in data enhancement are color change and random occlusion; if the second probabilities of motion blur, color change, and random occlusion are randomly generated by 0.4, 0.3, and 0.35 respectively, the second probabilities of the three data enhancement strategies are all less than the data enhancement probability, then the three data enhancement strategies are not used, and no data enhancement operation is performed on the original data collected by the visual sensor. It should be noted that the above values ​​are only examples, and the specific values ​​involved in actual applications may not be limited to these.

[0058] In another optional embodiment, an interval of randomly generated second probabilities can be set between 0 and 1 based on industry experience, reference materials or actual conditions, and then a random number generator is used to generate a random number within the interval as the second probability of the data enhancement strategy. Then, by comparing the magnitude relationship between the second probability of the data enhancement strategy and the data enhancement probability of the data enhancement strategy, it is determined whether to use the data enhancement strategy to perform data enhancement on the original data of the corresponding modality.

[0059] Optionally, in response to the modality type of the original data of the corresponding modality being a visual sensor, multiple data enhancement strategies are determined to include at least two of the following: motion blur, random noise, and random occlusion; in response to the modality type of the original data of the corresponding modality being a lidar sensor or a millimeter-wave radar sensor, multiple data enhancement strategies are determined to include at least two of the following: random point cloud erasure, intensity attenuation, and random noise addition.

[0060] The motion blur in the above steps can be used to simulate the instability of the visual sensor during the shooting process. Motion blur is a common phenomenon when the visual sensor captures images or videos, especially when the object being photographed moves quickly or is photographed using a slow shutter speed.

[0061] The random noise in the above steps is pixel-level noise randomly added to the image data collected by the visual sensor, which can simulate the degradation of image quality caused by poor lighting conditions, sensor quality problems or interference during image transmission.

[0062] The random occlusion in the above steps is to randomly add masks or occluders to certain parts of the image data collected by the visual sensor, thereby simulating the situation where the object is partially occluded. The added mask or occluder can be a simple black or white mask or an occlusion with a certain degree of transparency.

[0063] The random point cloud erasing in the above steps refers to randomly removing certain points in the point cloud data, simulating the situation in the actual environment where the sensor may lose some data due to occlusion, failure or other factors, wherein the point cloud data is data collected by the lidar sensor or the millimeter wave radar sensor.

[0064] The intensity reduction in the above steps refers to a data enhancement strategy that simulates signal attenuation or weakening by reducing the intensity value of the point cloud data collected by the lidar sensor or the millimeter wave radar sensor.

[0065] The addition of random noise in the above steps is a data enhancement measurement in which random noise is added to the point cloud data collected by the lidar sensor or millimeter wave radar sensor, that is, small random disturbances are added to the coordinates or intensity values ​​of the point cloud to simulate various interferences that the sensor may be subjected to in the actual environment, such as vibration, electronic noise, etc.

[0066] In an optional embodiment, in the actual operating environment, especially in bad weather, sensor anomalies may occur during the perception process. Sensor failures can be divided into signal pollution and signal loss. Signal pollution can be further divided into signal pollution caused by hardware and signal pollution caused by external objects. Signal pollution caused by hardware usually includes image distortion, point cloud noise, local loss, etc. Signal pollution caused by external objects is mainly reflected in rain, snow, mud, dust, leaves and other external objects blocking signal transmitters, camera lenses, etc. Signal loss refers to the situation where the data access of a certain sensor is abnormal, resulting in a part of the model input being completely blank.

[0067] To address these fault problems, different data enhancement strategies can be formulated for raw data of different modal types to simulate the fault problems that raw data of different modal types may encounter. When the modal type of the raw data is a visual sensor, data enhancement strategies such as motion blur, random noise, and random occlusion can be designed for image data collected by the visual sensor. When the modal type of the raw data is a lidar sensor or a millimeter-wave radar sensor, data enhancement strategies such as random point cloud erasure, intensity reduction, and random noise addition can be designed for point cloud data collected by the lidar sensor or the millimeter-wave radar sensor.

[0068] Optionally, feature fusion is performed on enhanced data of different modalities to obtain a perception result, including: using a feature fusion model to perform feature fusion on enhanced data of different modalities to obtain a perception result.

[0069] The feature fusion model in the above steps is a model that can perform feature fusion on enhanced data of different modalities. It can integrate features from data sets with different sources or different characteristics, so that enhanced data of different modalities can make up for each other's shortcomings, improve perception capabilities in severe weather, and make perception results more accurate.

[0070] In an optional embodiment, Figure 2 is a schematic diagram of an optional feature fusion model for feature fusion according to an embodiment of the present invention, such as Figure 2As shown, the feature fusion model includes a lidar backbone network, an image backbone network, a millimeter wave backbone network, and a BEV (Bird's Eye View) feature fusion module. The lidar data, image data, and millimeter wave radar data after data enhancement are respectively input into the lidar backbone network, the image backbone network, and the millimeter wave backbone network, and the original data of different modes are independently feature encoded. The encoding result is input into the BEV feature fusion module, uniformly converted to the BEV perspective for feature fusion, and the BEV features of the three modal fusions are output. Based on the BEV features of the three modal fusions, plus an arbitrary 3D (Three-Dimensional) target detection head, the position, size, and orientation of the target object are predicted, and the perception results are output.

[0071] In another optional embodiment, after data enhancement is performed on the original data of different modalities to obtain enhanced data, random data loss processing can be performed on the enhanced data, and the missing data can be filled with default values. The filled data is then preprocessed, including image cropping, scaling, point cloud voxelization, etc. The preprocessed data is then used as input data for the feature fusion model, and the feature fusion model is used to perform feature fusion processing on the input data to obtain a perception result.

[0072] Optionally, the feature fusion model is trained in the following manner: according to a preset learning rate, feature extraction modules of different modalities contained in the initial feature model are trained based on training data of multiple modalities, wherein the feature extraction modules of different modalities are used to extract features from enhanced data of the corresponding modalities; the preset learning rate is reduced to obtain a target learning rate; according to the target learning rate, the initial fusion model or the feature fusion module contained in the initial feature model is trained based on target data of different modalities to obtain a feature fusion model, wherein the feature fusion module is used to fuse features output by feature extraction modules of different modalities.

[0073] The preset learning rate in the above steps is a pre-set learning rate, which can be preset according to the actual application situation. The specific data of the preset learning rate is not limited here. The learning rate is an important parameter when training a machine learning model, which determines the step size or amplitude of the model parameter update in each iteration. The preset learning rate can be used to train feature extraction modules of different modalities.

[0074] The target learning rate in the above steps is the learning rate obtained by reducing the preset learning rate. Reducing the preset learning rate can make the training process more stable and prevent the model weights from fluctuating violently during the update process. The target learning rate can be used to train the feature fusion module.

[0075] The target data of different modes in the above steps are training data obtained by fusing the training data of multiple modes, and can be used to train the feature fusion module.

[0076] The initial feature model in the above steps includes feature extraction modules of different modalities. The feature extraction modules of different modalities may be a lidar backbone network, an image backbone network, or a millimeter wave radar backbone network, but are not limited thereto.

[0077] The initial fusion model in the above steps is a model that can include a feature fusion module. The feature fusion module can be located in the initial fusion model or in the initial feature model.

[0078] In an optional embodiment, Figure 3 is a flow chart of an optional training feature fusion model according to an embodiment of the present invention, such as Figure 3 As shown, training data is obtained, and the training data includes single-modal data for training feature extraction modules of different modalities, such as lidar data, image data, millimeter-wave radar data, etc., and also includes multi-modal data obtained after feature fusion of single-modal data. The multi-modal data is used to train a feature fusion module for feature fusion.

[0079] The single-modal data such as lidar data, image data, millimeter-wave radar data, etc. are enhanced according to the rules described above, and then the feature extraction modules of different modalities are trained using the data-enhanced single-modal data according to the preset learning rate to obtain the weights of the feature extraction modules of different modalities. The feature extraction modules are also the lidar backbone network, the image backbone network, and the millimeter-wave radar backbone network.

[0080] Then the preset learning rate is reduced to 1 / 2 to obtain the target learning rate. It should be noted that 1 / 2 is only an example, and the preset learning rate can also be reduced to other values ​​according to actual application conditions. According to the target learning rate, the multimodal data in the training data, that is, the target data of different modes, is used to train the feature fusion module using multi-model data to determine the feature fusion model.

[0081] Training the feature extraction module and feature fusion module in the feature fusion model in two stages can avoid the situation in which the model over-utilizes the data of a single modality in single-stage training, thereby ignoring the finer features of other modalities. This two-stage training scheme can help the model obtain better initialization weights, so that it can achieve higher accuracy in the second-stage training, and further improve the robustness of the feature fusion model used for perception. The feature fusion model is trained using single-modal data such as lidar data, image data, millimeter-wave radar data, and multi-modal data obtained by fusing single-modal data. This can reduce the false detection of rain, snow, fog, and dust as targets in bad weather, and can also reduce missed detections caused by rain, snow, fog, and dust blocking targets.

[0082] Optionally, the visual sensor includes: a first camera installed on the front of the work vehicle, multiple second cameras installed on the front and rear of the work vehicle, and multiple third cameras installed on the front of the work vehicle, wherein the focal length of the third camera is greater than the focal length of the second camera and smaller than the focal length of the first camera, the viewing angle of the third camera is greater than the viewing angle of the first camera and smaller than the viewing angle of the second camera, and the orientations of the multiple third cameras are different; and / or, the lidar sensor includes: a first lidar sensor installed on the front of the work vehicle, and multiple second lidar sensors installed on the front and rear of the work vehicle, wherein the wiring harness of the first lidar sensor is larger than the wiring harness of the second lidar sensor, and the orientation of at least two second lidar sensors installed on the front of the work vehicle is different from that of the first lidar sensor; and / or, the millimeter wave radar sensor is installed on the front and rear of the work vehicle.

[0083] The first camera in the above steps may be a telephoto camera, which is installed on the front of the working vehicle to cover the front distant area of ​​the working vehicle.

[0084] The second camera in the above steps may be a fisheye camera, which is installed at the front and rear of the working vehicle to cover the near area in front and behind the working vehicle.

[0085] The third camera in the above steps may be a medium-focus camera. The third camera installed on the front of the working vehicle has different orientations and can cover the side area of ​​the working vehicle.

[0086] The first laser radar sensor in the above steps is used as the main radar for perception, which can be a high-beam laser radar. The first laser radar sensor is installed on the front of the working vehicle and can cover the front-view distant area of ​​the working vehicle.

[0087] The second laser radar sensor in the above steps is used as a blind spot radar for perception, and can be a low-beam laser radar. The second laser radar sensor is installed on the front and rear of the working vehicle to cover the 360-degree surrounding area.

[0088] In an optional embodiment, Figure 4 is a schematic diagram of an optional acquisition range covered by a camera according to an embodiment of the present invention, such as Figure 4 As shown, the camera scheme adopts a combination of a telephoto camera, a medium focus camera, and a fisheye camera, and the telephoto camera, medium focus camera, and fisheye camera are all installed on the working vehicle 40. Among them, the telephoto camera is the first camera, the fisheye camera is the second camera, and the medium focus camera is the third camera, satisfying that the focal length of the third camera is greater than the focal length of the second camera and less than the focal length of the first camera. Among them, the forward telephoto camera is used to cover the forward distant area 41; a fisheye camera is installed in the middle of the front and the middle of the rear of the vehicle to cover the front and rear near areas 42; two side medium focus cameras with different directions are installed in the front left and right of the front of the vehicle to cover the side area 43 of the vehicle. Among them, the viewing angle of the medium focus camera is greater than that of the telephoto camera, and smaller than that of the fisheye camera.

[0089] Figure 5 is a schematic diagram of an optional collection range covered by a laser radar sensor according to an embodiment of the present invention, such as Figure 5 As shown, the laser radar uses a main radar and three blind spot radars. Among them, the main radar, that is, the first laser radar sensor, uses a high-beam laser radar; the blind spot radar, that is, the second laser radar sensor, uses a low-beam laser radar. The main radar is installed on the front of the working vehicle 51 and can cover the front distant area 52; the three blind spot radars are installed on the front left, front right, and rear middle of the working vehicle 51 to cover the 360-degree surrounding near area 53.

[0090] In addition, the front and rear of the working vehicle are each equipped with a millimeter-wave radar to detect obstacles in front and behind. The above sensor layout scheme not only ensures the need for forward long-distance coverage, but also ensures that the vehicle has 360-degree coverage without blind spots.

[0091] According to an embodiment of the present invention, an embodiment of a vehicle perception device is provided. It should be noted that the device can be used to execute the above-mentioned vehicle perception method.

[0092] Figure 6 is a schematic diagram of a vehicle sensing device according to an embodiment of the present invention, such as Figure 6 As shown, the vehicle's sensing device includes:

[0093] The data acquisition module 60 is used to acquire raw data of multiple modes sensed by the working vehicle, wherein the raw data of the multiple modes are acquired by collecting the surrounding environment of the working vehicle by multiple types of sensors installed on the working vehicle, and the multiple types of sensors include at least two of the following: a visual sensor, a laser radar sensor, and a millimeter wave radar sensor;

[0094] A strategy determination module 62 is used to determine the data enhancement strategy and the corresponding data enhancement probability corresponding to the original data of the corresponding modality according to the modality type of the original data of different modalities, wherein the data enhancement probability is used to characterize the probability of using the corresponding data enhancement strategy for data enhancement for the original data of the corresponding modality;

[0095] A data enhancement module 64 is used to perform data enhancement on the original data of the corresponding modality according to the corresponding data enhancement probability based on the data enhancement strategy corresponding to the original data of the different modalities, so as to obtain enhanced data of the different modalities;

[0096] The feature fusion module 66 is used to perform feature fusion on enhanced data of different modalities to obtain a perception result.

[0097] Optionally, the data enhancement module includes: a first random generation unit, used to randomly generate a first probability corresponding to the original data of the corresponding modality; a first enhancement unit, used to, in response to the first probability being greater than or equal to the corresponding data enhancement probability, perform data enhancement on the original data of the corresponding modality based on a target data enhancement strategy corresponding to the original data of the corresponding modality to obtain enhanced data of the corresponding modality; and a second enhancement unit, used to, in response to the first probability being less than the corresponding data enhancement probability, prohibit data enhancement on the original data of the corresponding modality based on the target data enhancement strategy corresponding to the original data of the corresponding modality.

[0098] Optionally, the data enhancement module also includes: a second random generation unit, used to randomly generate second probabilities corresponding to multiple data enhancement strategies; a third enhancement unit, used to respond to the second probability corresponding to any data enhancement strategy being greater than or equal to the probability corresponding to the preset data enhancement strategy, and based on the data enhancement strategy, perform data enhancement on the original data of the corresponding modality to obtain enhanced data of the corresponding modality; a fourth enhancement unit, used to prohibit data enhancement on the original data of the corresponding modality based on multiple data enhancement strategies in response to the second probabilities corresponding to multiple data enhancement strategies being less than the probabilities corresponding to the multiple data enhancement strategies.

[0099] Optionally, in response to the modal type of the original data of the corresponding modality being a visual sensor, the third enhancement unit determines that multiple data enhancement strategies include at least two of the following: motion blur, random noise, and random occlusion; in response to the modal type of the original data of the corresponding modality being a lidar sensor or a millimeter-wave radar sensor, the third enhancement unit determines that multiple data enhancement strategies include at least two of the following: random point cloud erasure, intensity attenuation, and random noise addition.

[0100] Optionally, the feature fusion module includes: a feature fusion unit, which is used to perform feature fusion on enhanced data of different modalities using a feature fusion model to obtain a perception result.

[0101] Optionally, the feature fusion model in the feature fusion unit is trained in the following manner: according to a preset learning rate, feature extraction modules of different modalities contained in the initial feature model are trained separately based on training data of multiple modalities, wherein the feature extraction modules of different modalities are used to extract features from enhanced data of the corresponding modalities; the preset learning rate is reduced to obtain a target learning rate; according to the target learning rate, the initial fusion model or the feature fusion module contained in the initial feature model is trained based on target data of different modalities to obtain a feature fusion model, wherein the feature fusion module is used to fuse features output by feature extraction modules of different modalities.

[0102] Optionally, the visual sensor in the data acquisition module includes: a first camera installed on the front of the work vehicle, multiple second cameras installed on the front and rear of the work vehicle, and multiple third cameras installed on the front of the work vehicle, wherein the focal length of the third camera is greater than the focal length of the second camera and smaller than the focal length of the first camera, the viewing angle of the third camera is greater than the viewing angle of the first camera and smaller than the viewing angle of the second camera, and the orientations of the multiple third cameras are different; and / or, the laser radar sensor includes: a first laser radar sensor installed on the front of the work vehicle, and multiple second laser radar sensors installed on the front and rear of the work vehicle, wherein the wiring harness of the first laser radar sensor is larger than the wiring harness of the second laser radar sensor, and the orientation of at least two second laser radar sensors installed on the front of the work vehicle is different from that of the first laser radar sensor; and / or, the millimeter wave radar sensor is installed on the front and rear of the work vehicle.

[0103] An embodiment of the present application further provides an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention when running.

[0104] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0105] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A vehicle perception method, characterized in that: include: Acquire raw data of multiple modes sensed by the working vehicle, wherein the raw data of the multiple modes are obtained by respectively collecting the surrounding environment of the working vehicle by multiple types of sensors installed on the working vehicle, and the multiple types of sensors include at least two of the following: a visual sensor, a laser radar sensor, and a millimeter wave radar sensor; Determine, according to the modality type of the original data of different modalities, a data enhancement strategy and a corresponding data enhancement probability corresponding to the original data of the corresponding modality, wherein the data enhancement probability is used to characterize the probability of using the corresponding data enhancement strategy for data enhancement for the original data of the corresponding modality; Based on the data enhancement strategies corresponding to the original data of the different modalities, the original data of the corresponding modalities are enhanced according to the corresponding data enhancement probabilities to obtain enhanced data of the different modalities; Feature fusion is performed on the enhanced data of different modalities to obtain a perception result.

2. The vehicle perception method according to claim 1, characterized in that: Based on the data enhancement strategy corresponding to the original data of the different modalities, the original data of the corresponding modalities are enhanced according to the corresponding data enhancement probability to obtain enhanced data of the different modalities, including: Randomly generate a first probability corresponding to the original data of the corresponding modality; In response to the first probability being greater than or equal to the corresponding data enhancement probability, based on a target data enhancement strategy corresponding to the original data of the corresponding modality, performing data enhancement on the original data of the corresponding modality to obtain enhanced data of the corresponding modality; In response to the first probability being less than the corresponding data enhancement probability, data enhancement of the original data of the corresponding modality based on the target data enhancement strategy corresponding to the original data of the corresponding modality is prohibited.

3. The vehicle perception method according to claim 1, characterized in that: In the case where the data enhancement strategy corresponding to the original data of the corresponding modality includes multiple data enhancement strategies, the corresponding data enhancement probability includes: the probabilities corresponding to the multiple data enhancement strategies; based on the data enhancement strategies corresponding to the original data of different modalities, the original data of the corresponding modality is data enhanced according to the corresponding data enhancement probability to obtain enhanced data of different modalities, including: Randomly generate second probabilities corresponding to the multiple data enhancement strategies; In response to a second probability corresponding to any one of the data enhancement strategies being greater than or equal to a probability corresponding to the preset data enhancement strategy, based on the data enhancement strategy, data enhancement is performed on the original data of the corresponding modality to obtain enhanced data of the corresponding modality; In response to the second probabilities corresponding to the multiple data enhancement strategies being smaller than the probabilities corresponding to the multiple data enhancement strategies, data enhancement of the original data of the corresponding modality based on the multiple data enhancement strategies is prohibited.

4. The vehicle perception method according to claim 3, characterized in that: In response to the modality type of the raw data of the corresponding modality being the visual sensor, determining that the multiple data enhancement strategies include at least two of the following: motion blur, random noise, and random occlusion; In response to the modality type of the original data of the corresponding modality being the lidar sensor or the millimeter wave radar sensor, determining that the multiple data enhancement strategies include at least two of the following: random point cloud erasing, intensity reduction, and random noise addition.

5. The vehicle sensing method according to any one of claims 1 to 4, characterized in that: The enhanced data of different modalities are subjected to feature fusion to obtain a perception result, including: The feature fusion model is used to perform feature fusion on the enhanced data of different modalities to obtain the perception result.

6. The vehicle sensing method according to claim 5, characterized in that: The feature fusion model is trained in the following way: According to a preset learning rate, the feature extraction modules of different modalities included in the initial feature model are trained based on the training data of multiple modalities, wherein the feature extraction modules of different modalities are used to extract features from the enhanced data of the corresponding modality; Reducing the preset learning rate to obtain a target learning rate; According to the target learning rate, the feature fusion module included in the initial fusion model or the initial feature model is trained based on the target data of the different modalities to obtain the feature fusion model, wherein the feature fusion module is used to fuse the features output by the feature extraction modules of the different modalities.

7. The vehicle sensing method according to any one of claims 1 to 6, characterized in that: The visual sensor comprises: a first camera installed on the front of the working vehicle, a plurality of second cameras installed on the front and rear of the working vehicle, and a plurality of third cameras installed on the front of the working vehicle, wherein the focal length of the third camera is greater than the focal length of the second camera and smaller than the focal length of the first camera, the viewing angle of the third camera is greater than the viewing angle of the first camera and smaller than the viewing angle of the second camera, and the plurality of third cameras have different orientations; and / or, The laser radar sensor comprises: a first laser radar sensor installed on the front of the working vehicle, and a plurality of second laser radar sensors installed on the front and rear of the working vehicle, wherein a wiring harness of the first laser radar sensor is larger than a wiring harness of the second laser radar sensor, and an orientation of at least two of the second laser radar sensors installed on the front of the working vehicle is different from an orientation of the first laser radar sensor; and / or, The millimeter wave radar sensors are installed on the front and rear of the working vehicle.

8. A vehicle sensing device, characterized in that: include: A data acquisition module, used to acquire raw data of multiple modes sensed by the working vehicle, wherein the raw data of the multiple modes are acquired by collecting the surrounding environment of the working vehicle by multiple types of sensors installed on the working vehicle, and the multiple types include at least two of the following: a visual sensor, a laser radar sensor, and a millimeter wave radar sensor; A strategy determination module, used to determine the data enhancement strategy and the corresponding data enhancement probability corresponding to the original data of the corresponding modality according to the modality type of the original data of different modalities, wherein the data enhancement probability is used to characterize the probability of using the corresponding data enhancement strategy for data enhancement for the original data of the corresponding modality; A data enhancement module, configured to perform data enhancement on the original data of the corresponding modality according to the corresponding data enhancement probability based on the data enhancement strategy corresponding to the original data of the different modalities, so as to obtain enhanced data of the different modalities; The feature fusion module is used to perform feature fusion on the enhanced data of different modalities to obtain a perception result.

9. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the vehicle perception method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the storage medium is located is controlled to execute the vehicle perception method described in any one of claims 1 to 7.

Images