End-to-end automatic driving system and vehicle

Abstract

The invention discloses an end-to-end automatic driving system and a vehicle, the system comprises a 4D environment sensing subsystem, the 4D environment sensing subsystem comprises a camera assembly, a 4D radar assembly and a microphone array, the camera assembly, the 4D radar assembly and the microphone array are arranged on the vehicle in a surrounding manner, the camera assembly comprises an infrared camera, a binocular camera and a fisheye camera, and the camera assembly comprises an infrared camera, a binocular camera and a fisheye camera; the 4D radar assembly comprises a 4D laser radar, a 4D millimeter wave radar, a FLASH laser radar and an ultrasonic radar and is used for generating 4D point cloud information of the surrounding environment of the vehicle, and the microphone array comprises a plurality of microphones and is used for collecting sound information of the surrounding environment of the vehicle; and the controller is in communication connection with the camera assembly, the 4D radar assembly and the microphone array, and is used for generating a vehicle control signal according to the image information, the 4D point cloud information and / or the sound information by using an AI large model, and controlling the vehicle according to the vehicle control signal.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to an end-to-end autonomous driving system and vehicle. Background Art

[0002] Autonomous driving primarily relies on conventional cameras and 3D millimeter-wave radar for 3D environmental monitoring. This type of environmental monitoring technology cannot address inherent sensor limitations, such as cameras' inability to identify objects at night and 3D millimeter-wave radar's low resolution, making it difficult to identify target types. Consequently, it cannot provide all-weather monitoring and early warning of the vehicle's surroundings. Summary of the Invention

[0003] The present invention aims to at least partially address one of the technical problems in the related art. To this end, one object of the present invention is to provide an end-to-end autonomous driving system that utilizes a 4D environmental perception subsystem to implement all-weather environmental monitoring and utilizes a large AI model to implement end-to-end autonomous driving based on the environmental information collected by the 4D environmental perception subsystem.

[0004] A second object of the present invention is to provide a vehicle.

[0005] To achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention proposes an end-to-end autonomous driving system, which includes: a 4D environmental perception subsystem, including a camera component, a 4D radar component and a microphone array, wherein the camera component, the 4D radar component and the microphone array are all arranged around the vehicle, the camera component includes an infrared camera, a binocular camera and a fisheye camera for collecting image information of the vehicle's surrounding environment, the 4D radar component includes a 4D lidar, a 4D millimeter-wave radar, a FLASH lidar and an ultrasonic radar for generating 4D point cloud information of the vehicle's surrounding environment, and the microphone array includes multiple microphones for collecting sound information of the vehicle's surrounding environment; a controller is communicatively connected to the camera component, the 4D radar component and the microphone array, and is used to use an AI large model to generate a vehicle control signal based on the image information, the 4D point cloud information and / or the sound information, and control the vehicle based on the vehicle control signal.

[0006] According to the end-to-end autonomous driving system of an embodiment of the present invention, a 4D environmental perception subsystem is used to achieve all-weather environmental monitoring, and an AI large model is used to achieve end-to-end autonomous driving based on the environmental information collected by the 4D environmental perception subsystem.

[0007] In addition, the end-to-end autonomous driving system proposed in the above embodiments of the present invention may also have the following additional technical features:

[0008] According to one embodiment of the present invention, the image information includes at least first image information, second image information and third image information; the number of the infrared camera is 1, which is arranged in the cab of the vehicle at the middle position of the upper edge of the front windshield of the vehicle, so as to collect the first image information of the front environment of the vehicle; the number of the binocular camera is 1, which is arranged in the cab of the vehicle at one side of the infrared camera, so as to collect the second image information of the front environment of the vehicle; the number of the fisheye cameras is 4, which are respectively arranged in the middle position of the front of the vehicle, the middle position of the rear of the vehicle, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror of the vehicle, so as to collect the third image information of the front, rear, left and right four directions of the vehicle environment.

[0009] According to one embodiment of the present invention, the 4D point cloud information includes at least first 4D point cloud information, second 4D point cloud information, first 3D point cloud information and ultrasonic echo signal; the number of the 4D laser radar is 1, which is arranged on the roof of the vehicle to generate the first 4D point cloud information of the environmental obstacles in front of the vehicle, wherein the first 4D point cloud information includes 3D point cloud information and instantaneous speed information; the number of the 4D millimeter radar is 5, which are respectively arranged in the middle and both sides of the rear of the front bumper and the two sides of the rear of the rear bumper of the vehicle to generate the vehicle Second 4D point cloud information of obstacles in the forward and rearward environments, wherein the second 4D point cloud information includes four-dimensional information of distance, speed, horizontal azimuth, and vertical height; three FLASH lidars, respectively disposed on the left front fender, right front fender, and rear of the vehicle, for generating first 3D point cloud information of obstacles in the left, right, and rearward environments of the vehicle; and twelve ultrasonic radars, uniformly disposed at the front and rear of the vehicle, for generating ultrasonic echo signals of obstacles within a preset distance in the forward and rearward environments of the vehicle.

[0010] According to one embodiment of the present invention, the microphone array includes four microphones, which are respectively arranged in the middle of the front bumper, the middle of the rear bumper, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror of the vehicle to collect sound information of the vehicle's surrounding environment.

[0011] According to one embodiment of the present invention, the controller is used to preprocess the image information, the 4D point cloud information and / or the sound information, use the AI ​​large model to extract environmental features in the preprocessed image information, the preprocessed 4D point cloud information and the preprocessed sound information, and generate the vehicle control signal based on the environmental features.

[0012] According to one embodiment of the present invention, the preprocessing includes at least time synchronization processing and noise removal processing.

[0013] According to one embodiment of the present invention, the infrared camera, the binocular camera and the fisheye camera are respectively connected to the controller through a gigabit multimedia serial link interface, the 4D lidar, the 4D millimeter wave radar and the FLASH lidar are respectively connected to the controller through gigabit Ethernet, and the ultrasonic radar is connected to the controller through a CAN bus.

[0014] According to one embodiment of the present invention, the controller supplies power to the infrared camera, the binocular camera and the fisheye camera, and the vehicle's power battery and / or storage battery supplies power to the 4D lidar, the 4D millimeter-wave radar, the FLASH lidar, the ultrasonic radar and the microphone.

[0015] According to one embodiment of the present invention, after the vehicle is powered on, the controller establishes a communication connection with the infrared camera, the binocular camera, the fisheye camera, the 4D lidar, the 4D millimeter-wave radar, the FLASH lidar, the ultrasonic radar and the microphone through a handshake protocol.

[0016] To achieve the above-mentioned objectives, a second embodiment of the present invention proposes a vehicle, comprising an end-to-end autonomous driving system as proposed in the first embodiment of the present invention.

[0017] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an end-to-end autonomous driving system according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the installation positions of a camera, radar, and microphone according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of an end-to-end AI large model according to an embodiment of the present invention;

[0021] Figure 4 is a structural block diagram of a controller according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a vehicle according to an embodiment of the present invention.

[0023] The accompanying drawings illustrate:

[0024] 1. Infrared camera; 2. Binocular camera; 3. Fisheye camera; 4. 4D laser radar; 5. 4D millimeter-wave radar; 6. FLASH laser radar; 7. Ultrasonic radar; 8. Microphone. DETAILED DESCRIPTION

[0025] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0026] The end-to-end autonomous driving system and vehicle according to the embodiments of the present invention are described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] Figure 1 FIG is a schematic diagram of an end-to-end autonomous driving system according to an embodiment of the present invention. Figure 1 As shown, the end-to-end autonomous driving system may include:

[0028] A 4D environment perception subsystem, comprising at least a camera assembly, a 4D radar assembly, and a microphone array, all of which are arranged around the vehicle. The camera assembly includes an infrared camera 1, a binocular camera 2, and a fisheye camera 3 for collecting image information of the vehicle's surroundings. The 4D radar assembly includes a 4D laser radar 4, a 4D millimeter-wave radar 5, a FLASH laser radar 6, and an ultrasonic radar 7 for generating 4D point cloud information of the vehicle's surroundings. The microphone array includes multiple microphones 8 for collecting sound information of the vehicle's surroundings.

[0029] The controller is communicatively connected to the camera assembly, the 4D radar assembly, and the microphone array, and is used to generate vehicle control signals based on image information, 4D point cloud information, and / or sound information using the AI ​​large model, and to control the vehicle based on the vehicle control signals.

[0030] The 4D environmental perception subsystem in this embodiment of the present invention utilizes a fusion of camera components, 4D radar components, and microphone arrays to address the need for reliable 4D environmental perception in autonomous driving systems. This system enables all-weather monitoring of the surrounding and ground 4D environment. The controller utilizes a large AI model to perceive the real environment based on image information, 4D point cloud information, and / or sound information, generating vehicle control signals and controlling the vehicle accordingly.

[0031] Specifically, the camera assembly includes an infrared camera 1, a binocular camera 2, and a fisheye camera 3. The infrared camera 1, the binocular camera 2, and the fisheye camera 3 are arranged around the vehicle to collect image information of the vehicle's surrounding environment.

[0032] The 4D radar assembly includes a 4D laser radar 4, a 4D millimeter-wave radar 5, a flash laser radar 6, and an ultrasonic radar 7, all of which are arranged around the vehicle. Each radar's transmitter emits high-frequency electromagnetic waves of a specific frequency around the vehicle body. The receiver receives echo signals reflected by various targets (obstacles) around the vehicle. The signal processor generates 4D point cloud information about the vehicle's surroundings based on the echo signals.

[0033] The microphone array includes a plurality of microphones 8 arranged around the vehicle to collect sound information from the vehicle's surrounding environment.

[0034] The controller captures image information of the vehicle's surroundings from infrared camera 1, binocular camera 2, and fisheye camera 3; 4D point cloud information of the vehicle's surroundings generated by 4D LiDAR 4, 4D millimeter-wave radar 5, FLASH LiDAR 6, and ultrasonic radar 7; and sound information from the vehicle's surroundings collected by a microphone array. The controller uses a large AI model to perceive the real environment around the vehicle, which then generates vehicle control signals based on this perception. The controller controls the vehicle based on these control signals, ensuring reliable and safe all-weather travel along the planned end-to-end path.

[0035] The end-to-end autonomous driving system of an embodiment of the present invention senses the surrounding 4D environment, ground stationary targets, space targets, dynamic targets, sounds and other targets during the vehicle's driving process through an infrared camera 1, a forward-looking binocular camera 2, a fisheye camera 3, a 4D lidar 4, a 4D millimeter-wave radar 5, a FLASH lidar 6, an ultrasonic radar 7 and a microphone array installed at different positions on the vehicle body, and transmits these targets in the form of images, point clouds and sound information to the autonomous driving AI big model for processing, analysis and extraction of real environmental information. Based on the results of the AI ​​big model analysis, it outputs reliable and safe vehicle control signals to control the vehicle to travel reliably and safely in all weather conditions according to the planned end-to-end path.

[0036] In one embodiment of the present invention, the image information includes at least first image information, second image information and third image information; the number of infrared cameras 1 is one, which is set in the vehicle's cab at the middle position of the upper edge of the vehicle's front windshield to collect first image information of the vehicle's forward environment; the number of binocular cameras 2 is one, which is set in the vehicle's cab at one side of the infrared camera 1 to collect second image information of the vehicle's forward environment; the number of fisheye cameras 3 is four, which are respectively set in the middle position of the front of the vehicle, the middle position of the rear of the vehicle, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror to collect third image information of the vehicle's front, rear, left and right four directions of the environment, see Figure 2 .

[0037] Specifically, infrared camera 1 is a camera device equipped with infrared (IR) technology. By emitting or receiving invisible infrared light (wavelength of approximately 760nm to 1mm), it overcomes the limitations of visible light and enables image capture at night or in dark environments. It is primarily used to capture images in low-light or no-light environments and supports specific biometric recognition functions. Infrared camera 1 is installed in the vehicle's cab, located on the side of the upper edge of the front windshield, and can capture first image information (infrared image information) of the environment forward of the vehicle in low-light or no-light environments.

[0038] Specifically, the two cameras of the binocular camera 2 are equivalent to the left and right eyes of a person. By calculating the parallax of the left and right views, that is, the position deviation of the same object in the left and right images, combined with the lens spacing (baseline distance) and focal length, the three-dimensional coordinates (X, Y, Z axis position) of the object are calculated using the triangulation principle, thereby achieving 1. Distance measurement: Accurately calculate the actual distance between the camera and the object (such as judging the distance to the vehicle in front during autonomous driving). 2. Three-dimensional modeling: restore the three-dimensional shape and spatial structure of the object. The forward-looking binocular camera 2 in the embodiment of the present invention is arranged in the cab of the vehicle on the side of the infrared camera 1 to collect second image information of the vehicle's forward environment.

[0039] The infrared camera 1 and the binocular camera 2 in the embodiment of the present invention cooperate to achieve image acquisition of the vehicle's forward environment in good light conditions as well as in low light or no light conditions.

[0040] In the embodiment of the present invention, fisheye cameras 3 are respectively arranged at the middle position of the front of the vehicle, the middle position of the rear of the vehicle, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror to collect third image information of the environment in the front, rear, left and right directions of the vehicle, and obtain image information of the environment around the vehicle.

[0041] In one embodiment of the present invention, the 4D point cloud information includes at least the first 4D point cloud information, the second 4D point cloud information, the first 3D point cloud information and the ultrasonic echo signal; the number of the 4D laser radar 44 is one, which is set on the roof of the vehicle to generate the first 4D point cloud information of the environmental obstacles in front of the vehicle, wherein the first 4D point cloud information includes 3D point cloud information and instantaneous speed information; the number of the 4D millimeter wave radar 55 is five, which are respectively set in the middle and both sides of the rear of the front bumper and the both sides of the rear of the rear bumper of the vehicle to generate the vehicle forward. and second 4D point cloud information of obstacles in the rearward environment, wherein the second 4D point cloud information includes four-dimensional information of distance, speed, horizontal azimuth, and vertical height; the number of FLASH laser radars 66 is 3, which are respectively arranged on the left front fender, right front fender, and rear of the vehicle to generate first 3D point cloud information of obstacles in the left, right, and rearward environment of the vehicle; the number of ultrasonic radars 77 is 12, which are respectively evenly arranged at the front and rear of the vehicle to generate ultrasonic echo signals of obstacles within a preset distance in the front and rear environments of the vehicle, see Figure 2 .

[0042] Specifically, 4D LiDAR 4 is an advanced sensor that adds a speed perception dimension to traditional 3D LiDAR (distance, horizontal angle, vertical angle). It can simultaneously provide the three-dimensional spatial coordinates (X / Y / Z) and instantaneous speed information of the target object, forming high-precision four-dimensional point cloud data for dynamic environments. After integrating speed information, the target's motion trajectory (such as vehicle lane changes and pedestrians crossing) can be predicted in real time, improving decision-making safety. 4D LiDAR 4 is installed on the roof or front windshield of the vehicle to generate the first 4D point cloud information of obstacles in the vehicle's forward environment.

[0043] The 4D laser radar 4 in the embodiment of the present invention uses FMCW (Frequency-Modulated Continuous Wave) technology to directly measure the relative speed between the target and the sensor by modulating the laser frequency and analyzing the frequency shift (Doppler effect) of the reflected wave.

[0044] The 4D laser radar 4 in the embodiment of the present invention improves the point cloud density through an algorithm, and its resolution can reach 20 times that of traditional laser radar (equivalent to a "thousand-line" effect), realizing centimeter-level detail capture (such as road sign text, pedestrian posture) and super-resolution imaging.

[0045] The FMCW technology used by the 4D laser radar 4 in the embodiment of the present invention avoids mechanical scanning delay and can still produce clear images without motion blur in dynamic scenes.

[0046] The wavelength characteristics of the laser emitted by the 4D laser radar 4 in the embodiment of the present invention make it less affected by rain and fog than the millimeter wave radar, and the FMCW technology can distinguish environmental clutter, reduce the false detection rate, and has strong anti-interference ability.

[0047] Specifically, the 4D millimeter-wave radar 5 is an upgraded version of the traditional millimeter-wave radar. By adding vertical height perception capabilities, it realizes the intelligent sensor of four-dimensional data detection of distance, speed, horizontal azimuth, and vertical height. It can identify three-dimensional spatial targets such as overpasses, traffic lights, and low obstacles.

[0048] The five 4D millimeter-wave radars 5 in the embodiment of the present invention are respectively arranged in the middle and both sides of the rear of the front bumper and on both sides of the rear of the rear bumper of the vehicle to realize four-dimensional data detection of distance, speed, horizontal azimuth and vertical height.

[0049] Specifically, FLASH LiDAR 6 (Flash LiDAR) is a pure solid-state LiDAR technology. Its core feature is that it does not require mechanical or electronic scanning. It directly illuminates the entire detection area through a single laser pulse and receives the reflected signal to achieve instantaneous three-dimensional imaging.

[0050] The three FLASH laser radars 6 in the embodiment of the present invention are set on the left front fender, right front fender and rear of the vehicle, obtain left and right and rear environmental information and output it in the form of 3D point cloud information, generating the first 3D point cloud information of environmental obstacles to the left, right and rear of the vehicle.

[0051] Specifically, the ultrasonic radar 7 is a sensor that uses ultrasonic waves (sound waves with a frequency higher than 20kHz) to measure distance and perceive the environment. Its core function is to calculate the distance, position and motion state of obstacles by transmitting and receiving ultrasonic signals.

[0052] The number of the 12 ultrasonic radars 7 in the embodiment of the present invention is evenly arranged at the front and rear of the vehicle. Figure 2 The front of the vehicle is evenly equipped with 6 ultrasonic radars 7, and the rear of the vehicle is evenly equipped with 6 ultrasonic radars 7. The 12 ultrasonic radars 7 are used to generate ultrasonic echo signals of obstacles within a preset distance in the front and rear directions of the vehicle.

[0053] In the embodiment of the present invention, a fusion technology of 4D laser radar 4, 4D millimeter wave radar 5, FLASH laser radar 6 and ultrasonic radar 7 is adopted to generate 4D point cloud information of the vehicle's surrounding environment.

[0054] The present invention proposes a 4D environmental perception subsystem that uses 4D lidars (long-range lidar and blind spot-filling lidar), which greatly enhances the reliability and availability of environmental perception and meets the scenario application of all-weather end-to-end AI large models for autonomous driving.

[0055] In one embodiment of the present invention, the microphone array includes four microphones 8, which are respectively arranged in the middle of the front bumper, the middle of the rear bumper, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror of the vehicle to collect sound information of the vehicle's surrounding environment.

[0056] Specifically, the microphone array (MIC) captures ambient acoustic signals to achieve multi-dimensional perception enhancement. The four microphones (MICs) in this embodiment are located in the center of the vehicle's front and rear bumpers, on one side of the left and right exterior mirrors, capturing sound information from the vehicle's surroundings.

[0057] Embodiments of the present invention can identify emergency vehicle sirens based on sound information collected by a microphone array. Specifically, the MIC array uses voiceprint recognition algorithms (such as Mel-Frequency Cepstral Coefficient (MFCC) feature extraction) to identify the frequency band characteristics of special vehicles such as ambulances and fire trucks, enabling real-time siren detection.

[0058] The present invention uses microphone arrays to collect sound information, enabling early warning of dangerous situations. Specifically, a deep neural network (DNN) can be used to classify ambient sounds, such as tire pops, pedestrian shouts, and collisions, to detect unusual acoustic events.

[0059] The embodiment of the present invention can realize voice interaction and in-vehicle monitoring based on the sound information collected by the microphone array. For example, by detecting the sound of a baby crying or an object falling, a safety alarm is triggered to realize abnormal monitoring in the cabin.

[0060] Embodiments of the present invention can implement acoustic SLAM (Simultaneous Localization and Mapping) based on sound information collected by a microphone array. Specifically, by combining LiDAR and MIC data, dynamic sound sources (such as construction warning sounds) can be located using Time Difference of Arrival (TDOA).

[0061] For the end-to-end autonomous driving vehicle based on the present invention, first of all, all the required sensors must be available and installed at the required vehicle body positions.

[0062] In one embodiment of the present invention, Figure 3 As shown, the controller is used to preprocess image information, 4D point cloud information and / or sound information, use the AI ​​large model to extract environmental features in the preprocessed image information, preprocessed 4D point cloud information and preprocessed sound information, and generate vehicle control signals based on the environmental features.

[0063] Specifically, the controller performs pre-processing such as time synchronization and denoising on the image information, 4D point cloud information and / or sound information, and uses the AI ​​big model to extract environmental features from the pre-processed image information, pre-processed 4D point cloud information and pre-processed sound information. The AI ​​big model generates vehicle control signals based on the environmental features.

[0064] In one embodiment of the present invention, the preprocessing includes at least time synchronization processing and noise removal processing.

[0065] Different sensors, radars, and microphones in the 4D environment perception subsystem of the embodiment of the present invention achieve information synchronization based on the time reference of the controller.

[0066] In one embodiment of the present invention, the infrared camera, binocular camera 2 and fisheye camera 3 are respectively connected to the controller through a gigabit multimedia serial link interface, the 4D laser radar 4, 4D millimeter wave radar 5, and FLASH laser radar 6 are respectively connected to the controller through gigabit Ethernet, and the ultrasonic radar 7 is connected to the controller through a CAN bus.

[0067] Specifically, the infrared camera, forward-looking driving camera, and fisheye camera 3 output image information via the GMSL (Gigabit Multimedia Serial Link) interface and send it to the controller. The 4D LiDAR 4, 4D millimeter-wave radar 5, and FLASH LiDAR 6 output 4D point cloud information via Gigabit Ethernet and send it to the controller. The ultrasonic radar 7 outputs close-range target point cloud information via CAN FD (Flexible Data Rate CAN bus).

[0068] In one embodiment of the present invention, the controller supplies power to the infrared camera, binocular camera 2 and fisheye camera 3, and the vehicle's power battery and / or storage battery supplies power to the 4D lidar 4, 4D millimeter wave radar 5, FLASH lidar 6, ultrasonic radar 7 and microphone 8.

[0069] Specifically, the infrared camera, binocular camera 2 and fisheye camera 3 are powered by the controller, and the 4D laser radar 4, 4D millimeter wave radar 5, FLASH laser radar 6, ultrasonic radar 7 and microphone 8 are bidirectionally powered by the vehicle's power battery and storage battery to ensure the reliability of the power supply system.

[0070] In one embodiment of the present invention, after the vehicle is powered on, the controller establishes a communication connection with the infrared camera, binocular camera 2, fisheye camera 3, 4D lidar 4, 4D millimeter wave radar 5, FLASH lidar 6, ultrasonic radar 7 and microphone 8 through a handshake protocol.

[0071] After the vehicle is powered on, all sensors, radars, and microphones 8 begin operating. A handshake protocol is performed between all sensors, radars, microphones 8, and the controller to ensure information security. After passing security verification, the perceived environmental information (image information, sound information) is output in real time (within 20ms). The output format of the overall environmental perception information is predefined, and the controller outputs it uniformly and synchronizes the output time to achieve synchronization of environmental information.

[0072] The controller acquires real-time 4D environmental information from the 4D environmental perception subsystem and, through AI model learning and intelligent driving task requirements, achieves end-to-end control. Following the globally planned path, the controller controls the vehicle's local trajectory through steering, braking, lighting, and voice commands, and automatically completes parking upon arrival at the destination.

[0073] The vehicle uses the above-mentioned end-to-end autonomous driving system. During end-to-end driving, the 4D environmental perception subsystem can ensure application in all weather and all scenarios, daytime / nighttime, rainy / foggy / snowy / sunny. The infrared camera in the forward-facing camera can provide reliable environmental information (such as lane markings, pedestrians, vehicles, etc.) both day and night. The binocular camera 2, as a redundancy for the infrared camera, can also reliably identify traffic lights and traffic signs. In rainy and foggy weather, the 4D millimeter-wave radar 5 and 4D lidar 4 provide additional redundancy to ensure reliable environmental perception of the driving direction and achieve reliable environmental perception capabilities. Reliable environmental perception is achieved on the left, right, and rearward sides through the fisheye camera 3, 4D millimeter-wave radar 5, and FLASH lidar 6. During parking, the fisheye camera 3, ultrasonic radar 7, and FLASH lidar 6 provide reliable environmental perception.

[0074] If a sensor of any 4D environmental perception subsystem in the embodiments of the present invention fails, the other surviving sensors will provide environmental perception capabilities, greatly improving the robustness, reliability and availability of the system, and providing strong support for achieving fully autonomous driving.

[0075] The present invention provides an automatic driving method.

[0076] The autonomous driving method in an embodiment of the present invention is based on the above-mentioned end-to-end autonomous driving system, and the method includes:

[0077] Acquire image information, 4D point cloud information and / or sound information of the vehicle's surroundings;

[0078] Using the AI ​​big model, a vehicle control signal is generated based on image information, 4D point cloud information and / or sound information, and the vehicle is controlled based on the vehicle control signal.

[0079] The present invention provides a computer-readable storage medium.

[0080] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the automatic driving method as described above is implemented.

[0081] The invention provides a controller.

[0082] In this embodiment, the controller 500 includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the automatic driving method as described above is implemented.

[0083] Figure 4 4 is a structural block diagram of a controller according to an embodiment of the present invention.

[0084] like Figure 4 As shown, controller 500 includes a processor 501 and a memory 503. Processor 501 and memory 503 are connected, for example, via a bus 502. Optionally, controller 500 may further include a transceiver 504. It should be noted that in practical applications, the number of transceivers 504 is not limited to one, and the structure of controller 500 does not constitute a limitation on the embodiments of the present invention.

[0085] The processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0086] The bus 502 may include a path for transmitting information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0087] Memory 503 is used to store a computer program corresponding to the autonomous driving method of the above-mentioned embodiment of the present invention, and the computer program is controlled and executed by processor 501. Processor 501 is used to execute the computer program stored in memory 503 to implement the content shown in the above-mentioned method embodiment.

[0088] Among them, the controller 500 includes but is not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0089] The present invention provides a vehicle.

[0090] Figure 5 FIG is a schematic diagram of a vehicle according to an embodiment of the present invention. Figure 5 As shown, the vehicle may include an end-to-end autonomous driving system as described above.

[0091] The vehicle of the embodiment of the present invention adopts the above-mentioned end-to-end autonomous driving system. During the end-to-end driving process, the 4D environmental perception subsystem can ensure its application in all weather scenarios at night / day, rain / fog / snow / sunny. The infrared camera in the forward-facing camera can provide reliable environmental information (such as lane lines, pedestrians, vehicles, etc.) both during the day and at night. The binocular camera serves as a redundancy for the infrared camera and can also reliably identify traffic lights and traffic signs. In rainy and foggy weather, the 4D millimeter-wave radar and 4D lidar are used as redundancy to ensure reliable environmental perception of the driving direction and achieve reliable environmental perception capabilities. Reliable environmental perception is achieved on the left and right sides and in the rear through fisheye cameras, 4D millimeter-wave radars, and FLASH lidars. During parking, the fisheye camera, ultrasonic radar, and FLASH lidar provide reliable environmental perception.

[0092] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0093] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0094] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0095] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0097] In the present invention, unless otherwise specified or limited, the terms "installed," "connected," "connect," "fixed," etc. should be understood in a broad sense. For example, they can refer to fixed connection, detachable connection, or integration; mechanical connection, electrical connection; direct connection, or indirect connection through an intermediate medium; internal communication between two components, or interaction between two components, unless otherwise specified. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0098] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0099] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An end-to-end autonomous driving system, characterized in that: The system comprises: a 4D environmental perception subsystem, comprising a camera assembly, a 4D radar assembly, and a microphone array, wherein the camera assembly, the 4D radar assembly, and the microphone array are all disposed around the vehicle; the camera assembly comprises an infrared camera, a binocular camera, and a fisheye camera for collecting image information of the vehicle's surroundings; the 4D radar assembly comprises a 4D lidar, a 4D millimeter-wave radar, a FLASH lidar, and an ultrasonic radar for generating 4D point cloud information of the vehicle's surroundings; and the microphone array comprises multiple microphones for collecting sound information of the vehicle's surroundings; A controller is communicatively connected to the camera assembly, the 4D radar assembly, and the microphone array, and is used to generate a vehicle control signal based on the image information, the 4D point cloud information, and / or the sound information using an AI large model, and to control the vehicle based on the vehicle control signal.

2. The end-to-end autonomous driving system according to claim 1, characterized in that: The image information includes at least first image information, second image information and third image information; The infrared camera is one and is disposed in the cab of the vehicle at the middle position of the upper edge of the front windshield of the vehicle, for collecting first image information of the environment in front of the vehicle; The number of the binocular camera is one, which is arranged in the cab of the vehicle on one side of the infrared camera, and is used to collect second image information of the environment forward of the vehicle; There are four fisheye cameras, which are respectively arranged at the middle position of the front of the vehicle, the middle position of the rear of the vehicle, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror, for collecting third image information of the vehicle's front, rear, left and right directions.

3. The end-to-end autonomous driving system according to claim 1, characterized in that: The 4D point cloud information includes at least first 4D point cloud information, second 4D point cloud information, first 3D point cloud information and an ultrasonic echo signal; The number of the 4D laser radar is one, which is arranged on the roof of the vehicle and is used to generate first 4D point cloud information of environmental obstacles in front of the vehicle, wherein the first 4D point cloud information includes 3D point cloud information and instantaneous speed information; The number of the 4D millimeter radars is 5, which are respectively arranged in the middle and both sides of the rear part of the front bumper of the vehicle and on both sides of the rear part of the rear bumper, so as to generate second 4D point cloud information of environmental obstacles in front of and behind the vehicle, wherein the second 4D point cloud information includes four-dimensional information of distance, speed, horizontal azimuth, and vertical height; There are three FLASH laser radars, which are respectively arranged on the left front fender, right front fender and rear of the vehicle, for generating first 3D point cloud information of environmental obstacles to the left, right and rear of the vehicle; The number of the ultrasonic radars is 12, which are evenly arranged at the front and rear of the vehicle, and are used to generate ultrasonic echo signals of obstacles within a preset distance in the front and rear environments of the vehicle.

4. The end-to-end autonomous driving system according to claim 1, wherein: The microphone array includes four microphones, which are respectively arranged in the middle of the front bumper, the middle of the rear bumper, one side of the left exterior rearview mirror, and one side of the right exterior rearview mirror of the vehicle to collect sound information of the vehicle's surrounding environment.

5. The end-to-end autonomous driving system according to any one of claims 1 to 4, characterized in that: The controller is used to preprocess the image information, the 4D point cloud information and / or the sound information, use the AI ​​large model to extract environmental features in the preprocessed image information, the preprocessed 4D point cloud information and the preprocessed sound information, and generate the vehicle control signal based on the environmental features.

6. The end-to-end autonomous driving system according to claim 5, characterized in that: The preprocessing includes at least time synchronization processing and noise removal processing.

7. The end-to-end autonomous driving system according to any one of claims 1 to 4, characterized in that: The infrared camera, the binocular camera and the fisheye camera are respectively connected to the controller through a gigabit multimedia serial link interface, the 4D laser radar, the 4D millimeter wave radar and the FLASH laser radar are respectively connected to the controller through gigabit Ethernet, and the ultrasonic radar is connected to the controller through a CAN bus.

8. The end-to-end autonomous driving system according to any one of claims 1 to 4, characterized in that: The controller supplies power to the infrared camera, the binocular camera and the fisheye camera, and the vehicle's power battery and / or storage battery supplies power to the 4D laser radar, the 4D millimeter-wave radar, the FLASH laser radar, the ultrasonic radar and the microphone.

9. The end-to-end autonomous driving system according to any one of claims 1 to 4, characterized in that: After the vehicle is powered on, the controller establishes a communication connection with the infrared camera, the binocular camera, the fisheye camera, the 4D laser radar, the 4D millimeter wave radar, the FLASH laser radar, the ultrasonic radar and the microphone through a handshake protocol.

10. A vehicle, characterized in that: An end-to-end autonomous driving system comprising any one of claims 1-9.

Images