Transparent chassis display system, method, device and storage medium

Abstract

The invention discloses a transparent chassis display system, method and equipment and a storage medium, and belongs to the technical field of vehicles. In the system, an unmanned aerial vehicle scans a road below a vehicle chassis to respectively obtain a scanning image, first point cloud data and second point cloud data, then a vehicle-mounted terminal generates a transparent chassis view based on the scanning image, the first point cloud data and the second point cloud data, and the transparent chassis view is displayed through a head-up display system. Therefore, the system obtains the road information below the chassis through the unmanned aerial vehicle, compared with a traditional fixedly-installed camera, the unmanned aerial vehicle can switch the posture at any time so as to adjust the shooting angle of the camera, the unmanned aerial vehicle cannot shake along with bumping of the vehicle, image distortion can be avoided, and therefore the road condition below the chassis can be accurately reflected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a transparent chassis display system, method, device and storage medium. BACKGROUND

[0002] With the development of vehicle intelligence, users pay more and more attention to the safety during vehicle driving. Among them, eliminating the blind area of the chassis view is an important safety requirement, and the transparent chassis technology can enable the driver to intuitively perceive the road surface condition under the vehicle chassis.

[0003] In related technologies, the transparent chassis technology mainly relies on the camera fixed on the vehicle chassis or the side, captures the picture under the chassis through the camera, and transmits the picture to the central control display screen, so that the user can intuitively determine the road surface condition under the chassis.

[0004] But the camera is fixed on the vehicle chassis or the side, and cannot dynamically adjust its shooting angle. For complex terrain, in the process of vehicle driving, the camera may vibrate, in which case the image captured by the camera may be distorted, resulting in the inability to accurately reflect the road surface condition under the chassis. SUMMARY

[0005] The embodiments of the present application provide a transparent chassis display system, method, device and storage medium, which can accurately reflect the road surface condition under the chassis. The technical solution is as follows: On the one hand, a transparent chassis display system is provided, which comprises a drone, a vehicle terminal and a head-up display system, wherein the drone is configured with a millimeter wave radar, a laser radar and a camera; The vehicle terminal is configured to send a scanning instruction to the drone through a vehicle display screen; The drone is configured to scan the road under the vehicle chassis through the camera based on the scanning instruction to obtain a scanning image, emit an electromagnetic wave signal through the millimeter wave radar, generate first point cloud data based on the electromagnetic wave signal, emit a laser signal through the laser radar, generate second point cloud data based on the laser signal, and send the scanning image, the first point cloud data and the second point cloud data to the vehicle terminal; The vehicle terminal is further configured to generate a transparent chassis view based on the scanning image, the first point cloud data and the second point cloud data, send the transparent chassis view to the head-up display system, and the transparent chassis view is used to reflect the road information under the vehicle chassis; The head-up display system is configured to display the transparent chassis view.

[0006] In a possible implementation, the vehicle-mounted terminal is further configured to fuse the first point cloud data and the second point cloud data to obtain third point cloud data; and generate the transparent chassis view based on the third point cloud data and the scan image.

[0007] In another possible implementation, the vehicle-mounted terminal is further configured to establish a four-dimensional coordinate system, the four-dimensional coordinate system comprising a time dimension and a space dimension; synchronize the third point cloud data and the scan image based on a time stamp; convert the synchronized third point cloud data and the scan image into the four-dimensional coordinate system; and fuse the third point cloud data in the four-dimensional coordinate system and the scan image by using a space-time fusion algorithm to generate the transparent chassis view with a time label.

[0008] In another possible implementation, the system further comprises a dock station controller, the dock station controller and the unmanned aerial vehicle are both arranged in a roof dock station, and the roof dock station further comprises a locking mechanism and an electromagnetic ejection module, the locking mechanism is configured to lock the unmanned aerial vehicle, and the electromagnetic ejection module is configured to eject the unmanned aerial vehicle. The vehicle-mounted terminal is further configured to send an outbound instruction to the dock station controller through the vehicle-mounted display screen. The dock station controller is configured to unlock the locking mechanism based on the outbound instruction, and eject the unmanned aerial vehicle through the electromagnetic ejection module, so that the unmanned aerial vehicle leaves the roof dock station.

[0009] In another possible implementation, the system further comprises an electronic auxiliary steering controller. The vehicle-mounted terminal is further configured to send a first early warning instruction to the electronic auxiliary steering controller when detecting that a road convexity height is greater than a preset height based on the transparent chassis view. The electronic auxiliary steering controller is configured to control a steering wheel to vibrate in a first vibration mode for vibration early warning based on the first early warning instruction. The vehicle-mounted terminal is further configured to send a second early warning instruction to the electronic auxiliary steering controller when detecting that a road concave depth is greater than a preset depth based on the transparent chassis view. The electronic auxiliary steering controller is further configured to control the steering wheel to vibrate in a second vibration mode for vibration early warning based on the second early warning instruction.

[0010] In another possible implementation, the scan instruction carries a vehicle speed. The unmanned aerial vehicle is configured to determine a terrain attenuation coefficient, determine a flight speed of the unmanned aerial vehicle based on the vehicle speed and the terrain attenuation coefficient, and perform the scanning of the road under the vehicle chassis by the camera to obtain the scan image based on the flight speed of the unmanned aerial vehicle.

[0011] In another aspect, a transparent chassis display method is provided, the method comprising: The vehicle terminal sends a scanning instruction to the unmanned aerial vehicle through a vehicle display screen. The unmanned aerial vehicle scans the road under the vehicle chassis by a camera to obtain a scan image based on the scanning instruction, generates first point cloud data based on an electromagnetic wave signal emitted by a millimeter wave radar, generates second point cloud data based on a laser signal emitted by a laser radar, and sends the scan image, the first point cloud data, and the second point cloud data to the vehicle terminal. The vehicle terminal generates a transparent chassis view based on the scan image, the first point cloud data, and the second point cloud data, and sends the transparent chassis view to a head-up display system, wherein the transparent chassis view is used to reflect road information under the vehicle chassis. The head-up display system displays the transparent chassis view.

[0012] In one possible implementation, the vehicle terminal generates a transparent chassis view based on the scan image, the first point cloud data, and the second point cloud data, comprising: The vehicle terminal fuses the first point cloud data and the second point cloud data to obtain third point cloud data. The transparent chassis view is generated based on the third point cloud data and the scan image.

[0013] In another possible implementation, the vehicle terminal generates the transparent chassis view based on the third point cloud data and the scan image, comprising: The vehicle terminal establishes a four-dimensional coordinate system, wherein the four-dimensional coordinate system comprises a time dimension and a space dimension. The third point cloud data is synchronized with the scan image based on a time stamp. The synchronized third point cloud data and the scan image are converted into the four-dimensional coordinate system. The third point cloud data in the four-dimensional coordinate system and the scan image are fused by a space-time fusion algorithm to generate the transparent chassis view with a time label.

[0014] In another possible implementation, the method further comprises: The vehicle terminal sends an outbound instruction to the docking station controller through the vehicle display screen. The docking station controller unlocks the locking mechanism based on the outbound instruction, and ejects the UAV through the electromagnetic ejection module, so that the UAV leaves the roof docking station.

[0015] In another possible implementation, the method further includes: When the vehicle terminal detects that the road convexity height is greater than a preset height based on the transparent chassis view, the vehicle terminal sends a first early warning instruction to an electronic auxiliary steering controller; The electronic auxiliary steering controller controls the steering wheel to vibrate in a first vibration mode for vibration early warning based on the first early warning instruction; or, When the vehicle terminal detects that the road concave depth is greater than a preset depth based on the transparent chassis view, the vehicle terminal sends a second early warning instruction to the electronic auxiliary steering controller; The electronic auxiliary steering controller controls the steering wheel to vibrate in a second vibration mode for vibration early warning based on the second early warning instruction.

[0016] In another possible implementation, the scanning instruction carries a vehicle speed; The UAV scans the road below the vehicle chassis based on the scanning instruction through the camera to obtain a scanning image, including: The UAV determines a terrain attenuation coefficient, determines a flight speed of the UAV based on the vehicle speed and the terrain attenuation coefficient, and scans the road below the vehicle chassis based on the flight speed of the UAV through the camera to obtain the scanning image.

[0017] In another aspect, an electronic device is provided, which includes a processor and a memory, the memory storing at least one program code, the at least one program code being loaded and executed by the processor to implement the transparent chassis display method of any of the above vehicle terminal or UAV.

[0018] In another aspect, a computer-readable storage medium is provided, which stores at least one program code, the at least one program code being loaded and executed by a processor to implement the transparent chassis display method of any of the above.

[0019] In another aspect, a computer program product is provided, which stores at least one program code, the at least one program code being loaded and executed by a processor to implement the transparent chassis display method of any of the above.

[0020] The embodiment of the present application provides a transparent chassis display system, a UAV scans a road below a chassis of a vehicle in the system, respectively obtains a scanning image, first point cloud data and second point cloud data, and then a vehicle terminal generates a transparent chassis view based on the scanning image, the first point cloud data and the second point cloud data, and displays the transparent chassis view through a heads-up display system. Therefore, the system obtains the road information below the chassis through the UAV, compared with a traditional fixedly installed camera, the UAV can switch the posture at any time to adjust the shooting angle of the camera, and cannot shake with the bumping of the vehicle, so that image distortion can be avoided, and the road condition below the chassis can be accurately reflected. Moreover, when the transparent chassis view is generated, the point cloud data of the millimeter wave radar and the laser radar is combined on the basis of the scanning image, so that the relative spatial relationship between the road and the chassis can be more accurately obtained, and the road condition below the chassis can be further accurately reflected.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a schematic diagram of a transparent chassis display system provided by the embodiment of the present application; Figure 2 is a flowchart of a transparent chassis display method provided by the embodiment of the present application; Figure 3 is a structural block diagram of a vehicle terminal provided by the embodiment of the present application; Figure 4 is a structural block diagram of a UAV provided by the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the technical solutions and advantages of the present application clearer, the following further describes the embodiments of the present application in detail.

[0024] The terms "first", "second", "third", and "fourth" and the like in the specification and claims of the present application and the drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or devices is not limited to the listed steps or devices, but can optionally include other steps or devices not listed, or can optionally include other steps or devices inherent to the process, method, product or device.

[0025] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the scanning image and point cloud data involved in the present application are obtained under sufficient authorization.

[0026] Figure 1 is a schematic diagram of a transparent chassis display system provided by an embodiment of the present application, referring to Figure 1 The system comprises a UAV 101, a vehicle terminal 102 and a head-up display system 103, the UAV 101 is configured with a millimeter wave radar, a laser radar and a camera; The vehicle terminal 102 is configured to send a scanning instruction to the UAV 101 through a vehicle display screen; The UAV 101 is configured to scan the road below the vehicle chassis through the camera based on the scanning instruction to obtain a scanning image, emit an electromagnetic wave signal through the millimeter wave radar, generate first point cloud data based on the electromagnetic wave signal, emit a laser signal through the laser radar, generate second point cloud data based on the laser signal, and send the scanning image, the first point cloud data and the second point cloud data to the vehicle terminal 102; The vehicle terminal 102 is further configured to generate a transparent chassis view based on the scanning image, the first point cloud data and the second point cloud data, send the transparent chassis view to the head-up display system 103, and the transparent chassis view is used to reflect the road information below the vehicle chassis; The head-up display system 103 is configured to display the transparent chassis view.

[0027] An embodiment of the present application provides a transparent chassis display system, in which the UAV 101 scans the road below the vehicle chassis to obtain a scanning image, first point cloud data and second point cloud data, and then the vehicle terminal 102 generates a transparent chassis view based on the scanning image, the first point cloud data and the second point cloud data, and displays the transparent chassis view through the head-up display system 103. Therefore, the system obtains the road information below the chassis through the UAV 101, compared with the traditional fixedly installed camera, the UAV 101 can switch the posture at any time to adjust the shooting angle of the camera, and will not shake with the vehicle, so that the image distortion can be avoided, and the road condition below the chassis can be accurately reflected. Moreover, when generating the transparent chassis view, the system combines the point cloud data of the millimeter wave radar and the laser radar on the basis of the scanning image, so that the relative spatial relationship between the road and the chassis can be more accurately obtained, and the road condition below the chassis can be further accurately reflected.

[0028] It's important to note that traditional fixed-installation cameras, besides lacking adjustable shooting angles, also have poor environmental adaptability. Conditions such as mud smears and chassis collisions can easily cause camera failure, and they lack predictive capabilities, failing to acquire the relative spatial relationship between the road surface and the chassis in the direction of travel. Current UAV-assisted solutions primarily focus on aerial photography and lack solutions that integrate deep visual fusion with the chassis. The system provided in this application creatively uses a UAV 101 to replace traditional fixed-installation cameras. This not only allows for adjustable shooting angles but also avoids camera failures caused by mud smears and chassis collisions, while accurately acquiring the relative spatial relationship between the road and the chassis.

[0029] Furthermore, in related technologies, users view images captured by cameras through an in-vehicle display screen, which can distract them and affect driving safety. In contrast, this application uses a head-up display system 103 to display a transparent chassis view directly in front of the user, eliminating the need for the user to view the in-vehicle display screen and thus improving driving safety.

[0030] In this embodiment, both the drone 101 and the head-up display system 103 are electrically connected to the vehicle-mounted terminal 102. This electrical connection can be a circuit connection or a wireless connection; no specific limitation is made. If the connection is a circuit connection, the connection method can be a cable connection, such as a CAN (Controller Area Network) connection. If the connection is a wireless connection, the connection method can be an infrared connection, a wireless LAN, or a WiFi (Wireless Fidelity) network connection. In this embodiment, no specific limitation is made.

[0031] Among them, the head-up display system 103 can be an AR-HUD (Augmented Reality Head-up Display) display system with a field of view of 12°×5°. The larger field of view allows more information to be displayed, enabling users to be aware of more information.

[0032] In this embodiment, if the drone 101 is currently outside the rooftop docking station, the vehicle-mounted terminal 102 directly sends a scanning command to the drone 101 via the vehicle-mounted display screen. If the drone 101 is currently inside the rooftop docking station, the vehicle-mounted terminal 102 first sends a departure command to the docking station controller via the vehicle-mounted display screen to make the drone 101 leave the rooftop docking station. After the drone 101 leaves the rooftop docking station, the vehicle-mounted terminal 102 then sends a scanning command to the drone 101 via the vehicle-mounted display screen. Accordingly, this process can be as follows: The docking station controller and the UAV 101 are both located in the roof docking station. The roof docking station is also equipped with a locking mechanism and an electromagnetic catapult module. The locking mechanism is used to lock the UAV 101, and the electromagnetic catapult module is used to launch the UAV 101. The vehicle-mounted terminal 102 is also used to send out-of-station commands to the docking station controller via the vehicle-mounted display screen; The docking station controller is used to unlock the locking structure based on the exit command and launch the UAV 101 through the electromagnetic catapult module so that the UAV 101 leaves the roof docking station.

[0033] In this implementation, the vehicle-mounted terminal 102 is equipped with a target application, and the vehicle-mounted terminal 102 displays the program interface of the target application through a vehicle-mounted display screen. This program interface includes an exit option. In response to triggering the exit option, the vehicle-mounted terminal 102 sends an exit command to the docking station controller. The docking station controller is electrically connected to the vehicle-mounted terminal 102. After receiving the exit command, the docking station controller unlocks the locking mechanism and launches the drone 101 via an electromagnetic catapult module.

[0034] The locking mechanism includes an electromagnetic adsorption layer and a mechanical latch, which lock the drone 101 in place. The magnetic flux density of the electromagnetic adsorption layer can be 1.2T, and the locking force of the mechanical latch is ≥200N. The electromagnetic catapult module uses a Halbach array permanent magnet track, which can accelerate the drone 101 to 8m / s within 0.3s.

[0035] The type and size of the drone 101 can be set and changed as needed, without specific limitations. For example, the drone 101 has a six-rotor folding mechanism with an unfolded diameter of 680mm and a folded diameter of 220mm. The camera on the drone 101 can be a liquid lens, and the lidar can be a MEMS (Micro-Electro-Mechanical System) lidar.

[0036] In one possible implementation, the rooftop dock also includes a bidirectional laser charging module for charging the drone 101.

[0037] In this embodiment, the efficiency of the bidirectional laser charging module is ≥91%, and its power can be set and changed as needed, without specific limitation. For example, the power of the bidirectional laser charging module is 50W.

[0038] In this embodiment, the rooftop dock station supports rapid deployment in 8 seconds and highly reliable fixation, with a vibration resistance level of MIL-STD-810H.

[0039] After the drone 101 leaves the rooftop docking station, the docking station controller can send a response message to the vehicle-mounted terminal 102. After receiving the response message, the vehicle-mounted terminal 102 displays a pop-up message indicating that the drone 101 has left the rooftop docking station on the vehicle-mounted display screen.

[0040] In this embodiment of the application, the working modes of the UAV 101 include: flight mode and ground mode. In flight mode, the UAV 101 performs wide-area forward reconnaissance. Its flight altitude is not a fixed value, but a dynamic value that is adaptively adjusted according to the terrain to ensure perception quality and flight safety.

[0041] Its flight altitude control formula can be expressed by the following formula (1): (1) in, This indicates the real-time altitude of drone 101 at time t. This indicates the reference altitude, which is set by the user or determined based on vehicle speed and task type, for example, during vehicle altitude cruising. Higher to cover a greater distance. This represents the terrain undulation compensation amount, which is the terrain roughness sensed in real time by the UAV's 101 tri-modal sensors (millimeter-wave radar / liquid lens / LiDAR). The decision is made using the following formula: Where k is the proportionality coefficient. The elevation standard deviation of the point cloud for the terrain ahead allows the UAV 101 to fly low over flat surfaces to capture details, while it rises over rugged terrain to ensure safety. The scanning frequency is directly proportional to the vehicle speed and inversely proportional to the required reconnaissance range. This ensures that the UAV 101 can perform rapid scanning when the vehicle is moving at high speed, while reducing the scanning frequency when detailed reconnaissance is required, thus increasing the dwell time in a specific area.

[0042] When the vehicle is in a complex, low-speed environment, such as off-road or congested road sections, the UAV 101 switches to ground mode and focuses on detailed modeling of the path the vehicle is about to take.

[0043] Its flight speed can be expressed by the following formula (2): (2) in, This indicates the flight speed of drone 101. Indicates vehicle speed. Represents the terrain attenuation coefficient, 0≤ ≤1, It can be calculated based on the road surface slope, smoothness, and adhesion coefficient. The calculation formula is as follows: , , and These represent the weights of slope, smoothness, and adhesion coefficient, respectively, and the sum of the three is 1. , and These represent the normalized slope, smoothness, and adhesion coefficient, respectively. On a flat asphalt road, ≈1; On muddy and potholed roads, ≈0.

[0044] In one possible implementation, the UAV 101 determines the slope, flatness, and adhesion coefficient based on a three-modal sensor (millimeter-wave radar / liquid lens / liquid radar). Then, the slope, flatness, and adhesion coefficient are normalized respectively. The normalized slope, flatness, and adhesion coefficient are substituted into the above formula to obtain the terrain attenuation coefficient. Then, the terrain attenuation coefficient and the vehicle speed are substituted into formula (2) to obtain the flight speed of the UAV 101.

[0045] In another possible implementation, the vehicle-mounted terminal 102 determines the slope, flatness, and adhesion coefficient based on sensors installed on the vehicle. Then, it normalizes the slope, flatness, and adhesion coefficient, and substitutes these normalized values ​​into the above formula to obtain the terrain attenuation coefficient. Furthermore, the vehicle-mounted terminal 102 acquires the vehicle speed and sends the terrain attenuation coefficient and vehicle speed to the drone 101. The drone 101 determines its flight speed based on the terrain attenuation coefficient and vehicle speed using the above formula (2). Alternatively, the vehicle-mounted terminal 102 determines the drone 101's flight speed based on the vehicle speed and terrain attenuation coefficient using the above formula (2), and sends its flight speed to the drone 101 so that the drone 101 flies at that speed.

[0046] In this embodiment of the application, the program interface of the target application further includes a scanning option. In response to the triggering operation of the scanning option, the vehicle terminal 102 sends a scanning command to the drone 101 through the vehicle display screen.

[0047] In one possible implementation, if the flight speed is determined by the UAV 101, the scanning command carries the vehicle speed. After receiving the scanning command, the UAV 101 determines the terrain attenuation coefficient in the above manner, and then determines the flight speed based on the terrain attenuation coefficient and the vehicle speed using the above formula (2), and then flies based on the flight speed.

[0048] In another possible implementation, if the flight speed of the drone 101 is determined by the vehicle terminal 102, the scanning command carries the flight speed of the drone 101, and the drone 101 flies based on that flight speed after receiving the scanning command.

[0049] It should be noted that the scanning command can also carry a mode identifier of the working mode of the UAV 101. After receiving the scanning command, the UAV 101 switches its working mode to ground mode based on the mode identifier.

[0050] In ground mode, the UAV 101 flies at the speed determined by the above method. Its flight altitude can be a preset altitude or determined according to the above formula (1), without specific limitation. During flight, the UAV 101 scans the road under the vehicle chassis with a camera to obtain a scanned image; it emits electromagnetic wave signals through a millimeter-wave radar and generates first point cloud data based on the electromagnetic wave signals; it emits laser signals through a lidar and generates second point cloud data based on the laser signals, and sends the scanned image, first point cloud data and second point cloud data to the vehicle terminal 102.

[0051] Among them, UAV 101 emits electromagnetic wave signals through millimeter-wave radar and generates the first point cloud data based on the echo signal of the electromagnetic wave signals; similarly, UAV 101 emits laser signals through lidar and generates the second point cloud data based on the echo signal of the laser signals.

[0052] In addition, the distance between the drone 101 and the vehicle can be set and changed as needed, without any specific limitation. For example, the distance between the drone 101 and the vehicle can be 0.5m to 1.2m.

[0053] In this embodiment, the drone 101 integrates three modes: flight, ground movement, and charging, and its size is reduced by 62% compared to traditional solutions. Furthermore, it can achieve multi-view coverage with a single drone through improved serpentine path planning.

[0054] In this embodiment of the application, the process by which the vehicle terminal 102 generates a transparent chassis view based on the scanned image, the first point cloud data, and the second point cloud data can be as follows: The vehicle-mounted terminal 102 is also used to fuse the first point cloud data and the second point cloud data to obtain the third point cloud data; and to generate a transparent chassis view based on the third point cloud data and the scanned image.

[0055] The first point cloud data is the point cloud data corresponding to the millimeter-wave radar, which is sparse point cloud data; the second point cloud data is the point cloud data corresponding to the lidar, which is high-density point cloud data. The vehicle terminal 102 fuses the first and second point cloud data at the same time based on the timestamp to obtain the third point cloud data. Based on the third point cloud data and the scanned image, a transparent chassis view is generated.

[0056] In one possible implementation, the process by which the vehicle-mounted terminal 102 generates a transparent chassis view based on third point cloud data and scanned images can be as follows: The vehicle terminal 102 is also used to establish a four-dimensional coordinate system, which includes time and space dimensions; synchronize the third point cloud data with the scanned image based on the timestamp; convert the synchronized third point cloud data and the scanned image into the four-dimensional coordinate system; and fuse the third point cloud data and the scanned image in the four-dimensional coordinate system through a spatiotemporal fusion algorithm to generate a transparent chassis view with time tags.

[0057] The vehicle-mounted terminal 102 establishes a four-dimensional coordinate system, which includes a time dimension and a spatial dimension. This four-dimensional coordinate system can be represented as (X, Y, Z, t), where X, Y, and Z represent the spatial dimension and t represents the time dimension. The vehicle-mounted terminal 102 aligns the third point cloud data and the scanned image to the same time based on the timestamp, and then transforms the synchronized third point cloud data and the scanned image into the four-dimensional coordinate system.

[0058] The vehicle terminal 102 includes an edge computing unit with a computing power of 10 TOPS (10 Tera Operations Per Second). The edge computing unit runs a space-time fusion algorithm to fuse the third point cloud data in the four-dimensional coordinate system with the scanned image to generate a transparent chassis view with time tags.

[0059] Because there is a certain distance between the drone 101 and the vehicle during scanning, the drone 101 not only scans the road information under the chassis, but also scans the area around the vehicle. Therefore, the generated transparent chassis view includes not only the road information under the chassis, but also the road information around the vehicle.

[0060] In this embodiment, after the vehicle terminal 102 generates a dynamic transparent chassis view that changes over time, it can perform detection based on the transparent chassis view, such as detecting the height of road bumps, the height of obstacles, or the depth of road depressions, and provide vibration warnings via the steering wheel. Correspondingly, the system also includes: an electronic assisted steering controller; The vehicle terminal 102 is also used to send a first warning command to the electronic steering controller when it detects that the height of a road bump is greater than a preset height based on a transparent chassis view; An electronic steering assist controller is used to control the steering wheel to provide a vibration warning in a first vibration mode based on a first warning command; The vehicle terminal 102 is also used to send a second warning command to the electronic steering controller when the depth of the road depression is detected to be greater than a preset depth based on the transparent chassis view. The electronic steering assist controller is also used to control the steering wheel to provide vibration warning in a second vibration mode based on a second warning command.

[0061] Among them, the electronic power steering controller is the controller of the electronic power steering system (EPS), and the electronic power steering controller is electrically connected to the vehicle terminal 102.

[0062] The first warning command may carry a first frequency identifier and a first intensity identifier. The electronic steering controller pre-stores the correspondence between vibration frequency identifiers and vibration frequency ranges, as well as the correspondence between vibration intensity identifiers and vibration intensity ranges. After receiving the first warning command, the electronic steering controller determines the corresponding vibration frequency range and vibration intensity range based on the first frequency identifier and first intensity identifier in the first warning command. Based on the vibration frequency range and vibration intensity range, it determines the first vibration mode and then controls the steering wheel to provide a vibration warning using the first vibration mode.

[0063] Correspondingly, the second warning command carries a second frequency identifier and a second intensity identifier. After receiving the second warning command, the electronic steering controller determines the corresponding vibration frequency range and vibration intensity range based on the second frequency identifier and the second intensity identifier, and then determines the second vibration mode. Then, it controls the steering wheel to perform vibration warning in the second vibration mode.

[0064] The first vibration mode can be a combination of increasing vibration frequency and increasing vibration intensity; the second vibration mode can be a combination of increasing vibration frequency and decreasing vibration intensity. Alternatively, the first vibration mode can be a combination of increasing vibration frequency and decreasing vibration intensity, and the second vibration mode can be a combination of increasing vibration frequency and increasing vibration intensity. No specific limitations are imposed on these combinations.

[0065] The vibration warning time can be set and changed as needed, without any specific limitation. For example, the vibration warning time can be ≤80ms.

[0066] The preset height can be set and changed as needed, without specific limitations. For example, the preset height can be 150mm. For instance, if the curb height is detected to be greater than 150mm, a steering wheel vibration warning will be triggered. Similarly, the preset depth can also be set and changed as needed, without specific limitations.

[0067] In this embodiment of the application, the electronic steering controller can control the steering wheel to use different methods to provide vibration warnings for different situations, so that the user can know whether the road surface under the chassis is raised or sunken according to the warning method.

[0068] It should be noted that when the vehicle terminal 102 detects that the height of a road bump is greater than a preset height or the depth of a road depression is greater than a preset depth based on the transparent chassis view, it can mark the corresponding position of the road bump or road depression in the transparent chassis view, such as marking the height of the road bump or the depth of the road depression. In this way, when the user views the transparent chassis view through the head-up display system 103, the height of the road bump or the depth of the road depression can be seen more intuitively.

[0069] In summary, the system provided in this application can be applied in off-road scenarios, such as those with uneven and bumpy roads, allowing users to know the road information beneath the chassis. The system can also be applied to scenarios involving oncoming traffic on narrow roads, where the vehicle needs to move to the side of the road to avoid oncoming vehicles; the system allows users to know the road information beneath the chassis. Of course, the system can also be applied to other scenarios, which will not be listed here.

[0070] Figure 2 This is a flowchart of a transparent chassis display method provided in an embodiment of this application. See also... Figure 2 The method includes: Step 201: The vehicle-mounted terminal sends a scanning command to the drone via the vehicle-mounted display screen.

[0071] Step 202: Based on the scanning command, the UAV scans the road under the vehicle chassis using its camera to obtain a scanned image; it emits electromagnetic wave signals using millimeter-wave radar and generates the first point cloud data based on the electromagnetic wave signals; it emits laser signals using lidar and generates the second point cloud data based on the laser signals.

[0072] Step 203: The drone sends scanned images, first point cloud data, and second point cloud data to the vehicle-mounted terminal.

[0073] Step 204: The vehicle terminal generates a transparent chassis view based on the scanned image, the first point cloud data, and the second point cloud data, and sends the transparent chassis view to the head-up display system. The transparent chassis view is used to reflect the road information under the vehicle chassis. Step 205: The head-up display system displays a transparent chassis view.

[0074] In one possible implementation, the on-board terminal generates a transparent chassis view based on the scanned image, first point cloud data, and second point cloud data, including: The vehicle-mounted terminal merges the first point cloud data and the second point cloud data to obtain the third point cloud data; A transparent chassis view is generated based on third-point cloud data and scanned images.

[0075] In another possible implementation, the in-vehicle terminal generates a transparent chassis view based on third-party point cloud data and scanned images, including: The vehicle-mounted terminal establishes a four-dimensional coordinate system, which includes time and space dimensions. Synchronize the third-point cloud data with the scanned image based on the timestamp; The synchronized third point cloud data and scanned image are converted to a four-dimensional coordinate system; By using a spatiotemporal fusion algorithm, the third point cloud data in the four-dimensional coordinate system and the scanned image are fused to generate a transparent chassis view with time labels.

[0076] In another possible implementation, the method also includes: The vehicle-mounted terminal sends an outbound command to the docking station controller via the vehicle-mounted display screen. Based on the departure command, the docking station controller unlocks the locking mechanism and launches the drone via the electromagnetic catapult module, allowing the drone to leave the rooftop docking station.

[0077] In another possible implementation, the method also includes: When the vehicle terminal detects that the height of a road protrusion is greater than a preset height based on a transparent chassis view, it sends a first warning command to the electronic steering assist controller. The electronic steering assist controller, based on a first warning command, controls the steering wheel to provide a vibration warning using a first vibration mode; or... When the vehicle terminal detects a road indentation depth greater than a preset depth based on a transparent chassis view, it sends a second warning command to the electronic steering assist controller. The electronic steering assist controller controls the steering wheel to provide vibration warning in a second vibration mode based on the second warning command.

[0078] In another possible implementation, the scanning command carries the vehicle speed; Based on scanning commands, the drone uses its camera to scan the road beneath the vehicle chassis, obtaining scanned images, including: The drone determines the terrain attenuation coefficient; based on the vehicle speed and the terrain attenuation coefficient, the drone's flight speed is determined; based on the drone's flight speed, the road under the vehicle chassis is scanned using a camera to obtain a scanned image.

[0079] This application provides a method for displaying a transparent chassis. In this method, a drone scans the road beneath the vehicle chassis, obtaining scanned images, first point cloud data, and second point cloud data. Then, an onboard terminal generates a transparent chassis view based on the scanned images, first point cloud data, and second point cloud data, and displays the transparent chassis view using a head-up display method. Therefore, this method uses a drone to acquire road information beneath the chassis. Compared to traditional fixed cameras, the drone can switch its posture at any time to adjust the camera's shooting angle and will not shake with vehicle movement, avoiding image distortion and thus accurately reflecting the road conditions beneath the chassis. Furthermore, when generating the transparent chassis view, this method combines point cloud data from millimeter-wave radar and lidar with the scanned images, which can more accurately obtain the relative spatial relationship between the road and the chassis, thereby further accurately reflecting the road conditions beneath the chassis.

[0080] refer to Figure 3 , Figure 3 A structural block diagram of an in-vehicle terminal 300 provided in an exemplary embodiment of this application is shown.

[0081] Typically, the vehicle terminal 300 includes a processor 301 and a memory 302.

[0082] Processor 301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0083] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one piece of program code, which is executed by the processor 301 to implement the operations performed by the vehicle terminal in the transparent chassis display method provided in the method embodiments of this application.

[0084] In some embodiments, the vehicle terminal 300 may optionally include a peripheral device interface 303 and at least one peripheral device. The processor 301, memory 302, and peripheral device interface 303 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 304, a display screen 305, a camera assembly 306, an audio circuit 307, and a power supply 308.

[0085] The peripheral device interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, memory 302, and peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, memory 302, and peripheral device interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0086] The radio frequency (RF) circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 304 can communicate with other vehicle terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 304 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0087] Display screen 305 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 305 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 305 may be a single screen, disposed on the front panel of vehicle terminal 300; in other embodiments, display screen 305 may be at least two screens, disposed on different surfaces of vehicle terminal 300 or in a folded design; in other embodiments, display screen 305 may be a flexible display screen, disposed on a curved or folded surface of vehicle terminal 300. Furthermore, display screen 305 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 305 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0088] The camera assembly 306 is used to acquire images or videos. Optionally, the camera assembly 306 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the vehicle terminal, and the rear-facing camera is located on the back of the vehicle terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0089] The audio circuit 307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 301 for processing, or to the radio frequency circuit 304 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the vehicle terminal 300. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 301 or the radio frequency circuit 304 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 307 may also include a headphone jack.

[0090] Power supply 308 is used to power the various components in the vehicle terminal 300. Power supply 308 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 308 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0091] In some embodiments, the vehicle terminal 300 further includes one or more sensors 309. The one or more sensors 309 include, but are not limited to: an acceleration sensor 310, a gyroscope sensor 311, a pressure sensor 312, an optical sensor 313, and a proximity sensor 314.

[0092] Accelerometer 310 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the vehicle terminal 300. For example, accelerometer 310 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 301 can control display screen 305 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 310. Accelerometer 310 can also be used for games or for acquiring user motion data.

[0093] The gyroscope sensor 311 can detect the orientation and rotation angle of the vehicle terminal 300. The gyroscope sensor 311 can work in conjunction with the accelerometer sensor 310 to collect 3D motion data from the user on the vehicle terminal 300. Based on the data collected by the gyroscope sensor 311, the processor 301 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0094] The pressure sensor 312 can be installed on the side bezel of the vehicle terminal 300 and / or on the lower layer of the display screen 305. When the pressure sensor 312 is installed on the side bezel of the vehicle terminal 300, it can detect the user's grip signal on the vehicle terminal 300, and the processor 301 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 312. When the pressure sensor 312 is installed on the lower layer of the display screen 305, the processor 301 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 305. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0095] Optical sensor 313 is used to collect ambient light intensity. In one embodiment, processor 301 can control the display brightness of display screen 305 based on the ambient light intensity collected by optical sensor 313. Specifically, when the ambient light intensity is high, the display brightness of display screen 305 is increased; when the ambient light intensity is low, the display brightness of display screen 305 is decreased. In another embodiment, processor 301 can also dynamically adjust the shooting parameters of camera assembly 306 based on the ambient light intensity collected by optical sensor 313.

[0096] The proximity sensor 314, also known as a distance sensor, is typically installed on the front panel of the vehicle terminal 300. The proximity sensor 314 is used to detect the distance between the user and the front of the vehicle terminal 300. In one embodiment, when the proximity sensor 314 detects that the distance between the user and the front of the vehicle terminal 300 is gradually decreasing, the processor 301 controls the display screen 305 to switch from a screen-on state to a screen-off state; when the proximity sensor 314 detects that the distance between the user and the front of the vehicle terminal 300 is gradually increasing, the processor 301 controls the display screen 305 to switch from a screen-off state to a screen-on state.

[0097] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the vehicle terminal 300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0098] The structural block diagram of the drone can be found here. Figure 4The drone 400 can vary considerably depending on its configuration or performance. It may include a central processing unit (CPU) 401 and a memory 402. The memory 402 stores at least one line of program code, which is loaded and executed by the processor 401 to implement the operations performed by the drone in the aforementioned transparent chassis display method. Of course, the drone 400 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The drone 400 may also include other components for implementing device functions, which will not be elaborated here.

[0099] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one piece of program code that is loaded and executed by a processor to implement the transparent chassis display method in the above embodiments.

[0100] In an exemplary embodiment, a computer program product is also provided, which stores at least one piece of program code that is loaded and executed by a processor to implement the transparent chassis display method in the above embodiments.

[0101] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0102] The above description is only for the purpose of enabling those skilled in the art to understand the technical solution of this application, and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A transparent chassis display system, characterized in that, The system includes: a drone, a vehicle-mounted terminal, and a head-up display system. The drone is equipped with millimeter-wave radar, lidar, and a camera. The vehicle-mounted terminal is used to send scanning commands to the drone via the vehicle-mounted display screen; The drone is used to scan the road under the vehicle chassis using the camera based on the scanning command to obtain a scanned image; to emit electromagnetic wave signals using the millimeter-wave radar and generate first point cloud data based on the electromagnetic wave signals; to emit laser signals using the lidar and generate second point cloud data based on the laser signals; and to send the scanned image, the first point cloud data, and the second point cloud data to the vehicle-mounted terminal. The vehicle terminal is also used to generate a transparent chassis view based on the scanned image, the first point cloud data and the second point cloud data, and send the transparent chassis view to the head-up display system. The transparent chassis view is used to reflect the road information under the vehicle chassis. The head-up display system is used to display the transparent chassis view.

2. The system according to claim 1, characterized in that, The vehicle-mounted terminal is further configured to fuse the first point cloud data and the second point cloud data to obtain third point cloud data; and generate the transparent chassis view based on the third point cloud data and the scanned image.

3. The system according to claim 2, characterized in that, The vehicle-mounted terminal is also used to establish a four-dimensional coordinate system, which includes a time dimension and a spatial dimension; synchronize the third point cloud data with the scanned image based on a timestamp; and convert the synchronized third point cloud data and the scanned image into the four-dimensional coordinate system. The third point cloud data in the four-dimensional coordinate system and the scanned image are fused using a spatiotemporal fusion algorithm to generate the transparent chassis view with time tags.

4. The system according to claim 1, characterized in that, The system also includes: a docking station controller, both the docking station controller and the UAV are installed in a rooftop docking station, the rooftop docking station is also equipped with a locking mechanism and an electromagnetic catapult module, the locking mechanism is used to lock the UAV, and the electromagnetic catapult module is used to launch the UAV; The vehicle-mounted terminal is also used to send an outbound command to the docking station controller via the vehicle-mounted display screen; The docking station controller is used to unlock the locking mechanism based on the departure command, and launch the UAV through the electromagnetic catapult module so that the UAV leaves the rooftop docking station.

5. The system according to claim 1, characterized in that, The system also includes: an electronic steering assist controller; The vehicle terminal is also used to send a first warning command to the electronic steering controller when it detects that the height of a road bump is greater than a preset height based on the transparent chassis view; The electronic steering assist controller is used to control the steering wheel to provide vibration warning in a first vibration mode based on the first warning command; The vehicle terminal is also used to send a second warning command to the electronic steering controller when it detects that the depth of the road depression is greater than a preset depth based on the transparent chassis view; The electronic steering assist controller is also used to control the steering wheel to provide vibration warning in a second vibration mode based on the second warning command.

6. The system according to claim 1, characterized in that, The scanning command includes the vehicle speed; The drone is used to determine the terrain attenuation coefficient; based on the vehicle speed and the terrain attenuation coefficient, the drone's flight speed is determined; based on the drone's flight speed, the step of scanning the road under the vehicle chassis using the camera to obtain the scanned image is performed.

7. A method for displaying a transparent chassis, characterized in that, The method includes: The vehicle-mounted terminal sends scanning commands to the drone via the vehicle-mounted display screen. Based on the scanning command, the drone scans the road under the vehicle chassis using a camera to obtain a scanned image; it emits electromagnetic wave signals using a millimeter-wave radar and generates first point cloud data based on the electromagnetic wave signals; it emits laser signals using a lidar and generates second point cloud data based on the laser signals; and it sends the scanned image, the first point cloud data, and the second point cloud data to the vehicle-mounted terminal. The vehicle terminal generates a transparent chassis view based on the scanned image, the first point cloud data, and the second point cloud data, and sends the transparent chassis view to the head-up display system. The transparent chassis view is used to reflect the road information under the vehicle chassis. The head-up display system shows the transparent chassis view.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one piece of program code, which is loaded and executed by the processor to implement the transparent chassis display method as described in claim 7 for a vehicle-mounted terminal or drone.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the transparent chassis display method as described in claim 7 for a vehicle-mounted terminal or a drone.

10. A computer program product, characterized in that, The computer program product stores at least one piece of program code, which is loaded and executed by a processor to implement the transparent chassis display method as described in claim 7 for the vehicle-mounted terminal or drone.

Images