A method, system, and storage medium for displaying a vehicle's panoramic perspective.

Abstract

This invention provides a method, system, and storage medium for displaying a panoramic view of a vehicle. The method includes: an AR user terminal entering a working mode after connecting to the network; acquiring multiple images of the vehicle's surrounding environment using cameras to obtain a panoramic image; and obtaining a panoramic image corresponding to the user's current viewpoint based on the AR user terminal's real-time coordinates and pose data; simultaneously detecting obstacles around the target vehicle and generating corresponding instruction animations and prompts based on the obstacles; and then further fusing the panoramic image and instruction animations and presenting them to the AR user terminal. The AR user terminal used in this invention can perceive the location of obstacles, quickly assess their impact, and generate corresponding instruction animations and prompts, allowing users to perceive the direction of risk and make appropriate driving decisions; it can also remotely assist the driver in the target vehicle, improving driving safety; and it can also provide a three-dimensional interactive experience.

Description

Technical Field

[0001] This invention relates to the field of panoramic display technology, and in particular to a method, system, and storage medium for panoramic perspective display of vehicles. Background Technology

[0002] Intelligent vehicles refer to a new generation of automobiles that, by incorporating advanced sensors and other devices and utilizing new technologies such as artificial intelligence, possess autonomous driving capabilities and are gradually becoming intelligent mobile spaces and application terminals. They typically feature sensors such as cameras, ultrasonic radar, millimeter-wave radar, and lidar to "perceive" the vehicle body and its surrounding environment, and intelligent antennas to acquire internet information.

[0003] Augmented Reality Head-Up Displays (AR-HUDs) are used to display information for user perception. In the field of smart cars, fixed AR-HUDs use display screens, transparent glass, and other display carriers, while wearable devices include AR glasses and display helmets. The purpose of AR-HUDs is to improve the safety, convenience, and comfort of driving. Smart cars use fisheye or ultra-wide-angle lenses to capture images of signs and patterns, and then use distortion correction, affine transformation, and image stitching techniques to obtain a panoramic view. This technology is mature, stable, and widely used.

[0004] Existing wearable AR-HUD systems are built based on the needs of driving safety, information acquisition, and convenient interaction, primarily focusing on forward-looking driving scenarios. When facing complex scenarios other than medium-to-high-speed driving, these systems cannot achieve 3D panoramic perspective, making it difficult for users to directly observe the external environment obscured by the vehicle body. Furthermore, the information interaction methods of existing wearable AR-HUD systems have the risk of failure; their prompts are difficult to distinguish in noisy environments, and their warning images cannot indicate direction based on human posture, hindering user reaction time. They also primarily display two-dimensional icons and numerical information, making it difficult to provide a three-dimensional interactive experience, and are only suitable for use inside the car, unable to be used for remote monitoring of vehicle status and potential risks in the external environment. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method, system, and storage medium for displaying a panoramic view of a vehicle. By connecting an AR user terminal to the target vehicle via a network, it collects information about the vehicle's surrounding environment to generate corresponding panoramic images and instruction animations. These panoramic images and instruction animations are then fused and presented to the AR user terminal. The AR user terminal used in this invention can perceive the location of obstacles, quickly assess their impact, and generate corresponding instruction animations and prompts, allowing users to perceive the direction of risk and make appropriate driving decisions. It can also remotely assist the driver in the target vehicle, improving driving safety, and provides a three-dimensional interactive experience.

[0006] Specifically, the present invention provides a method for displaying a vehicle's panoramic perspective, comprising the following steps:

[0007] Step S10: After the AR user terminal completes the network connection, it enters the working mode.

[0008] Step S20: Acquire multiple images of the vehicle's surrounding environment and stitch the multiple images together to obtain a panoramic image.

[0009] Step S30: Obtain the real-time coordinates of the AR user terminal, calculate the pose data of the AR user terminal in real time and detect obstacles around the target vehicle, obtain the panoramic image corresponding to the user's current viewpoint based on the real-time coordinates and pose data, and generate corresponding instruction animations and prompts based on the obstacles.

[0010] Step S40: Fuse the panoramic image and the instruction animation and present them to the AR user terminal.

[0011] In step S10, before the AR user terminal enters the working mode, the following steps are also included: authenticating the current user's identity and configuring the user's driving preferences based on the authentication result; the AR user terminal is bound to at least one target vehicle and configured with at least one user.

[0012] Step S10 further includes: defining an AR user terminal coordinate system, wherein the AR user terminal coordinate system has the center of the target vehicle as the origin, the width direction as the X-axis, the length direction as the Y-axis, and the height direction as the Z-axis.

[0013] Step S20 specifically involves:

[0014] Step S21: Install cameras around the target vehicle, as well as at the bottom and top, and calibrate each camera.

[0015] Step S22: Select a projection model and generate its corresponding mapping table based on the projection model.

[0016] Step S23: Acquire multiple images of the vehicle's surrounding environment using each camera, and perform optical flow tracking calculations on the multiple images of the vehicle's surrounding environment to obtain the environmental parameters in the projection model.

[0017] Step S24: Match the mapping table according to the environmental parameters, and perform texture mapping on multiple vehicle surrounding environment images according to the mapping table to obtain a panoramic image; the panoramic image includes a perspective view and a filled display of the vehicle surrounding environment image.

[0018] The step S30, which generates corresponding instruction animations and prompts based on the obstacles, specifically involves: calculating the relative position of the obstacle in the AR user terminal coordinate system, determining the relative distance between the obstacle and the target vehicle based on the relative position, and generating the corresponding instruction animation and prompts based on the relative distance if the relative distance is less than or equal to a preset threshold; otherwise, continuing to detect obstacles around the target vehicle.

[0019] Step S40 further includes:

[0020] When the user is inside the vehicle, the user can interact with the panoramic image and directional animations through gestures, voice, or touch; the system also interacts with the user through obstacle voice announcements.

[0021] The gestures include at least clicking, grabbing, and expanding.

[0022] When a user is in a remote environment, the user uses voice to assist the driver in driving the vehicle, and the driver is in the target vehicle that is bound to the AR user terminal configured by the user.

[0023] Based on the same inventive concept, the present invention also provides a vehicle panoramic perspective display system, the system comprising:

[0024] Acquisition module: includes at least cameras respectively installed around the vehicle body, as well as at the bottom and top, the cameras being used to acquire images of the surrounding environment of the target vehicle.

[0025] Storage module: Used to store user identity and corresponding driving preferences.

[0026] Identification module: Used to authenticate the user's identity in order to configure the user's driving preferences based on the user's identity in the storage module.

[0027] Display module: used to display panoramic images and directional animations via an AR user terminal; the panoramic image includes a perspective view and a filled-in view of the vehicle's surrounding environment.

[0028] First communication module: used to connect the AR user terminal and the target vehicle via WIFI for in-vehicle or remote communication.

[0029] The second communication module is used to assist in-vehicle or remote communication via a microphone and a speaker; the microphone is used for user voice interaction; and the speaker is used for obstacle voice announcements.

[0030] Interaction module: When the user is in the vehicle environment, it is used for the user to interact with the panoramic image and instruction animation through gestures, voice or touch; and the second communication module interacts with the user through obstacle voice broadcast; the gestures include at least clicking, grabbing and expanding.

[0031] When a user is in a remote environment, the system allows the user to drive the vehicle with voice assistance, and the driver is in the target vehicle that is bound to the AR user terminal configured by the user.

[0032] The system also includes:

[0033] Measurement module: Used to measure the pose data of the AR user terminal in real time.

[0034] Acquisition module: used to acquire the real-time coordinates of the AR user terminal, and obtain the panoramic image corresponding to the user's current viewpoint based on the real-time coordinates and the pose data.

[0035] Detection module: Used to detect obstacles around the target vehicle in real time.

[0036] The first generation module is used to perform optical flow tracking and texture mapping on multiple vehicle surrounding environment images acquired by the calibrated camera to generate panoramic images corresponding to the multiple vehicle surrounding environment images.

[0037] The second generation module is used to calculate the relative position of the obstacle in the AR user terminal coordinate system, and generate the corresponding indicator animation and prompt instructions based on the relative position.

[0038] The display module includes at least a first display area, a second display area, and a third display area. Each display area displays the instruction animation in coordination with the relative position of the obstacle. The first and second display areas are used to display perspective views. The third display area is used to fill and display images of the vehicle's surrounding environment captured by the camera.

[0039] Based on the same inventive concept, the present invention also provides a storage medium, which is a type of computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed by a processor, implements the vehicle panoramic perspective display method.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. The present invention can obtain a panoramic image corresponding to the user's current perspective based on the real-time coordinates and pose data of the AR user terminal. The panoramic image includes a perspective view and a filled display of the vehicle's surrounding environment. This solves the technical problem in the prior art that 3D panoramic perspective cannot be achieved when facing complex scenes that are not at medium or high speeds, making it difficult for users to directly observe the external environment obscured by the vehicle body.

[0042] 2. This invention acquires the relative position of obstacles in the AR user terminal coordinate system in real time, and obtains the relative distance between the obstacle and the target vehicle based on the relative position, so as to generate corresponding instruction animations and prompts based on the relative distance, enabling users to perceive the direction of risks, shortening the user's reaction time, and assisting safe driving; it solves the technical problems in the prior art where the prompt sound is difficult to distinguish in noisy environments, and the warning screen cannot indicate the direction based on human posture.

[0043] 3. This invention allows users to interact with the acquired panoramic images and instruction animations via gestures, voice, or touch, and can also provide voice prompts for obstacles to assist driving; it solves the technical problem that existing systems mainly display two-dimensional icons and digital information.

[0044] 4. This invention connects the AR user terminal to the target vehicle in a remote environment, enabling remote assistance to the driver in the target vehicle via voice calls; it solves the technical problem that existing technologies cannot remotely monitor vehicle status and potential risks in the external environment. Attached Figure Description

[0045] Figure 1 This is a flowchart of the vehicle panoramic perspective display method described in this invention.

[0046] Figure 2 for Figure 1 The flowchart of the method for obtaining panoramic images.

[0047] Figure 3 for Figure 1 The system framework diagram of the vehicle panoramic perspective display method is shown below.

[0048] Figure 4 This is a schematic diagram of the environment in front of an AR user terminal in one embodiment.

[0049] Figure 5 This is a schematic diagram of the environment after using an AR user terminal in one embodiment.

[0050] Figure 6 for Figure 1 A schematic diagram of the display area of ​​the AR user terminal. Detailed Implementation

[0051] This invention provides a method, system, and storage medium for displaying a vehicle's panoramic perspective, thereby addressing the technical problems in the prior art that cannot achieve 3D panoramic perspective, cannot perceive risk postures, cannot provide a three-dimensional interactive experience, and cannot achieve remote assistance.

[0052] The technical solutions in the embodiments of the present invention are intended to solve the above-mentioned technical problems, and the overall approach is as follows:

[0053] The AR user terminal completes network connectivity and authenticates the current user's identity. Based on the authentication result, driving preferences corresponding to the user are configured. Then, it enters working mode and acquires a panoramic image by collecting multiple images of the target vehicle's surrounding environment through various cameras. Based on the real-time coordinates and pose data of the AR user terminal, a panoramic image corresponding to the user's current viewpoint is obtained. At the same time, obstacles around the target vehicle are detected, and corresponding instruction animations and prompts are generated based on the obstacles. The panoramic image and instruction animation are then fused and presented to the AR user terminal.

[0054] The following detailed description of a vehicle panoramic perspective display method, system, and storage medium according to the present invention, with reference to specific embodiments and accompanying drawings, is provided in further detail.

[0055] Please see Figure 1 This invention provides a method for displaying a vehicle's panoramic perspective, comprising the following steps:

[0056] Step S10: After the AR user terminal completes the network connection, it enters the working mode.

[0057] The AR user terminal is an AR-HUD, which is an augmented reality head-up display, and can be selected as AR glasses or a display helmet.

[0058] Before the AR user enters working mode, the process also includes: authenticating the current user's identity and configuring the user's driving preferences based on the authentication result.

[0059] The AR user terminal is bound to at least one target vehicle and configured with at least one user.

[0060] It should be noted that when users use the AR client for the first time, they need to enter their personal information, which can be voice, fingerprint or face.

[0061] After the AR user terminal connects to the target vehicle via Wi-Fi, if voice authentication is used, the user can speak any voice command, and the system will compare the voice command with the stored voice data. If the comparison is successful, the authentication is successful. If fingerprint authentication is used, the user needs to touch a designated button with the finger that has been entered to complete the authentication. If facial authentication is used, the system will automatically perform the authentication after the user wears the AR user terminal. If the facial information matches the stored facial data, the authentication is successful.

[0062] After completing user identity verification, the system will automatically configure driving preferences corresponding to that user; the driving preferences include at least data such as rearview mirror position information, driver's seat position information, and multimedia control information.

[0063] Step S10 further includes: defining an AR user terminal coordinate system, wherein the AR user terminal coordinate system has the center of the target vehicle as the origin, the width direction as the X-axis, the length direction as the Y-axis, and the height direction as the Z-axis.

[0064] After the AR user terminal and the target vehicle are connected, step S20 can be executed.

[0065] Step S20: Acquire multiple images of the vehicle's surrounding environment and stitch the multiple images together to obtain a panoramic image.

[0066] Please see Figure 2 Step S20 specifically includes:

[0067] Step S21: Fisheye cameras are installed around the target vehicle, as well as at the bottom and top. Images are acquired through six fisheye cameras. The acquired images are processed to correct distortion using a fisheye image distortion correction algorithm and the built-in matrix parameters and deformation coefficients of the fisheye cameras. Then, each camera is calibrated, and corresponding projection transformations are performed based on the actual position information of the objects in the images.

[0068] It should be noted that by adding bottom and top cameras to the existing cameras around the vehicle, blind spots are minimized and information exchange efficiency is improved.

[0069] Step S22: Select the bottom of the model as the panoramic top-down projection model, and the rest as the panoramic surround projection model. Based on the projection model, construct a position mapping table from the fisheye image to the distortion-free image.

[0070] Step S23: Collect multiple videos of the vehicle's surrounding environment using each camera, obtain image frames from the videos, perform feature point tracking and matching, determine the feature points on the main plane of the obstacle based on the tracking results, and obtain the optical flow value and relative depth value of the feature points based on the different distances between the obstacle and the target vehicle, the different optical flow magnitudes and changing patterns; then obtain the parameter values ​​of the main plane of the obstacle, i.e., environmental parameters, based on the optical flow value and relative depth value.

[0071] Step S24: Match the location mapping table according to the environmental parameters, and perform texture mapping on multiple vehicle surrounding environment images according to the location mapping table to obtain a panoramic image; the panoramic image includes a perspective view and a filled display of the vehicle surrounding environment image.

[0072] In the process of texture mapping, in order to make the boundary area transition smoothly, an overlapping area is reserved between the stitched images of adjacent cameras, and the overlapping area is fused.

[0073] After obtaining the panoramic image, step S30 can be executed.

[0074] Step S30: Obtain the real-time coordinates of the AR user terminal, calculate the pose data of the AR user terminal in real time, and detect obstacles around the target vehicle in real time. Obtain a panoramic image corresponding to the user's current viewpoint based on the real-time coordinates and pose data. At the same time, calculate the relative position of the obstacle in the coordinate system of the AR user terminal, and determine the relative distance between the obstacle and the target vehicle based on the relative position. If the relative distance is less than or equal to a preset threshold, generate an indication animation and prompt command corresponding to the obstacle; otherwise, continue to detect obstacles around the target vehicle.

[0075] It should be noted that an attitude sensor is used to acquire the pose data, which includes both displacement data and angle data; the attitude sensor contains multiple accelerometers and gyroscopes.

[0076] The instruction animation is a superimposed extension of virtual objects generated by the system into the user's three-dimensional environment. These virtual objects include at least digital road images, directional arrows, step distance, speed limit, speed, engine speed, battery voltage, engine malfunction, tire indicator lights, driver fatigue warning, water temperature alarm, music playback, weather forecast, voice interaction, fuel consumption, and obstacle information, etc.

[0077] One possible implementation, regarding lane keeping, is that lane keeping is essential in inclement weather and darkness. The AR client can draw lines on the driving path to make the current lane more prominent, thereby reducing vehicle deviation.

[0078] One possible implementation method for detecting critical road events involves drivers constantly monitoring vehicles, road hazards, lanes, pedestrians, and traffic signs while controlling speed and direction. This increases both physical and mental strain, and is particularly dangerous for the elderly and those with slower cognitive reactions. Therefore, a vivid and comprehensive warning system can help reduce driver workload and thus decrease traffic accidents. Displaying obstacle information and directional arrows can aid in identifying dangerous events and, to some extent, shorten driver reaction time.

[0079] One possible implementation, regarding night vision, is that the AR user terminal can indicate the location information of pedestrians or vehicles, allowing drivers to understand it directly and making information transmission more efficient.

[0080] In one possible implementation, the prompting command is a vibration command, allowing the user to perceive the degree of danger and direction of danger posed by the obstacle to the target vehicle. The degree of danger is primarily indicated by the vibration intensity, while the direction of danger is indicated by vibration at a corresponding location on the AR user terminal.

[0081] Step S40: Fuse the panoramic image and the instruction animation and present them to the AR user terminal.

[0082] Step S40 further includes:

[0083] When the user is inside the vehicle, the user can interact with the panoramic image and directional animations through gestures, voice, or touch; the system also interacts with the user through obstacle voice announcements.

[0084] The gestures include at least clicking, grabbing, and expanding.

[0085] It should be noted that the AR user terminal is also equipped with a depth camera and a pressure sensor to determine the interactive action the user wants to perform; the depth camera can collect point cloud data of the user's gestures; the pressure sensor collects touch information; both the point cloud data and the touch information are sent to the target vehicle via WIFI signal to identify the corresponding interactive action information and complete the interaction.

[0086] When a user is in a remote environment, the user uses voice to assist the driver in driving the vehicle, and the driver is in the target vehicle that is bound to the AR user terminal configured by the user.

[0087] It should be noted that a microphone and a speaker are used in combination to achieve the above-mentioned obstacle voice broadcasting and voice assistance.

[0088] Based on the same inventive concept, this invention also provides a vehicle panoramic perspective display system; please refer to [link to relevant documentation]. Figure 3The system includes:

[0089] Acquisition module: includes at least cameras respectively installed around the vehicle body, as well as at the bottom and top, the cameras being used to acquire images of the surrounding environment of the target vehicle.

[0090] Storage module: Used to store user identity and corresponding driving preferences.

[0091] It should be noted that the driving preferences include at least data such as the position information of the vehicle's rearview mirrors, the position information of the driver's seat, and multimedia control information.

[0092] Identification module: Used to authenticate the user's identity in order to configure the user's driving preferences based on the user's identity in the storage module.

[0093] Display module: used to display panoramic images and directional animations via an AR user terminal; the panoramic image includes a perspective view and a filled-in view of the vehicle's surrounding environment.

[0094] First communication module: used to connect the AR user terminal and the target vehicle via WIFI for in-vehicle or remote communication.

[0095] The second communication module is used to assist in-vehicle or remote communication via a microphone and a speaker; the microphone is used for user voice interaction; and the speaker is used for obstacle voice announcements.

[0096] Interaction module: When the user is in the vehicle environment, it is used for the user to interact with the panoramic image and instruction animation through gestures, voice or touch; and the second communication module interacts with the user through obstacle voice broadcast; the gestures include at least clicking, grabbing and expanding.

[0097] When a user is in a remote environment, the system allows the user to drive the vehicle with voice assistance, and the driver is in the target vehicle that is bound to the AR user terminal configured by the user.

[0098] The system also includes:

[0099] Measurement module: Used to measure the pose data of the AR user terminal in real time.

[0100] Acquisition module: used to acquire the real-time coordinates of the AR user terminal, and obtain the panoramic image corresponding to the user's current viewpoint based on the real-time coordinates and the pose data.

[0101] One possible implementation is that the current user wants to observe environmental information on the right side of the driver's seat, which includes both in-vehicle and out-of-vehicle environmental information; Figure 4This is the environment that users can observe before using the AR client. As you can see, the environment at this time only includes the interior of the car and the exterior environment that can be seen through the car window. Figure 5 This is the environment that users can observe after using the AR client. As you can see, the environment at this time includes not only the interior environment of the car, but also the exterior environment that is blocked by the car door, realizing the perspective display function.

[0102] Detection module: Used to detect obstacles around the target vehicle in real time.

[0103] The first generation module is used to perform optical flow tracking and texture mapping on multiple vehicle surrounding environment images acquired by the calibrated camera to generate panoramic images corresponding to the multiple vehicle surrounding environment images.

[0104] The second generation module is used to calculate the relative position of the obstacle in the AR user terminal coordinate system, and generate the corresponding indicator animation and prompt instructions based on the relative position.

[0105] Please see Figure 6 The display module W4 includes at least a first display area W4-1, a second display area W4-2, and a third display area W4-3. Each display area displays the instruction animation in coordination with the relative position of the obstacle. The first display area W4-1 and the second display area W4-2 are used to display perspective images. The third display area W4-3 is used to fill and display the vehicle surrounding environment image captured by the camera.

[0106] Based on the same inventive concept, the present invention also provides a storage medium, which is a type of computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed by a processor, implements the vehicle panoramic perspective display method.

[0107] In summary, this invention provides a method, system, and storage medium for displaying a panoramic view of a vehicle. By connecting an AR user terminal to the target vehicle via a network, it collects information about the vehicle's surrounding environment to generate corresponding panoramic images and instruction animations. These panoramic images and instruction animations are then fused and presented to the AR user terminal. The AR user terminal used in this invention can perceive the location of obstacles, quickly assess their impact, and generate corresponding instruction animations and prompts, allowing users to perceive the direction of risk and make appropriate driving decisions. It can also remotely assist the driver in the target vehicle, improving driving safety, and provides a three-dimensional interactive experience.

[0108] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0111] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] Although the invention has been described in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.

Claims

1. A method for displaying a vehicle's panoramic perspective, characterized in that, Includes the following steps: S10: The AR user terminal enters working mode after completing network connection; S20: Acquire multiple images of the surrounding environment of the target vehicle, and stitch the multiple images of the surrounding environment to obtain a panoramic image; Specifically, step S20 is as follows: S21: Install cameras around the target vehicle, as well as at the bottom and top, and calibrate each camera. S22: Select the bottom of the model as the panoramic top-down projection model, and the rest as the panoramic surround projection model. Construct a position mapping table from the fisheye image to the distortion-free image based on the projection model. S23: Collect multiple videos of the vehicle's surrounding environment using various cameras, acquire image frames from the videos to serve as images of the vehicle's surrounding environment, perform feature point tracking and matching, determine feature points on the main plane of the obstacle based on the tracking results, and obtain the optical flow value and relative depth value of the feature points based on the distance between the obstacle and the target vehicle, the magnitude and variation of the optical flow; obtain the parameter values ​​of the main plane of the obstacle based on the optical flow value and relative depth value, as environmental parameters; S24: Match the mapping table according to the environmental parameters, and perform texture mapping on multiple vehicle surrounding environment images according to the mapping table to obtain a panoramic image; the panoramic image includes a perspective view and a filled display of the vehicle surrounding environment image; S30: Obtain the real-time coordinates of the AR user terminal, calculate the pose data of the AR user terminal in real time and detect obstacles around the target vehicle, obtain the panoramic image corresponding to the user's current viewpoint based on the real-time coordinates and pose data, and generate corresponding instruction animations and prompts based on the obstacles. Specifically, in step S30, generating corresponding indicator animations and prompts based on the obstacles involves: calculating the relative position of the obstacles in the AR user terminal coordinate system; determining the relative distance between the obstacles and the target vehicle based on the relative position; and generating indicator animations and prompts corresponding to the obstacles if the relative distance is less than or equal to a preset threshold. Otherwise, the detection of obstacles around the target vehicle continues. S40: The panoramic image and the directional animation are merged and presented to the AR user terminal.

2. The vehicle panoramic perspective display method according to claim 1, characterized in that, In step S10, before the AR user terminal enters the working mode, the following steps are also included: authenticating the current user's identity and configuring the user's driving preferences based on the authentication result; the AR user terminal is bound to at least one target vehicle and configured with at least one user.

3. The vehicle panoramic perspective display method according to claim 2, characterized in that, Step S10 further includes: defining an AR user terminal coordinate system, wherein the AR user terminal coordinate system has the center of the target vehicle as the origin, the width direction as the X-axis, the length direction as the Y-axis, and the height direction as the Z-axis.

4. The vehicle panoramic perspective display method according to claim 3, characterized in that, Step S40 further includes: When a user is in the vehicle environment, the user interacts with the panoramic image and directional animation through gestures, voice or touch; and the system interacts with the user through obstacle voice broadcasts; the gestures include at least clicking, grabbing and expanding; When a user is in a remote environment, the user uses voice to assist the driver in driving the vehicle, and the driver is in the target vehicle that is bound to the AR user terminal configured by the user.

5. A system employing the vehicle panoramic perspective display method as described in any one of claims 1-4, characterized in that, The system includes: Acquisition module: includes at least cameras respectively installed around the vehicle body, as well as at the bottom and top, the cameras being used to acquire images of the surrounding environment of the target vehicle; Storage module: Used to store user identity and corresponding driving preferences; Identification module: used to authenticate the user's identity in order to configure the user's driving preferences based on the user's identity in the storage module; Display module: used to display panoramic images and directional animations via an AR user terminal; the panoramic image includes a perspective view and a filled-in image of the vehicle's surrounding environment; First communication module: used to connect the AR user terminal and the target vehicle via WIFI for in-vehicle or remote communication; The second communication module is used to assist in-vehicle or remote communication via a microphone and a speaker; the microphone is used for user voice interaction; the speaker is used for obstacle warning voice announcements. Interaction module: When the user is in the vehicle environment, it is used for the user to interact with the panoramic image and instruction animation through gestures, voice or touch; and the second communication module interacts with the user through obstacle voice broadcast; the gestures include at least clicking, grabbing and expanding; When a user is in a remote environment, the system allows the user to drive the vehicle with voice assistance, and the driver is in the target vehicle that is bound to the AR user terminal configured by the user.

6. The system according to claim 5, characterized in that, The system also includes: Measurement module: used to measure the pose data of the AR user terminal in real time; Acquisition module: used to acquire the real-time coordinates of the AR user terminal, and obtain the panoramic image corresponding to the user's current viewpoint based on the real-time coordinates and the pose data; Detection module: Used to detect obstacles around the target vehicle in real time; The first generation module is used to perform optical flow tracking and texture mapping on multiple vehicle surrounding environment images acquired by the calibrated camera to generate panoramic images corresponding to the multiple vehicle surrounding environment images. The second generation module is used to calculate the relative position of the obstacle in the AR user terminal coordinate system, and generate the corresponding indicator animation and prompt instructions based on the relative position.

7. The system according to claim 6, characterized in that, The display module includes at least a first display area, a second display area, and a third display area, and each display area displays the instruction animation in coordination according to the relative position of the obstacle; The first and second display areas are used to display perspective images; The third display area is used to fill and display the vehicle surrounding environment image captured by the camera.

8. A storage medium, one of the computer-readable storage media, characterized in that, It stores a computer program, which, when executed by a processor, implements the vehicle panoramic perspective display method as described in claims 1-4.

Images