Method, device and vehicle for controlling a vehicle based on a fisheye camera

Abstract

This invention discloses a method, apparatus, and vehicle for controlling a vehicle based on a fisheye camera, relating to the field of automotive technology. One specific embodiment of the method includes: responding to a vehicle operation instruction, controlling an onboard fisheye camera to acquire images of the vehicle's surroundings; inputting the images of the vehicle's surroundings into a monocular depth estimation model suitable for the fisheye camera, the monocular depth estimation model outputting a corresponding three-dimensional image of the vehicle's surroundings; determining the distance between obstacles in the three-dimensional image of the vehicle's surroundings and the vehicle based on depth information in the image, marking and displaying the corresponding color in the image of the vehicle's surroundings, and sending a prompt message according to the distance between the obstacle and the vehicle. This embodiment can improve the environmental applicability and measurement accuracy of obstacle detection.

Description

Technical Field

[0001] This invention relates to the field of automotive technology, and in particular to a method, apparatus, and vehicle for controlling a vehicle based on a fisheye camera. Background Technology

[0002] A fisheye camera is an ultra-wide-angle lens with a field of view that can typically reach 180° or even more than 220°, enabling it to capture extremely wide scenes.

[0003] Vehicles typically have more than one fisheye camera, which is mounted around the vehicle body and works together. For example, fisheye cameras on a vehicle can be categorized by their installation location as front cameras, rear cameras, and left and right side cameras. The front camera is mounted on the front grille or vehicle emblem. The rear camera is mounted near the trunk handle. The left and right side cameras are mounted at the bottom of the exterior rearview mirrors.

[0004] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: radar is often used to detect the distance to obstacles during vehicle operation, but radar is greatly affected by the weather. The performance of radar drops significantly in adverse weather conditions such as rain, fog, and snow, resulting in false detections or missed detections. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, vehicle, device, and computer-readable medium for controlling a vehicle based on a fisheye camera, which can improve the environmental applicability and measurement accuracy of detecting vehicle obstacles.

[0006] A method for controlling a vehicle based on a fisheye camera includes: In response to vehicle operation instructions, control the onboard fisheye camera to capture images of the vehicle's surroundings; The vehicle perimeter image is input into a monocular depth estimation model suitable for fisheye cameras, and the monocular depth estimation model suitable for fisheye cameras outputs the corresponding three-dimensional image of the vehicle perimeter. Based on the depth information in the three-dimensional image around the vehicle, the distance between the obstacle and the vehicle in the image around the vehicle is determined, the distance is marked and displayed in the image around the vehicle with a color corresponding to the distance, and a prompt message is sent according to the distance between the obstacle and the vehicle.

[0007] After distortion correction is performed on multiple vehicle periphery images, the multiple corrected vehicle periphery images are stitched together to obtain a panoramic image of the vehicle periphery. The step of identifying and displaying the color corresponding to the distance in the image surrounding the vehicle includes: Display a panoramic image of the vehicle's surroundings, and identify and display the color corresponding to the distance in the panoramic image of the vehicle's surroundings; Preferably, the method further includes: in response to an image operation request, determining a local image of the angle corresponding to the image operation request in the panoramic image surrounding the vehicle; Display a partial image of the angle corresponding to the image operation request, wherein the distance is indicated by the color in the partial image.

[0008] The vehicle operation instructions include vehicle status operation instructions, vehicle environment change instructions, or vehicle prompt operation instructions. The method further includes: displaying in the vehicle surrounding image the vehicle operation corresponding to the vehicle status operation indicator, the environmental parameters corresponding to the vehicle environment change indicator, or the prompt parameters corresponding to the vehicle prompt operation indicator.

[0009] The method further includes; A depth estimation training dataset was established using images of obstacles captured by an onboard fisheye camera and distances between obstacles and the vehicle detected by onboard radar. A monocular depth estimation model is trained using the aforementioned depth estimation training dataset to obtain a monocular depth estimation model suitable for fisheye cameras.

[0010] Determining the distance between obstacles and the vehicle in the surrounding 3D image based on depth information includes: The monocular depth estimation model for fisheye cameras is deployed on the vehicle or cloud server. Based on the depth information in the three-dimensional image of the vehicle's surroundings output by the monocular depth estimation model for fisheye cameras on the vehicle or cloud server, the distance between obstacles and the vehicle in the image of the vehicle's surroundings is determined. or, The monocular depth estimation model applicable to fisheye cameras is deployed on both the vehicle and cloud servers; The vehicle receives a cloud-based 3D image of the vehicle's surroundings from a monocular depth estimation model for a fisheye camera on a cloud server. Based on the depth information in the cloud-based 3D image of the vehicle's surroundings, the distance between obstacles in the image and the vehicle is determined. or, The vehicle does not receive the cloud-based 3D image of the vehicle's surroundings output by the monocular depth estimation model for the fisheye camera on the cloud server. Instead, it receives the vehicle-side 3D image of the vehicle's surroundings output by the monocular depth estimation model for the fisheye camera on the vehicle side. Based on the depth information in the vehicle-side 3D image of the vehicle's surroundings, the distance between obstacles and the vehicle in the surrounding image is determined.

[0011] The method further includes: The monocular depth estimation model for the vehicle-side fisheye camera is updated based on the images of the vehicle's surroundings and the distances between obstacles and the vehicle detected by the vehicle-mounted radar. or, After updating the monocular depth estimation model for fisheye cameras in the cloud server based on the images of the vehicle's surroundings and the distance between obstacles and the vehicle detected by the vehicle-mounted radar, the cloud server updates the monocular depth estimation model for fisheye cameras on the vehicle side.

[0012] If a moving obstacle is detected in the image around the vehicle, the frequency of the vehicle-mounted fisheye camera acquiring images around the vehicle is increased, and the movement trend of the moving obstacle is determined based on the continuously acquired images around the vehicle. The image around the vehicle identifies and displays the color corresponding to the distance, and sends a prompt message according to the distance between the obstacle and the vehicle, including: The system identifies and displays the color corresponding to the distance and the color corresponding to the movement trend of the moving obstacle in the image surrounding the vehicle, and sends a prompt message according to the distance between the obstacle and the vehicle and the movement trend.

[0013] After the vehicle-mounted fisheye camera acquires images of the vehicle's surroundings, the system also includes: The vehicle perimeter image is input into a monocular depth estimation model for fisheye cameras deployed on a cloud server to obtain a cloud-based 3D image of the vehicle perimeter, and the vehicle perimeter image is input into a monocular depth estimation model for fisheye cameras deployed on the vehicle end to obtain a vehicle-side 3D image of the vehicle perimeter. If the depth difference between the cloud-based 3D image of the vehicle's surroundings and the vehicle-side 3D image of the vehicle's surroundings is greater than a depth difference threshold, then the monocular depth estimation model deployed on the vehicle side, which is suitable for fisheye cameras, is updated using the monocular depth estimation model deployed on the cloud server and suitable for fisheye cameras.

[0014] According to a second aspect of the present invention, an apparatus for controlling a vehicle based on a fisheye camera is provided, comprising: The control module is used to control the onboard fisheye camera to acquire images of the vehicle's surroundings in response to vehicle operation instructions; The input module is used to input the vehicle surrounding image into a monocular depth estimation model suitable for a fisheye camera, and the monocular depth estimation model suitable for a fisheye camera outputs the corresponding three-dimensional image of the vehicle surrounding. The display module is used to determine the distance between obstacles and the vehicle in the three-dimensional image around the vehicle based on the depth information in the image around the vehicle, identify and display the color corresponding to the distance in the image around the vehicle, and send prompt information according to the distance between the obstacle and the vehicle.

[0015] According to a third aspect of the present invention, a vehicle is provided, including the device for controlling the vehicle based on a fisheye camera as described above.

[0016] According to a fourth aspect of the present invention, an electronic device for controlling a vehicle based on a fisheye camera is provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0017] According to a fifth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0018] One embodiment of the above invention has the following advantages or beneficial effects: It controls an onboard fisheye camera to acquire images of the vehicle's surroundings. The images are then input into a monocular depth estimation model suitable for the fisheye camera to output a three-dimensional image of the vehicle's surroundings. Furthermore, the distance between obstacles in the surrounding images and the vehicle is determined, and the distance is identified and displayed in the surrounding images with a corresponding color. A prompt message is sent according to the distance between the obstacle and the vehicle. No additional equipment is required; the fisheye camera can directly detect the distance to obstacles. The fisheye camera is less affected by the environment, improving the environmental applicability and measurement accuracy for detecting obstacles around the vehicle.

[0019] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0020] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main process of a method for controlling a vehicle based on a fisheye camera according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a fisheye camera capturing obstacles according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the marking and display of distances in an image surrounding a vehicle according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the vehicle's surrounding image at the angle corresponding to the image operation request according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a monocular depth estimation model for fisheye cameras obtained by training according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the process of sending prompt information according to the distance to the obstacle and the moving speed according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the main structure of a vehicle control device based on a fisheye camera according to an embodiment of the present invention; Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0021] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] To improve the environmental applicability and measurement accuracy of vehicle obstacle detection, the following technical solutions from the embodiments of the present invention can be adopted.

[0023] See Figure 1 , Figure 1 This is a schematic diagram of the main flow of a method for controlling a vehicle based on a fisheye camera according to an embodiment of the present invention. A monocular depth estimation model suitable for fisheye cameras is used to determine the distance between obstacles and the vehicle in the surrounding image, and the distance is indicated by color. Figure 1 As shown in Figure 100, the specific steps include: S101, in response to vehicle operation instructions, controls the on-board fisheye camera to acquire images of the vehicle's surroundings.

[0024] Vehicle-mounted fisheye cameras, positioned in multiple directions around the vehicle, acquire images of the vehicle's surroundings upon receiving vehicle operation instructions. These instructions include vehicle status instructions, environmental change instructions, or vehicle alert instructions.

[0025] As an example, the vehicle status operation indicator can be operated by the vehicle or by the user. Vehicle operations include one or more of the following: vehicle power-on, vehicle power-off, vehicle switching from a non-driving state to a driving state, and vehicle speed falling below a threshold. When the vehicle is powered off, the vehicle status operation indicator controls the onboard fisheye camera to capture the last frame of the vehicle's surroundings to record the vehicle's parking environment. Switching from a non-driving state to a driving state is an example, such as shifting the vehicle gear from P to D. When the vehicle speed falls below a threshold, indicating the need to stop, the onboard fisheye camera is controlled to capture images of the vehicle's surroundings to promptly alert the user to obstacles in the vicinity.

[0026] User operations include one or more of the following: pressing the surround view system button, engaging reverse gear, activating the turn signal, and requesting surround view via the central control screen. Engaging reverse gear triggers the vehicle's fisheye camera to capture a reversing image. Activating the turn signal triggers the vehicle's fisheye camera to capture a side view image.

[0027] As an example, vehicle environment change indicators include changes in light and / or rain. For example, changes in light caused by entering a tunnel or nighttime.

[0028] As an example, vehicle prompts may include one or more of the following: airbag deployment, emergency braking, vehicle entry into a parking lot, and vehicle entry into complex road conditions.

[0029] S102. Input the vehicle surrounding image into the monocular depth estimation model suitable for fisheye cameras, and the monocular depth estimation model suitable for fisheye cameras outputs the corresponding three-dimensional image of the vehicle surrounding.

[0030] Vehicle-mounted fisheye cameras capture two-dimensional images of the vehicle's surroundings, making it impossible to directly obtain depth information. To obtain a corresponding three-dimensional image of the vehicle's surroundings, the images need to be input into a monocular depth estimation model suitable for fisheye cameras. This allows for the acquisition of a three-dimensional image using only a monocular fisheye camera.

[0031] S103. Based on the depth information in the three-dimensional image around the vehicle, determine the distance between the obstacle and the vehicle in the image around the vehicle, mark and display the corresponding color in the image around the vehicle, and send a prompt message according to the distance between the obstacle and the vehicle.

[0032] The vehicle's surrounding image includes obstacles, such as other vehicles. The 3D image of the vehicle's surroundings includes depth information of these obstacles. Based on the fisheye camera's position and the obstacle's depth information, the distance between the obstacle and the vehicle in the surrounding image can be determined. For example, the fisheye camera's location can be used as the vehicle's position, and the obstacle's depth information as the distance between the obstacle and the vehicle.

[0033] When displaying images of the vehicle's surroundings on the central control screen or other vehicle displays, the distance between obstacles and the vehicle is indicated by color.

[0034] See Figure 2 That is, 200. Figure 2This is a schematic diagram illustrating obstacle acquisition using a fisheye camera according to an embodiment of the present invention. The fisheye camera, positioned at the front grille, acquires images of the vehicle's surroundings in front of it. The images of the vehicle's surroundings include obstacles 201, 202, and 203 (e.g., obstacles are road surfaces or traffic cones in parking lots). Using the technical solution in this embodiment of the invention, it is determined that obstacle 201 is 10 meters away from the vehicle, obstacle 202 is 6 meters away, and obstacle 203 is 0.8 meters away.

[0035] The vehicle's central control screen or other displays show an image of the area around the vehicle. Red represents a distance of 0 to 1 meter; yellow represents a distance of 1 to 8 meters; and green represents a distance greater than 8 meters. Figure 2 In the diagram, red is represented by a dot matrix; yellow by a grid; and green by a vertical line.

[0036] As can be seen, when the image around the vehicle is displayed, obstacle 203 is marked in red, obstacle 202 is marked in yellow, and obstacle 201 is marked in green.

[0037] The system uses color-coded indicators to show the distance between the vehicle and obstacles around it. It can also send alerts based on the relationship between the distance to the obstacle and a distance threshold. For example, the distance threshold is 1 meter. If the distance to the obstacle is less than the threshold, an audible alert and a displayed icon are sent. The sound and icon then further alert the driver to the obstacle.

[0038] To remind the driver of the vehicle's operating status, the vehicle's surrounding image is displayed, along with vehicle operation instructions. These instructions include vehicle status operation instructions, vehicle environment change instructions, or vehicle prompt operation instructions. Furthermore, the surrounding image can also display the corresponding vehicle operation for the vehicle status operation instructions, the environmental parameters for the vehicle environment change instructions, or the prompt parameters for the vehicle prompt operation instructions.

[0039] In the embodiments of the present invention described above, an in-vehicle fisheye camera is used to acquire images of the vehicle's surroundings. Combined with a monocular depth estimation model suitable for fisheye cameras, depth information is added to each pixel of the vehicle's surroundings image to determine the distance between obstacles and the vehicle in the surroundings image. The distance is then identified and displayed in a color within the vehicle's surroundings image, and a prompt message is sent according to the distance between the obstacle and the vehicle.

[0040] Currently, depth information is mainly created from depth data obtained from LiDAR sensors, binocular stereo matching, or Structure from Motion (SfM). SfM is a computer vision technique that reconstructs the structure of a 3D scene from multiple 2D image sequences and estimates camera motion trajectories.

[0041] In embodiments of the present invention, depth information is determined directly using images of the vehicle's surroundings captured by a monocular fisheye camera. This has the following advantages: it is easy to collect, requiring no other equipment. Images captured by a monocular fisheye camera cover a wider range of scenes, which is beneficial for improving the model's generalization ability and scalability. The 3D image output by the model has a denser depth map than that acquired by LiDAR, and the computationally intensive stereo matching process is omitted. Since the fisheye camera can obtain the distance between the vehicle and obstacles, and is less affected by the environment, it improves the environmental applicability and measurement accuracy for detecting vehicles and obstacles, thus eliminating the need for radar.

[0042] See Figure 3 That is, 300. Figure 3 This is a schematic diagram illustrating the marking and display of distances in an image surrounding a vehicle, according to an embodiment of the present invention. Specifically, it includes the following steps: S301. After performing distortion correction on multiple vehicle perimeter images, stitch the multiple corrected vehicle perimeter images together to obtain a panoramic image of the vehicle perimeter.

[0043] Multiple fisheye cameras are used to collect images of the vehicle's surroundings. For example, a front fisheye camera, a left fisheye camera, a right fisheye camera, and a rear fisheye camera are used to collect images of the vehicle's surroundings in front, on the left, on the right, and behind.

[0044] Distortion correction is performed on the images surrounding the front, left, right, and rear vehicles. As an example, distortion correction can be achieved using a distortion correction mapping table.

[0045] Then, the corrected images of the front, left, right, and rear sides of the vehicle are stitched together to obtain a panoramic image of the vehicle's surroundings. As an example, features and matching features are extracted from the corrected images of the vehicle's surroundings. After image alignment and registration, stitching seams are processed and images are fused sequentially to obtain the panoramic image of the vehicle's surroundings. This panoramic image includes a 360-degree panoramic view.

[0046] S302, Display a panoramic image of the vehicle's surroundings, identify and display colors corresponding to distances in the panoramic image of the vehicle's surroundings, and in response to an image operation request, determine a local image of the angle corresponding to the image operation request in the panoramic image of the vehicle's surroundings.

[0047] By using the distances between obstacles and the vehicle in the surrounding image, and the correspondence between distance and color, the colors of obstacles are marked in the panoramic image surrounding the vehicle. The distance between obstacles and the vehicle can be determined by their colors in the panoramic image surrounding the vehicle.

[0048] For the driver, it's necessary to observe obstacles from different angles. The driver adjusts the angle of the vehicle image by sending an image operation request. As an example, the image operation request can be a voice operation request or a touch screen operation request. The driver moves the viewpoint of the vehicle image on the screen to send the corresponding touch screen operation request.

[0049] See Figure 4 That is, 400. Figure 4 This is a schematic diagram of the vehicle's surrounding image at the corresponding angle according to an embodiment of the present invention. Figure 4 The vertical line section represents the original image of the area surrounding the vehicle, while the grid section represents a local image of the area around the vehicle based on the angle requested by the image manipulation request. This local image is a portion of the panoramic image of the vehicle's surroundings. The angle can be adjusted via the image manipulation request to display the local image corresponding to that angle, thereby increasing the observation range.

[0050] S303. Display a partial image of the angle corresponding to the image operation request, with distances indicated by color in the partial image.

[0051] Display a partial image corresponding to the angle of the image operation request on the vehicle's central control display screen or other displays within the vehicle. Determine the distance between the obstacle and the vehicle by identifying the color of obstacles in the partial image.

[0052] As an example, the display screen can simultaneously show the original image of the vehicle's surroundings and a partial image of the angle corresponding to the image operation request. Obstacles can be accurately identified by comparing the two images.

[0053] exist Figure 3 In one embodiment, based on the image operation request sent by the driver, local images from different angles are displayed to improve the field of view.

[0054] See Figure 5 That is, 500. Figure 5 This is a schematic diagram of a monocular depth estimation model for fisheye cameras obtained through training according to an embodiment of the present invention.

[0055] Figure 1 In the diagram, A1 and A2 are both training data. A1 is an image of an obstacle captured by the vehicle's fisheye camera. A2 is the distance between the obstacle and the vehicle detected by the vehicle's radar. Both A1 and A2 are real data collected while the vehicle is in motion. As an example, with the vehicle in reverse gear, the rear-mounted fisheye camera captures images of the vehicle's surroundings behind it, and the vehicle's radar detects the distance between obstacles and the vehicle behind it. The images of the vehicle's surroundings captured by the rear-mounted fisheye camera and the distances between obstacles and the vehicle detected by the vehicle's radar are stored and uploaded. The above images and distances correspond to each other. This data can be used as training data.

[0056] In addition, during normal vehicle operation, the onboard fisheye cameras in four directions simultaneously collect images of the vehicle's surroundings, and the onboard radar detects the distance between the vehicle and obstacles nearby. These images and distances can be used as training data.

[0057] The training data includes images of the vehicle's surroundings captured by one or more onboard fisheye cameras and the distance between obstacles and the vehicle detected by the vehicle's radar. Figure 5 The A1 and A2 in the dataset are used to construct the depth estimation training dataset B.

[0058] The purpose of the depth estimation training dataset B is to train the monocular depth estimation model M. As an example, the monocular depth estimation model M is a model built based on a self-supervised encoder-decoder model or a Transformer.

[0059] Compared to ordinary cameras, fisheye cameras have greater barrel distortion. Therefore, the images of the vehicle's surroundings captured by the vehicle-mounted fisheye camera, as well as the distance detected by the vehicle-mounted radar, need to be used as input data for the monocular depth estimation model M in order to train a monocular depth estimation model suitable for fisheye cameras.

[0060] In embodiments of the present invention, a monocular depth estimation model suitable for fisheye cameras is used to process images surrounding the vehicle. This monocular depth estimation model can be deployed on the vehicle itself or on a cloud server. That is, images surrounding the vehicle can be processed either on the vehicle itself or on a cloud server.

[0061] When a monocular depth estimation model for fisheye cameras is deployed on a vehicle or cloud server, the images of the vehicle's surroundings captured by the fisheye camera can be sent to the monocular depth estimation model for fisheye cameras on the vehicle or server for processing.

[0062] When a monocular depth estimation model for fisheye cameras is deployed on both the vehicle and cloud servers, the model deployed on the cloud server can be prioritized because it has more iterations and higher accuracy in determining distance.

[0063] Specifically, if the vehicle receives a cloud-based 3D image of the vehicle's surroundings from the monocular depth estimation model for the fisheye camera within a preset time period, it indicates that the vehicle is communicating normally with the cloud server and can directly use the depth information in the cloud-based 3D image of the vehicle's surroundings to determine the distance between obstacles and the vehicle in the surrounding image.

[0064] If the vehicle does not receive the cloud-based 3D image of its surroundings from the monocular depth estimation model used for the fisheye camera within a preset time period, it indicates that the vehicle and the cloud server are having difficulty exchanging data normally. For example, this could be due to poor communication signal quality when the vehicle is driving in an underground parking garage or tunnel. To provide timely distance feedback, the vehicle-side 3D image of its surroundings, output by the monocular depth estimation model of the fisheye camera deployed on the vehicle, is used to determine the distance between obstacles in the surrounding image and the vehicle.

[0065] In one embodiment of the present invention, a monocular depth estimation model for fisheye cameras deployed on the vehicle can be updated using images of the vehicle's surroundings and the distance between obstacles and the vehicle detected by onboard radar.

[0066] As an example, the training data includes images of the vehicle's surroundings captured by the vehicle's fisheye camera, and the distances between obstacles and the vehicle corresponding to these images. Using this training data, the monocular depth estimation model for the vehicle's fisheye camera is updated. The images of the vehicle's surroundings captured by the fisheye camera mainly involve common scenarios for the vehicle, such as parking lots, driving roads, and driving time.

[0067] By utilizing data collected from the vehicle's fisheye camera and onboard radar, the monocular depth estimation model applicable to the fisheye camera on the vehicle side is updated, improving the model's personalized adaptability.

[0068] In one embodiment of the present invention, the monocular depth estimation model applicable to the fisheye camera on the vehicle side is updated via a cloud server. Specifically, the vehicle uploads data collected by the fisheye camera and onboard radar. The cloud server uses the received data from the fisheye camera and onboard radar as training data to update the monocular depth estimation model applicable to the fisheye camera on the cloud server. Then, the cloud server pushes the updated monocular depth estimation model applicable to the fisheye camera to the vehicle side, thereby updating the model on the vehicle side.

[0069] To improve the effectiveness of the training data, the data uploaded by this vehicle can be processed as follows: Add timestamps, location information, and vehicle status parameters to the images of the vehicle's surroundings captured by the vehicle-mounted fisheye camera and the distance between the vehicle and obstacles detected by the vehicle-mounted radar corresponding to the images of the vehicle's surroundings.

[0070] Timestamps are used to align vehicle surrounding images with distances accurate to milliseconds. Location information is used to label the positional features of the vehicle surrounding images. The cloud server uses the location information to determine the corresponding scene, such as a parking lot or driving road. Vehicle state parameters include speed and direction of travel. The cloud server uses these parameters to process the vehicle surrounding images. As an example, the cloud server uses the driving mode to filter vehicle surrounding images captured during reversing, improving the accuracy of the model in determining depth information at slow speeds. Similarly, the cloud server uses the driving mode to filter vehicle surrounding images captured outside of reversing. During normal vehicle movement, multiple onboard fisheye cameras capture images of the vehicle surroundings. Given the large number of images and the high vehicle speed, a tracking algorithm is used to associate detection boxes in consecutive frames to accurately identify each obstacle. The model is then trained using the onboard radar detection results.

[0071] Cloud servers offer greater computing power than vehicle-side systems, and their training data encompasses diverse data from various vehicle models, scenarios, and time periods. Models updated using this training data exhibit greater robustness and are applicable to a wider range of scenarios.

[0072] See Figure 6 , Figure 6 This is a schematic diagram illustrating the process of sending prompt information based on obstacle distance and movement speed according to an embodiment of the present invention. Specifically, it includes the following steps: S601. When a moving obstacle is detected in the image around the vehicle, the frequency of the vehicle-mounted fisheye camera acquiring images around the vehicle is increased, and the movement trend of the moving obstacle is determined based on the continuously acquired images around the vehicle.

[0073] Obstacles in vehicle perimeter images include both moving and stationary obstacles. Moving obstacles often have a greater impact on vehicle safety. To monitor moving obstacles, when an obstacle in the vehicle perimeter image is identified as moving, the frequency of one or more vehicle-mounted fisheye cameras capturing vehicle perimeter images is increased. As an example, the frequency of all vehicle-mounted fisheye cameras capturing vehicle perimeter images is increased. Alternatively, to reduce data volume, the frequency of vehicle perimeter images captured by the vehicle-mounted fisheye cameras specifically targeting moving obstacles can be increased. For example, if a rear vehicle-mounted fisheye camera captures a moving obstacle, the frequency of its vehicle perimeter images can be increased.

[0074] The time interval between two frames is calculated using timestamps from images surrounding the vehicle. Bits of moving obstacles are removed and factored by the time interval to obtain the obstacle's speed, thus determining its movement trend. As an example, movement trends include moving away from and towards the vehicle.

[0075] S602. Identify and display the colors corresponding to the distance and the movement trend of moving obstacles in the image around the vehicle, and send prompt information according to the distance and movement trend of the obstacles to the vehicle.

[0076] In addition to displaying the colors corresponding to the distance to the vehicle's surrounding image, the system can also identify and display the colors corresponding to the movement trend of moving obstacles. For example, distance-related colors could be red, yellow, or green. Obstacles moving away from the vehicle would be marked in blue, and those moving closer would be marked in orange. The prompts would include both distance values ​​and movement trends. Alternatively, prompts could be delivered via voice broadcast.

[0077] exist Figure 6 In one embodiment, when a moving obstacle is identified, a prompt is given using distance and movement trend.

[0078] In one embodiment of the present invention, considering that the update speed of the monocular depth estimation model for fisheye cameras deployed on the vehicle is relatively slow, in order to improve the accuracy of depth estimation, a model deployed on a cloud server is used to verify the model deployed on the vehicle. If the verification model fails, the model deployed on the cloud server is used to update the model deployed on the vehicle.

[0079] Specifically, the images of the vehicle's surroundings are input into a monocular depth estimation model for fisheye cameras deployed on a cloud server, and a monocular depth estimation model for fisheye cameras deployed on the vehicle, respectively, to obtain 3D images of the vehicle's surroundings in the cloud and on the vehicle.

[0080] Compare the depth differences of obstacle regions in the cloud-based 3D image of the vehicle's surroundings and the on-device 3D image of the vehicle's surroundings. As an example, the obstacle region includes the area within 5 meters of the obstacle. The depth difference is equal to the average depth difference of each pixel in the obstacle region.

[0081] If the depth difference is greater than the depth difference threshold, it means that the model deployed on the vehicle needs to be updated. In this case, the monocular depth estimation model for fisheye cameras deployed on the cloud server is sent to the vehicle to update the model on the vehicle.

[0082] In the embodiments of the present invention described above, a monocular depth estimation model is trained using images of the vehicle's surroundings acquired by an in-vehicle fisheye camera, thereby improving the model's relevance. The distance to obstacles is displayed by color to warn of potential collision risks.

[0083] See Figure 7 , Figure 7 This is a schematic diagram of the main structure of a vehicle control device based on a fisheye camera according to an embodiment of the present invention. The device can implement a method for controlling a vehicle based on a fisheye camera, such as... Figure 7As shown in Figure 700, the device for controlling a vehicle based on a fisheye camera specifically includes: Control module 701 is used to control the on-board fisheye camera to acquire images of the vehicle's surroundings in response to vehicle operation instructions; The input module 702 is used to input the vehicle periphery image into a monocular depth estimation model suitable for a fisheye camera, and the monocular depth estimation model suitable for a fisheye camera outputs the corresponding vehicle periphery three-dimensional image. The display module 703 is used to determine the distance between obstacles and the vehicle in the three-dimensional image around the vehicle based on the depth information in the image around the vehicle, mark and display the color corresponding to the distance in the image around the vehicle, and send prompt information according to the distance between the obstacle and the vehicle.

[0084] In one embodiment of the present invention, the display module 703 is specifically used to perform distortion correction on multiple vehicle periphery images and then stitch together multiple corrected vehicle periphery images to obtain a panoramic image of the vehicle periphery. Display a panoramic image of the vehicle's surroundings, and identify and display the color corresponding to the distance in the panoramic image of the vehicle's surroundings; In response to an image manipulation request, a local image corresponding to the angle of the image manipulation request is determined from the panoramic image around the vehicle. Display a partial image of the angle corresponding to the image operation request, wherein the distance is indicated by the color in the partial image.

[0085] In one embodiment of the present invention, the vehicle operation indication includes a vehicle status operation indication, a vehicle environment change indication, or a vehicle prompt operation indication; The display module 703 is used to display in the vehicle surrounding image the vehicle operation corresponding to the vehicle status operation instruction, the environmental parameters corresponding to the vehicle environment change instruction, or the prompt parameters corresponding to the vehicle prompt operation instruction.

[0086] In one embodiment of the present invention, the input module 702 is used to establish a depth estimation training dataset by acquiring images of obstacles with an onboard fisheye camera and measuring the distance between obstacles and the vehicle detected by onboard radar. A monocular depth estimation model is trained using the aforementioned depth estimation training dataset to obtain a monocular depth estimation model suitable for fisheye cameras.

[0087] In one embodiment of the present invention, a display module 703 is used to deploy the monocular depth estimation model suitable for fisheye cameras on a vehicle or cloud server, and determine the distance between obstacles and the vehicle in the three-dimensional image of the vehicle's surroundings based on the depth information in the three-dimensional image of the vehicle's surroundings output by the monocular depth estimation model suitable for fisheye cameras on the vehicle or cloud server. or, The monocular depth estimation model applicable to fisheye cameras is deployed on both the vehicle and cloud servers; The vehicle receives a cloud-based 3D image of the vehicle's surroundings from a monocular depth estimation model for a fisheye camera on a cloud server. Based on the depth information in the cloud-based 3D image of the vehicle's surroundings, the distance between obstacles in the image and the vehicle is determined. or, The vehicle does not receive the cloud-based 3D image of the vehicle's surroundings output by the monocular depth estimation model for the fisheye camera on the cloud server. Instead, it receives the vehicle-side 3D image of the vehicle's surroundings output by the monocular depth estimation model for the fisheye camera on the vehicle side. Based on the depth information in the vehicle-side 3D image of the vehicle's surroundings, the distance between obstacles and the vehicle in the surrounding image is determined.

[0088] In one embodiment of the present invention, the display module 703 is used to update the monocular depth estimation model of the vehicle end suitable for the fisheye camera with the vehicle surrounding image and the distance between the vehicle and the obstacle detected by the vehicle radar. or, After updating the monocular depth estimation model for fisheye cameras in the cloud server based on the images of the vehicle's surroundings and the distance between obstacles and the vehicle detected by the vehicle-mounted radar, the cloud server updates the monocular depth estimation model for fisheye cameras on the vehicle side.

[0089] In one embodiment of the present invention, the display module 703 is used to increase the frequency of the vehicle-mounted fisheye camera acquiring images of the vehicle's surroundings when a moving obstacle is detected in the image of the vehicle's surroundings, and to determine the movement trend of the moving obstacle based on the continuously acquired images of the vehicle's surroundings. The image around the vehicle identifies and displays the color corresponding to the distance, and sends a prompt message according to the distance between the obstacle and the vehicle, including: The system identifies and displays the color corresponding to the distance and the color corresponding to the movement trend of the moving obstacle in the image surrounding the vehicle, and sends a prompt message according to the distance between the obstacle and the vehicle and the movement trend.

[0090] In one embodiment of the present invention, the display module 703 is used to input the vehicle periphery image into a monocular depth estimation model suitable for fisheye cameras deployed on a cloud server to obtain a cloud-based three-dimensional image of the vehicle periphery, and input the vehicle periphery image into a monocular depth estimation model suitable for fisheye cameras deployed on the vehicle end to obtain a vehicle-side three-dimensional image of the vehicle periphery. If the depth difference between the cloud-based 3D image of the vehicle's surroundings and the vehicle-side 3D image of the vehicle's surroundings is greater than a depth difference threshold, then the monocular depth estimation model deployed on the vehicle side, which is suitable for fisheye cameras, is updated using the monocular depth estimation model deployed on the cloud server and suitable for fisheye cameras.

[0091] The vehicle control device based on a fisheye camera in this embodiment of the invention can be applied to vehicles.

[0092] Figure 8 An exemplary system architecture 800 is shown, which can be applied to a method or apparatus for controlling a vehicle based on a fisheye camera according to embodiments of the present invention.

[0093] like Figure 8 As shown, the vehicle system architecture 800 may include various systems, such as a driving control system 801, a power system 802, a sensor system 803, a control system 804, a lane change assist system 805, one or more peripheral devices 806, a power supply 807, a computer system 808, and a user interface 809. The method for controlling a vehicle based on a fisheye camera provided in this embodiment can be implemented through interaction with the aforementioned systems, or through control of the systems by external devices, or through operation of the systems by a robot driving the vehicle. Optionally, the vehicle system architecture 800 may include more or fewer systems, and each system may include multiple components. Furthermore, each system and component of the vehicle system architecture 800 may be interconnected via wired or wireless means.

[0094] The vehicle system architecture 800 includes a driving control system 801, which can be in a fully or partially automated driving mode. For example, the driving control system 801 can automatically control the vehicle's movement based on control signals or control commands without interaction with a human, external devices, or a robot driving the vehicle.

[0095] The power system 802 may include components that provide power for the vehicle. For example, the power system 802 may include an engine, an energy source, a transmission, wheels, tires, etc.

[0096] The sensor system 803 may include sensors for sensing the vehicle's surrounding environment (such as sensors for detecting the presence of obstacles) and pressure sensors for sensing the presence of passengers in the seats. Examples include a positioning system (which may be a Global Positioning System (GPS), BeiDou Navigation Satellite System, or other positioning systems), radar, a laser rangefinder, an inertial measurement unit (IMU), and cameras. The positioning system can be used to determine the vehicle's geographical location. The IMU is used to sense changes in the vehicle's position and orientation based on inertial acceleration. In one embodiment, the IMU may be a combination of an accelerometer and a gyroscope. The radar can use radio signals to sense objects in the vehicle's surrounding environment. In some embodiments, in addition to sensing objects, the radar can also be used to sense the speed and / or direction of travel of objects.

[0097] To detect environmental information and objects outside the vehicle, cameras can be configured at appropriate locations on the vehicle's exterior. For example, to acquire environmental images of the vehicle's sides, a camera can be mounted on the side mirror. The camera can be a still or video camera.

[0098] The control system 804 may include software systems for implementing vehicle driving control, such as systems for analyzing the vehicle's surrounding environment, pretensioning seat belts, route planning, obstacle avoidance, and image analysis. The control system 804 may also include hardware systems such as an accelerator, steering wheel system, seat belt system, airbag system, and peripheral devices (such as projection equipment and displays). Furthermore, the control system 804 may add or replace components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0099] In addition, the control system 804 can also interact with external sensors, other autonomous driving devices, other computer systems, or users via peripheral devices 806. Peripheral devices 806 may include wireless communication systems, on-board computers, microphones and / or speakers, cameras, and projectors, etc.

[0100] In some embodiments, peripheral device 806 provides a means for user interaction with the control system 804 via a user interface. For example, an onboard computer may provide information to a user of the vehicle. The user interface may also operate the onboard computer to receive user input. The onboard computer may be operated via a touchscreen. In other cases, peripheral device may provide a means for communicating with other devices located within the vehicle. For example, a microphone may receive audio (e.g., voice commands or other audio input) from a user of the control system. Similarly, a speaker may output audio to a user of the control system.

[0101] Wireless communication systems can communicate wirelessly with one or more devices, either directly or via a communication network. For example, wireless communication systems can use networks such as cellular networks, WiFi, and wireless local area networks (WLANs), or they can use infrared links, Bluetooth, or ZigBee to communicate directly with devices. Other wireless protocols include those used in various autonomous driving communication systems.

[0102] The power source 807 can provide power to various components of the vehicle. The power source 807 can be a rechargeable lithium-ion battery or a lead-acid battery.

[0103] The computer system 808 controls some or all of the vehicle control functions for ramp entry scenarios. The computer system 808 may include at least one processor that executes instructions stored in a non-transitory computer-readable medium such as memory. The computer system 808 provides the aforementioned control system with execution code that implements vehicle control for ramp entry scenarios.

[0104] The processor can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor can be a special-purpose device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Those skilled in the art will understand that the processor, computer, or memory can actually include multiple processors, computers, or memories that may or may not be stored in the same physical housing. For example, memory can be a hard disk drive or other storage media located in a housing different from that of a computer. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, may each have their own processor that performs only determinations related to the component's specific function.

[0105] User interface 809 is used to provide information to or receive information from users of the vehicle. Optionally, user interface 809 may include one or more input / output devices within a set of peripheral devices 806, such as wireless communication systems, on-board computers, microphones, and speakers.

[0106] It should be understood that the components described above are merely an example. In actual applications, components in the various modules or systems mentioned above may be added or removed as needed. Figure 8 This should not be construed as a limitation on the embodiments of this application.

[0107] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0108] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0109] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0110] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0111] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a control module, an input module, and a display module. The names of these modules do not necessarily limit the module itself; for example, a control module may also be described as "for controlling an onboard fisheye camera to acquire images of the vehicle's surroundings in response to vehicle operation instructions."

[0114] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: In response to vehicle operation instructions, control the onboard fisheye camera to capture images of the vehicle's surroundings; The vehicle perimeter image is input into a monocular depth estimation model suitable for fisheye cameras, and the monocular depth estimation model suitable for fisheye cameras outputs the corresponding three-dimensional image of the vehicle perimeter. Based on the depth information in the three-dimensional image around the vehicle, the distance between the obstacle and the vehicle in the image around the vehicle is determined, the distance is marked and displayed in the image around the vehicle with a color corresponding to the distance, and a prompt message is sent according to the distance between the obstacle and the vehicle.

[0115] According to the technical solution of this invention, an onboard fisheye camera is controlled to acquire images of the vehicle's surroundings. These images are then input into a monocular depth estimation model suitable for the fisheye camera to output a three-dimensional image of the vehicle's surroundings. Furthermore, the distance between obstacles in the surrounding images and the vehicle is determined, and the distance is identified and displayed in the surrounding images with a corresponding color. A prompt message is sent according to the distance between the obstacle and the vehicle. This method allows for direct detection of obstacle distances using a fisheye camera without the need for additional equipment. Fisheye cameras are less affected by environmental conditions, improving the environmental applicability and measurement accuracy for detecting obstacles around the vehicle.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention. It should be noted that the acquisition, storage, and application of user personal information involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for controlling a vehicle based on a fisheye camera, characterized in that, include: In response to vehicle operation instructions, control the onboard fisheye camera to capture images of the vehicle's surroundings; The vehicle perimeter image is input into a monocular depth estimation model suitable for fisheye cameras, and the monocular depth estimation model suitable for fisheye cameras outputs the corresponding three-dimensional image of the vehicle perimeter. Based on the depth information in the three-dimensional image around the vehicle, the distance between the obstacle and the vehicle in the image around the vehicle is determined, the distance is marked and displayed in the image around the vehicle with a color corresponding to the distance, and a prompt message is sent according to the distance between the obstacle and the vehicle.

2. The method for controlling a vehicle based on a fisheye camera according to claim 1, characterized in that, After the vehicle-mounted fisheye camera acquires images of the vehicle's surroundings, the system also includes: After distortion correction is performed on multiple vehicle periphery images, the multiple corrected vehicle periphery images are stitched together to obtain a panoramic image of the vehicle periphery. The step of identifying and displaying the color corresponding to the distance in the image surrounding the vehicle includes: Display a panoramic image of the vehicle's surroundings, and identify and display the color corresponding to the distance in the panoramic image of the vehicle's surroundings; Preferably, the method further includes: in response to an image operation request, determining a local image of the angle corresponding to the image operation request in the panoramic image surrounding the vehicle; Display a partial image of the angle corresponding to the image operation request, wherein the distance is indicated by the color in the partial image.

3. The method for controlling a vehicle based on a fisheye camera according to claim 1, characterized in that, The vehicle operation instructions include vehicle status operation instructions, vehicle environment change instructions, or vehicle prompt operation instructions. The method further includes: displaying in the vehicle surrounding image the vehicle operation corresponding to the vehicle status operation indicator, the environmental parameters corresponding to the vehicle environment change indicator, or the prompt parameters corresponding to the vehicle prompt operation indicator.

4. The method for controlling a vehicle based on a fisheye camera according to claim 1, characterized in that, The method further includes; A depth estimation training dataset was established using images of obstacles captured by an onboard fisheye camera and distances between obstacles and the vehicle detected by onboard radar. A monocular depth estimation model is trained using the aforementioned depth estimation training dataset to obtain a monocular depth estimation model suitable for fisheye cameras.

5. The method for controlling a vehicle based on a fisheye camera according to claim 1, characterized in that, Determining the distance between obstacles and the vehicle in the surrounding 3D image based on depth information includes: The monocular depth estimation model for fisheye cameras is deployed on the vehicle or cloud server. Based on the depth information in the three-dimensional image of the vehicle's surroundings output by the monocular depth estimation model for fisheye cameras on the vehicle or cloud server, the distance between obstacles and the vehicle in the image of the vehicle's surroundings is determined. or, The monocular depth estimation model applicable to fisheye cameras is deployed on both the vehicle and cloud servers; The vehicle receives a cloud-based 3D image of the vehicle's surroundings from a monocular depth estimation model for a fisheye camera, sent by a cloud server. Based on the depth information in the cloud-based 3D image of the vehicle's surroundings, the distance between obstacles and the vehicle in the image is determined. or, If the vehicle does not receive the cloud-based 3D image of the vehicle's surroundings from the monocular depth estimation model for the fisheye camera sent by the cloud server, the vehicle directly uses the 3D image of the vehicle's surroundings output by the monocular depth estimation model for the fisheye camera it deploys, and determines the distance between the vehicle and obstacles in the 3D image of the vehicle's surroundings based on the depth information in the 3D image of the vehicle's surroundings.

6. The method for controlling a vehicle based on a fisheye camera according to claim 1, characterized in that, The method further includes: The monocular depth estimation model for the vehicle-side fisheye camera is updated based on the images of the vehicle's surroundings and the distances between obstacles and the vehicle detected by the vehicle-mounted radar. or, After updating the monocular depth estimation model for fisheye cameras in the cloud server based on the images of the vehicle's surroundings and the distance between obstacles and the vehicle detected by the vehicle-mounted radar, the cloud server updates the monocular depth estimation model for fisheye cameras on the vehicle side.

7. The method for controlling a vehicle based on a fisheye camera according to claim 1, characterized in that, Also includes: If a moving obstacle is detected in the image around the vehicle, the frequency of the vehicle-mounted fisheye camera acquiring images around the vehicle is increased, and the movement trend of the moving obstacle is determined based on the continuously acquired images around the vehicle. The image around the vehicle identifies and displays the color corresponding to the distance, and sends a prompt message according to the distance between the obstacle and the vehicle, including: The system identifies and displays the color corresponding to the distance and the color corresponding to the movement trend of the moving obstacle in the image surrounding the vehicle, and sends a prompt message according to the distance between the obstacle and the vehicle and the movement trend.

8. The method for controlling a vehicle based on a fisheye camera according to any one of claims 1 to 7, characterized in that, After the vehicle-mounted fisheye camera acquires images of the vehicle's surroundings, the system also includes: The vehicle perimeter image is input into a monocular depth estimation model for fisheye cameras deployed on a cloud server to obtain a cloud-based 3D image of the vehicle perimeter, and the vehicle perimeter image is input into a monocular depth estimation model for fisheye cameras deployed on the vehicle end to obtain a vehicle-side 3D image of the vehicle perimeter. If the depth difference between the cloud-based 3D image of the vehicle's surroundings and the vehicle-side 3D image of the vehicle's surroundings is greater than a depth difference threshold, then the monocular depth estimation model deployed on the vehicle side, which is suitable for fisheye cameras, is updated using the monocular depth estimation model deployed on the cloud server and suitable for fisheye cameras.

9. A device for controlling a vehicle based on a fisheye camera, characterized in that, include: The control module is used to control the onboard fisheye camera to acquire images of the vehicle's surroundings in response to vehicle operation instructions; The input module is used to input the vehicle surrounding image into a monocular depth estimation model suitable for a fisheye camera, and the monocular depth estimation model suitable for a fisheye camera outputs the corresponding three-dimensional image of the vehicle surrounding. The display module is used to determine the distance between obstacles and the vehicle in the three-dimensional image around the vehicle based on the depth information in the image around the vehicle, identify and display the color corresponding to the distance in the image around the vehicle, and send prompt information according to the distance between the obstacle and the vehicle.

10. A vehicle, characterized in that, Includes the device for controlling a vehicle based on a fisheye camera as described in claim 9.

Images