Method, device, controller, vehicle and program product for displaying an obstacle

Abstract

The invention relates to a method, an apparatus, a controller, a vehicle and a program product for displaying obstacles. The method comprises the following steps: acquiring an environment image acquired by a camera of a vehicle and sensing information of a radar of the vehicle; the method further includes determining a virtual object of the obstacle around the vehicle based on the perceptual information. In addition, the method includes displaying a vehicle image of the vehicle, the environment image, and a virtual object of the obstacle on a display screen. By means of the method for displaying the obstacle, the environment image and the virtual object of the obstacle can be displayed on the display screen at the same time, a user can determine the obstacle around the vehicle more clearly through the content displayed on the display screen, and therefore the user experience can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle technology, and more specifically, to methods, apparatus, controllers, vehicles, and program products for displaying obstacles. Background Technology

[0002] With economic development, vehicles are becoming increasingly common in daily life. While driving, users need to constantly observe and pay attention to their surroundings to avoid traffic violations or accidents. However, blind spots exist in this observation; for example, due to the vehicle itself obstructing view, users may not be able to see objects nearby. This can affect driving decisions and, consequently, driving safety.

[0003] As vehicles become increasingly intelligent, more and more vehicles are equipped with cameras. These cameras capture images of the vehicle's surroundings, which can then be presented to the user through in-vehicle displays and other devices, allowing the user to gain a more comprehensive understanding of the true state of their environment. However, in some cases, the results of the in-vehicle cameras' capture or perception of the environment may be inaccurate. Summary of the Invention

[0004] This disclosure provides a method, apparatus, controller, vehicle, and program product for displaying obstacles.

[0005] In a first aspect of this disclosure, a method for displaying obstacles is provided, the method comprising acquiring environmental images captured by a vehicle's camera and perception information from the vehicle's radar. The method further comprises determining virtual objects of obstacles around the vehicle based on the perception information. Furthermore, the method comprises displaying a vehicle image, an environmental image, and the virtual objects of obstacles on a display screen.

[0006] In a second aspect of this disclosure, an apparatus for displaying obstacles is provided. The apparatus includes a first acquisition unit configured to acquire environmental images captured by a vehicle's camera. The apparatus also includes a second acquisition unit configured to acquire perception information from the vehicle's radar. The apparatus further includes an image determination unit configured to determine virtual objects of obstacles around the vehicle based on the perception information. Additionally, the apparatus includes a display unit configured to display vehicle images, the environment, and the virtual objects of obstacles.

[0007] In a third aspect of this disclosure, a controller is provided. The controller may include: at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the controller to perform the methods provided according to the first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a vehicle is provided. The vehicle includes a controller provided according to a third aspect of this disclosure.

[0009] In a fifth aspect of this disclosure, a machine-readable storage medium is provided. The machine-readable storage medium stores computer-executable instructions, which are executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0010] In a sixth aspect of this disclosure, a computer program product is provided. This computer program product may include computer-executable instructions that, when executed, cause a computer to perform the steps of the method provided according to a first aspect of this disclosure.

[0011] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0013] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0014] Figure 2A A schematic diagram of a scene in which environmental images are captured by a camera according to some embodiments of the present disclosure is shown;

[0015] Figure 2B A schematic diagram illustrating the content displayed on a display screen according to some embodiments of the present disclosure is shown;

[0016] Figure 3 A flowchart of a method for displaying obstacles according to some embodiments of the present disclosure is shown;

[0017] Figure 4 A schematic diagram of a scene for displaying obstacles according to some embodiments of the present disclosure is shown;

[0018] Figure 5 A schematic diagram of perceived information according to some embodiments of the present disclosure is shown;

[0019] Figure 6 A schematic diagram illustrating the content displayed on a display screen according to some embodiments of the present disclosure is shown;

[0020] Figure 7 A schematic diagram showing yet another content displayed on a display screen according to some embodiments of the present disclosure;

[0021] Figure 8 A flowchart of a method for displaying obstacles according to some embodiments of the present disclosure is shown;

[0022] Figure 9 A block diagram of an apparatus for displaying obstacles according to some embodiments of the present disclosure is shown; and

[0023] Figure 10 A block diagram of a controller that can implement several embodiments of the present disclosure is shown.

[0024] Generally, the same reference numerals are used throughout the drawings and in the accompanying specific embodiments to denote the same or similar parts. The drawings are not necessarily drawn to scale. The dimensions of parts or regions in the drawings may be enlarged for illustrative purposes. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] In this disclosure, the term "vehicle" (sometimes simply referred to as "car") is used in a broad sense, encompassing transportation vehicles (such as cars, trucks, motorcycles, airplanes, trains, ships, etc.), industrial vehicles (such as forklifts, trailers, tractors, etc.), engineering vehicles (such as excavators, bulldozers, cranes, etc.), agricultural equipment (such as lawnmowers, harvesters, etc.), amusement equipment, toy vehicles, etc. This disclosure does not specifically limit the type of vehicle. In the embodiments of this disclosure, the vehicle may be equipped with a camera, which can capture images of the surrounding environment. In the embodiments and accompanying drawings of this disclosure, a car is used as an example to illustrate the solution provided by this disclosure. It should be understood that this is only for illustrative purposes and should not be construed as limiting the solution provided by this disclosure.

[0028] As mentioned earlier, vehicles can capture images of their surroundings using cameras. However, in some situations, the images captured by the cameras may not clearly reflect the true state of the environment. For example, at night, due to insufficient lighting, the images of the vehicle's surroundings may be blurry, making it impossible for the user to judge the specific conditions of the environment. Alternatively, the vehicle's camera may malfunction, preventing the user from obtaining images of the surroundings. This will prevent the user from accurately understanding the true state of the environment, potentially affecting driving safety.

[0029] Therefore, embodiments of this disclosure propose a method for displaying obstacles. In embodiments of this disclosure, environmental images captured by a vehicle's camera and perception information from the vehicle's radar can be acquired. Virtual objects of obstacles around the vehicle can be determined based on the radar perception information, and the vehicle image, the acquired environmental image, and the virtual objects of obstacles can be displayed on a screen.

[0030] This method presents the vehicle's surroundings to the user on the display screen by considering not only environmental images captured by the vehicle's cameras but also radar perception information. Virtual obstacles identified by radar perception information can be displayed on the screen, providing a more intuitive view of the actual surroundings and improving the user experience. Furthermore, even when environmental images cannot accurately reflect the location of obstacles, radar perception information can still identify virtual obstacle objects and display them on the screen, resulting in a more accurate presentation of the surrounding environment and ultimately enhancing the driving experience and safety.

[0031] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, the environment 100 may include a vehicle 101, which may be equipped with cameras and a display screen. The cameras of vehicle 101 can capture images of the environment around the vehicle, and the display screen can display the captured environmental images. The cameras configured on vehicle 101 may include, but are not limited to, a front-view camera, a left-side camera, a right-side camera, and a rear-view camera. The cameras can be mounted around the vehicle body, allowing vehicle 101 to achieve panoramic imaging of the surrounding environment.

[0032] In environment 100, vehicle 101 may be located within a parking area 102 enclosed by dashed lines. The area surrounding vehicle 101 may include obstacles 103 to 109 and vehicle 110. During parking, vehicle 101 can acquire environmental images of its surroundings via a configured camera. These images may include corresponding images of obstacles 103 to 109 and vehicle 110. The environmental images acquired by the camera can be displayed on a display screen configured on the vehicle. For example, it can be as follows... Figure 2A As shown.

[0033] Figure 2A A schematic diagram of scenario 200, illustrating the acquisition of environmental images via cameras according to some embodiments of the present disclosure, is shown. In scenario 200, vehicle 101 may be configured with cameras 201, 202, 203, and 204. Camera 201 may be deployed at the front left of vehicle 101, capturing images of the environment at the front left of vehicle 101, resulting in environmental image 205. Camera 202 may be deployed at the front right of vehicle 101, capturing images of the environment at the front right of vehicle 101, resulting in environmental image 206. Camera 203 may be deployed at the rear left of vehicle 101, capturing images of the environment at the rear left of vehicle 101, resulting in environmental image 207. Camera 204 may be deployed at the rear right of vehicle 101, capturing images of the environment at the rear right of vehicle 101, resulting in environmental image 208.

[0034] In some embodiments, environmental images 205, 206, 207, and 208 can be jointly displayed on a display screen 209 configured on the vehicle 101. In some embodiments, environmental images 205, 206, 207, and 208 can be processed by merging or other methods to form a panoramic image 210 of the vehicle's surrounding environment, which can be displayed on the display screen 209. In some embodiments, the display screen 209 can also display a vehicle image 211 corresponding to the vehicle 101, which can be displayed, for example, at the center of the panoramic image 210.

[0035] It should be understood that the listing of camera positions and numbers in the embodiments of this disclosure is merely illustrative and should not be construed as limiting the embodiments of this disclosure. For example, in some embodiments, the cameras configured on the vehicle 101 may be located at the front, rear, and left and right rearview mirror positions of the vehicle 101, respectively, to capture environmental images of the front, rear, left, and right sides of the vehicle 101. These environmental images may be processed by merging to form a panoramic image of the vehicle's surrounding environment.

[0036] In embodiments of this disclosure, vehicle 101 may be equipped with radar, which may include, but is not limited to, lidar, millimeter-wave radar, ultrasonic radar, etc. The radar can emit detection signals and receive echo signals reflected by obstacles. Based on these echo signals, the radar can perceive obstacles in the environment, such as their position, height, outline, and size. In embodiments of this disclosure, based on the radar's perception information, virtual objects of obstacles around the vehicle can be determined. These virtual objects may be, for example, lines, two-dimensional images, or three-dimensional models. The virtual objects can be displayed on display screen 209 along with environmental images captured by a camera. For example, it can be as follows... Figure 2B As shown in the image.

[0037] Figure 2B A schematic diagram illustrating the content displayed on display screen 209 in some embodiments of this disclosure is shown. (See also:) Figure 2B The display screen 209 can display a panoramic image 210, a vehicle image 211, and virtual objects 221 to 223. Virtual objects 221 to 223 can be determined based on radar perception information of obstacles around the vehicle 101. Specifically, virtual object 221 can correspond to obstacle 103 in the environment 100, virtual object 222 can correspond to obstacle 104 in the environment 100, and virtual object 223 can correspond to obstacle 108 in the environment 100.

[0038] It should be understood that the above content is combined with Figure 1 , Figure 2A and Figure 2B The description of the environments in which the embodiments of this disclosure can be implemented is merely an example of the embodiments of this disclosure and should not be construed as limiting the solutions provided by this disclosure. For example, in some embodiments, vehicle 101 can travel on a road. In some embodiments, environment 100 may include more or fewer obstacles. In some embodiments, vehicle 101 may be configured with more or fewer cameras, and the cameras configured in vehicle 101 may be fisheye cameras, high-definition cameras, or other types of cameras. In some embodiments, the cameras in vehicle 101 may be deployed in other locations within the vehicle.

[0039] In some embodiments, the display screen 209 may be a display screen configured on the in-vehicle terminal of vehicle 101, or it may be a display screen of other types of terminal devices. Terminal devices may include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), consumer electronics, minicomputers, mainframe computers, cloud computing resources, etc. The terminal device may be deployed in vehicle 101 or outside of vehicle 101. The terminal device may acquire environmental images and / or radar perception information from vehicle 101 via wired or wireless means, and display the environmental images and / or virtual objects on the display screen of the terminal device. In some embodiments, the terminal device may process the images and / or perception information before displaying the images and / or virtual objects on the display screen.

[0040] Figure 3 A flowchart of a method 300 for displaying obstacles according to some embodiments of the present disclosure is shown. Method 300 can be executed by a controller, which can be a chip or chip system in a vehicle, such as an Electronic Control Unit (ECU) in vehicle 101. In some embodiments, the controller can also be a vehicle-independent device. For ease of explanation, method 300 will now be described executively using a controller as the execution entity. Figure 3 As shown, method 300 may include blocks 302 to 308.

[0041] In block 302, the controller can acquire environmental images captured by the vehicle's cameras. The vehicle may be equipped with one or more cameras, and the environmental images may be captured by these cameras. For example, the vehicle may be one of the aforementioned... Figure 2A The vehicle 101 in the system may include cameras 201 to 204, and environmental images may include environmental images 205 to 208. In some embodiments, the vehicle may be configured with multiple cameras, and the controller may acquire one or more environmental images captured by one or more of the cameras.

[0042] In block 304, the controller can acquire perception information from the vehicle's radar. This perception information can be information collected by the radar related to obstacles around the vehicle. For example, the radar's perception information can be the echo signal of a radar-emitted detection signal reflected by an obstacle; based on this echo signal, the location, height, outline, and / or size of the obstacle can be determined. In some embodiments, the radar can process the received echo signal to determine the specific location, height, outline, and / or size of obstacles in the environment, and the perception information acquired by the controller can be the location, height, outline, and / or size of the obstacle determined by the radar. In some embodiments, after the radar emits a detection signal and receives the echo signal, it can generate point cloud data, and the perception information acquired by the controller can be this point cloud data. It should be understood that the perception information described above is only an example and does not constitute a limitation of this disclosure.

[0043] In block 306, the controller determines virtual objects of obstacles around the vehicle based on perception information. These virtual objects can be lines, two-dimensional images, or three-dimensional models indicating obstacles. For example, an obstacle can be a pedestrian, and the virtual object can be a model or image of a person. Similarly, an obstacle can be a vehicle, and the virtual object can be a model or image of a vehicle. In some embodiments, the virtual objects can be predefined; for example, the controller's memory can store multiple predefined virtual objects of different types, and the controller can match these virtual objects with the perception information to determine the virtual objects indicating obstacles. In some embodiments, the virtual objects can be generated in real-time by the controller based on perception data; for example, the perception information may include point cloud data, and the controller can generate a three-dimensional model of the obstacle in real-time based on the point cloud data.

[0044] In box 308, the controller controls the display screen to show a vehicle image, an environmental image, and virtual objects of obstacles. The vehicle image can be a predefined image corresponding to a vehicle, used to indicate the vehicle to which the aforementioned radar and cameras are deployed. For example, the vehicle image can be the aforementioned... Figure 2A The vehicle image 211 is displayed on the screen. In some embodiments, the vehicle image, the environment image, and the virtual objects of obstacles can be displayed independently at different positions on the display screen. For example, in the aforementioned scenario 200, the vehicle image 211 can be displayed in the center of the display screen 209, and the environment image 206 can be displayed in the upper right position of the display screen 209. In some embodiments, the vehicle image, the environment image, and the virtual objects of obstacles can be displayed in combination, for example, as described above. Figure 2B As shown in the image. It should be understood that the vehicle image, environment image, and virtual objects displayed on the screen can indicate the position of obstacles in the real environment relative to the vehicle.

[0045] Through the above technical solution, the content displayed on the screen includes not only the environmental images captured by the camera, but also virtual objects of obstacles identified based on radar perception information. In other words, more content can be displayed, thus avoiding the limitations of only displaying environmental images captured by the camera, such as inaccurate content displayed in the event of camera malfunction. This allows for a clearer indication of the state of the environment around the vehicle, enabling users to understand their surroundings more clearly and conveniently through the display screen, thereby improving the user experience.

[0046] Figure 4 A schematic diagram of a scene 400 for displaying obstacles is shown in some embodiments of this disclosure. Scene 400 includes a controller 402, a camera 404, a radar 406, and a display screen 408. The controller 402 can be deployed in a vehicle or can be a vehicle-independent device, and the controller 402 can be implemented in software and / or hardware. In some embodiments, the controller 402 may be, for example, an ECU in vehicle 101.

[0047] Camera 404 and radar 406 can be deployed in the same vehicle, such as vehicle 101 in scenario 100. Camera 404 can acquire images of the environment surrounding the vehicle, while radar 406 can sense the environment by emitting detection signals and receiving echo signals reflected from obstacles, generating perception information. In some embodiments, the environmental images captured by camera 404 may include images of obstacles.

[0048] Controller 402 can execute the aforementioned method 300. For example, controller 402 can acquire environmental images captured by camera 404 and perception information from radar 406 using the methods in blocks 302 and 304. Controller 402 can determine virtual objects of obstacles based on the perception information using the method in block 306. Controller 402 can control display screen 408 to display the environmental images and the virtual objects using the method in block 308.

[0049] In some embodiments, the perception information acquired by the controller 402 from the radar 406 may include information such as the outline, size, height, and distance of the obstacle. In some embodiments, the radar 406 may classify obstacles, and the perception information acquired by the controller 402 may include classified obstacle indication information. In some embodiments, the radar 406 may classify obstacles into point obstacles and line obstacles based on their width. For example, in the perception information of the radar 406, point obstacles can be represented by points, and line obstacles by lines. In some embodiments, the radar 406 may classify obstacles based on their height. For example, the height of the obstacle can be determined, and the height can be compared with a predefined height threshold to classify the obstacle into two categories: "high" and "low". As an example, Figure 5 A schematic diagram of perception information 500 in some embodiments of this disclosure is shown. In perception information 500, dots can represent point obstacles, lines can represent the outline of line obstacles, solid lines can represent obstacles with a height of "high," and dashed lines can represent obstacles with a height of "low." For greater clarity, in... Figure 5 The image also shows a vehicle image 510 corresponding to the vehicle.

[0050] It should be understood that the above combination Figure 5 The description of the perception information is merely an example of some embodiments of this disclosure and should not be construed as limiting the disclosure. For example, in some embodiments, the perception information may include more or fewer categories. In some embodiments, the radar 406 may not classify obstacles, and the perception information acquired by the controller 402 may be the raw data collected by the radar 406. In some embodiments, the controller 402 may determine the type of obstacle and classify the obstacle based on the raw data collected by the radar 406.

[0051] The controller 402 can determine virtual objects of obstacles based on the perception information from the radar 406. The virtual objects of obstacles can be determined by the controller 402 from a predefined set of multiple virtual objects. For example, the controller 402's local memory can store a predefined set of virtual objects, which includes multiple predefined virtual objects that may have different types, outlines, and sizes. The controller 402 can determine the type, outline, and size of the obstacle based on the perception information from the radar 406, and match the type, outline, and size of the obstacle with the type, outline, and size of the virtual objects in the virtual object set, identifying virtual objects with the same type, outline, and size as the virtual objects of the obstacle.

[0052] In some embodiments, the virtual object of the obstacle may be generated by the controller 402 based on the perception information of the radar 406. In some embodiments, if the controller 402 determines that there is no virtual object in the predefined set of virtual objects that matches the outline and size of the obstacle, it may generate a new virtual object corresponding to the obstacle based on the perception information of the radar 406.

[0053] In some embodiments, the sensing information of radar 406 may include information indicating the height of an obstacle, and controller 402 may also control display screen 408 to display information associated with the height of the obstacle. For example, in some embodiments, controller 402 may control display screen 408 to display the height of the obstacle by displaying a numerical value while simultaneously controlling display screen 408 to display an environmental image, a vehicle image, and a virtual object of the obstacle. This numerical value may be displayed to the side of the virtual object of the obstacle or at the top of display screen 408, and this disclosure is not limited thereto. In some embodiments, controller 402 may adjust the size of the virtual object of the obstacle according to the environmental image displayed on display screen 408, such that the size ratio of the virtual object on display screen 408 relative to the environmental image is equal to the height ratio of the obstacle relative to its surrounding environment in the real environment, thereby visually representing the height of the obstacle through display screen 408.

[0054] In some embodiments, the sensing information of radar 406 may include information indicating the distance between an obstacle and a vehicle, and controller 402 may also control display screen 408 to display information associated with the distance to the obstacle. Exemplarily, in some embodiments, the distance between an obstacle and a vehicle can be represented by controlling the relative distance between the displayed vehicle image and a virtual object. Controller 402 may determine the distance between an obstacle and a vehicle in the real environment (referred to as a first distance for ease of distinction and explanation) based on the sensing information of radar 406. During the process of controller 402 controlling display screen 408 to display a vehicle image, an environmental image, and a virtual object of an obstacle, controller 402 may control the distance between the virtual object of the obstacle displayed on display screen 408 and the vehicle image based on the first distance (referred to as a second distance for ease of distinction and explanation). For example, the second distance may be proportional to the first distance. In some embodiments, controller 402 may control display screen 408 to display the distance between the obstacle and the vehicle numerically. In some embodiments, controller 402 may, if it determines that the distance between the obstacle and the vehicle is less than a predetermined distance threshold, then control display screen 408 to display the distance between the obstacle and the vehicle and / or the height of the obstacle numerically.

[0055] In some embodiments, the controller 402 can determine the position of an obstacle relative to the vehicle based on the perception information from the radar 406, and control the display screen 408 to display a virtual object of the obstacle at the corresponding position in the vehicle image. In some embodiments, the controller 402 can control the display screen 408 to simultaneously display the distance between the obstacle and the vehicle, the height of the obstacle, and the direction of the obstacle relative to the vehicle. In this way, the virtual object displayed on the screen can clearly reflect the position, height, distance, and other information of the obstacle in the real environment, thereby enabling the user to clearly determine the true state of the environment and improving the driving experience.

[0056] It should be understood that Figure 4 The scenario 400 shown for displaying obstacles is merely an example of an embodiment of this disclosure and should not be construed as a limitation on the embodiments of this disclosure. For example, in some embodiments, the controller 402 may also acquire environmental images from multiple cameras. In some embodiments, the controller 402 may acquire perception information from multiple radars. In some embodiments, the controller 402 may control images displayed on multiple displays. In some embodiments, the functionality of the controller 402 may be implemented by one or more modules. In some embodiments, the controller 402, camera 404, radar 406, and / or display 408 may be deployed in the same device or in different devices. In some embodiments, the camera 404 and radar 406 may be deployed in the same vehicle or each may be deployed in a roadside device or other device, and the controller 402 may acquire environmental images collected by the camera 404 and perception information from the radar 406 via wired or wireless means.

[0057] In some embodiments, the controller 402 can control the display screen 408 to display vehicle images and environmental images, and, under certain conditions, further control the display screen 408 to display virtual objects. For example, the controller 402 can further control the display screen 408 to display virtual objects of obstacles upon receiving user input. In some embodiments, the control device 402 can receive user instructions to display virtual objects on the display screen 408 via the display screen 408 or buttons.

[0058] In some embodiments, the controller 402 can determine whether the camera 404 can accurately capture images of obstacles around the vehicle. If the camera 404 cannot accurately capture images of obstacles around the vehicle, the controller then controls the display screen 408 to display virtual objects. For example, in some embodiments, the controller 402 can determine the perception performance level of the camera 404 based on the resolution of the environmental images captured by the camera 404, and compare this perception performance level with a predetermined perception performance level. If the perception performance level of the camera 404 is lower than the predetermined perception performance level, it is determined that the camera 404 cannot accurately capture images of obstacles around the vehicle. In this case, the radar 406 is used to detect obstacles in the environment, and the controller then controls the display screen 408 to display virtual objects of the obstacles.

[0059] For example, Figure 6 A schematic diagram illustrating the content displayed on a display screen 408 of some embodiments of this disclosure is shown. For example... Figure 6 As shown, the display screen 408 can display an environmental image 610 and a vehicle image 620. The vehicle image 620 may, for example, correspond to the aforementioned... Figure 1 and Figure 2A Vehicle 101 in the image. Environmental image 610 can be, for example, the aforementioned... Figure 2A The image displayed on the screen 408 is a combination of the environmental image 206 captured by camera 202 and the environmental image 208 captured by camera 204. Due to malfunctions or other reasons, the cameras 201 and 203 installed on the vehicle 101 cannot clearly capture the environment, and therefore cannot obtain environmental images 205 and 207. Consequently, the environmental image 610 displayed on the screen 408 does not include complete images corresponding to obstacles 103, 104, 107, and 108. The controller 402 can identify that cameras 201 and 203 have malfunctioned, and based on the perception information of radar 406, it determines virtual objects 631, 632, 633 and 634 corresponding to obstacles 103, 104, 107 and 108 respectively, and controls the display screen 408 to display virtual objects 631, 632, 633 and 634 together with the environmental image 610 and the vehicle image 620.

[0060] In some embodiments, the controller 402 can determine whether the vehicle is in a parking scenario based on user instructions or by recognizing environmental images captured by the camera 404. If the controller 402 determines that the vehicle is in a parking scenario, it can then control the display screen 408 to display virtual images of obstacles. That is, the controller 402 can only execute the aforementioned method 300 if the vehicle is in a parking scenario. In some embodiments, if the controller determines that the vehicle is in a parking scenario, it can acquire multiple environmental images captured by multiple cameras to generate a panoramic image of the vehicle's surroundings, and the environmental image displayed on the display screen 408 can be this panoramic image.

[0061] In some embodiments, the display screen 408 may not display virtual objects of obstacles that are more than a predetermined distance threshold from the vehicle. For example, in scenario 100, the area around the vehicle 101 may include more obstacles that are farther away from the vehicle 101, for example, exceeding a predetermined distance threshold (e.g., 2 meters, 3 meters, etc.). The controller 402 can control the display screen 408 to not display virtual objects of these obstacles. That is, the display screen 408 can be controlled to display virtual objects of obstacles only when the obstacles are close to the vehicle.

[0062] In some embodiments, when the distance between the obstacle and the vehicle is determined to be less than a predetermined distance threshold, the controller 402 can control the display screen 408 to display a new interface, such as displaying a new virtual object of the obstacle. For example, it can be as follows: Figure 7 As shown in the image. Figure 7 A schematic diagram illustrating the content displayed on a display screen 408 according to some embodiments of the present disclosure is shown. For example... Figure 7 As shown, the display screen 408 can display content including a specific angle of the virtual object 702. The virtual object 702 can be, for example, the obstacle 104 in the aforementioned scene 100, and can be related to the aforementioned... Figure 2B The virtual object 222 displayed on the central display screen 209 corresponds to this. The controller 402 can display, for example, on the control display screen 408. Figure 2B During the process shown, the distance between the obstacle 104 and the vehicle 101 is determined. If the distance between the obstacle 104 and the vehicle 101 is less than a predetermined distance threshold, the controller 402 can control the display screen 408 as follows: Figure 7 The image shows content including virtual object 702 at a specific angle.

[0063] In some embodiments, Figure 7In the displayed interface, the controller 402 can also control the display screen 408 to display the distance D between the obstacle 104 and the vehicle in the real environment, as well as the height H of the obstacle 104. In some embodiments, while controlling the display screen to display the virtual object 702, the controller 402 can also control the display screen 408 to display a local environmental image including an image of the obstacle 104 captured by a single camera. In this way, obstacles close to the vehicle can be displayed more clearly, thereby improving the user experience.

[0064] Figure 8 A flowchart of a method 800 for displaying obstacles according to some embodiments of the present disclosure is shown. Figure 8 The method shown can be executed by a controller, such as the aforementioned controller 402, or an ECU configured in the vehicle. For ease of explanation, the method 800 will be illustrated below using a controller as the executing entity. Figure 8 As shown, method 800 may include blocks 802 to 820.

[0065] In block 802, the controller receives a user's request to perceive the surrounding environment. In block 804, the controller determines whether virtual objects need to be displayed on the screen, for example, whether it has received user input indicating the display of virtual objects. If yes, proceed to block 806; otherwise, proceed to block 818. In block 806, the controller receives environmental images captured by a camera and perception information from radar. In block 808, the controller determines the type and size of obstacles based on radar perception information; in block 810, the controller determines the location of obstacles based on radar perception information. In block 812, the controller identifies virtual objects associated with obstacles. In block 814, the controller determines the position of the virtual object of the obstacle within the environmental image captured by the camera, based on the obstacle's location. In block 816, the controller controls the display screen to display the environmental image, vehicle image, and virtual object of the obstacle. In block 818, the controller receives environmental images captured by a camera. In block 820, the controller controls the display screen to display the vehicle image and environmental image.

[0066] It should be understood that the steps of the method for displaying obstacles in the embodiments of this disclosure are merely illustrative and should not be construed as limiting the embodiments of this disclosure. For example, method 800 may also include more or fewer steps. The method of this disclosure enables the display screen to show environment-related information both as images and as virtual objects, thereby providing a better and clearer presentation of overall environmental information on the display screen and improving the user experience. Even if the camera cannot accurately capture images of the environment around the vehicle, obstacles in the environment around the vehicle can still be accurately presented to the user as virtual objects, enhancing the user's driving experience.

[0067] Figure 9 A block diagram of a device 900 for displaying obstacles according to some embodiments of the present disclosure is shown. Device 900 may, for example, correspond to the controller in the foregoing method embodiments. Figure 9 As shown, the device 900 includes a first acquisition unit 902 configured to acquire environmental images captured by the vehicle's camera. The device 900 also includes a second acquisition unit 904 configured to acquire perception information from the vehicle's radar. The device 900 further includes an image determination unit 906 configured to determine virtual objects of obstacles around the vehicle based on the perception information. Furthermore, the device 900 includes a display unit 908 configured to display vehicle images, environmental images, and virtual objects of obstacles.

[0068] In some embodiments, the image determination unit 906 further includes: a first determination unit configured to determine the outline and size of an obstacle based on perceptual information; and a second determination unit configured to determine virtual objects of the obstacle from a predefined set of virtual objects based on the outline and size.

[0069] In some embodiments, the device 900 further includes a third determining unit configured to determine, based on perception information, the orientation of the obstacle relative to the vehicle and a first distance between the obstacle and the vehicle. The display unit 908 further includes a first display unit configured to display a vehicle image and an environment image, and to display a virtual object of the obstacle at the direction of the vehicle image and at a second distance from the vehicle image, wherein the second distance is associated with the first distance.

[0070] In some embodiments, the third determining unit further includes a fourth determining unit configured to determine the height of the obstacle based on perceived information. The device 900 also includes a second display unit configured to display a first distance and the height of the obstacle.

[0071] In some embodiments, the device 900 further includes a distance determination unit configured to determine that the first distance is less than a predetermined distance threshold before the second display unit displays the first distance and the height of the obstacle.

[0072] In some embodiments, the device 900 further includes: a fifth determining unit configured to determine a second virtual object of an obstacle based on the height of the obstacle, wherein the second virtual object has height information associated with the height of the obstacle; and a third display unit configured to display target content on a display screen, wherein the target content includes the second virtual object and a first distance.

[0073] In some embodiments, the device 900 further includes: a fourth display unit configured to display an environmental image before the second acquisition unit 904 acquires the radar's perception information; and an instruction information receiving unit configured to receive instruction information input by a user before the second acquisition unit 904 acquires the radar's perception information, wherein the instruction information is used to instruct the display of a virtual object on the display screen.

[0074] In some embodiments, the apparatus 900 further includes: a sixth determining unit configured to determine the perception performance level of the camera based on an environmental image before the second acquiring unit 904 acquires the perception information of the radar; and a seventh determining unit configured to determine that the perception performance level of the camera is lower than a predetermined perception performance level before the second acquiring unit 904 acquires the perception information of the radar.

[0075] In some embodiments, the device 900 further includes an eighth determining unit configured to determine that the vehicle is in a parking scenario. The first acquiring unit 902 further includes a third acquiring unit configured to acquire panoramic images of the vehicle's surroundings captured by a camera. The display unit 908 further includes a fifth display unit configured to display the vehicle image, the panoramic image, and virtual objects of obstacles.

[0076] Figure 10 A schematic block diagram of a controller 1000 suitable for implementing embodiments of this application is illustrated. The controller 1000 may, for example, correspond to the controller described in the foregoing method embodiments. Figure 10 As shown, the controller 1000 includes a processor 1001, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 1003 according to computer program instructions stored in read-only memory (ROM) 1002. The RAM 1003 may also store various programs and data required for the operation of the controller 1000. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0077] The various methods or processes described above, such as methods 300 and 800, can be executed by processor 1001. For example, in some embodiments, method 300 may be implemented as a computer software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on controller 1000 via ROM 1002. When the computer program is loaded and executed by processor 1001, one or more steps or actions in the methods or processes described above may be performed.

[0078] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0079] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method (300) for displaying obstacles, comprising: Acquire environmental images captured by the camera of vehicle (302); Acquire the radar perception information of the vehicle described in (304); Based on the perceived information, determine (306) the virtual objects of obstacles around the vehicle; as well as The virtual objects of the vehicle, the environment, and the obstacles are displayed on the screen (308).

2. The method (300) according to claim 1, wherein determining (306) virtual objects of obstacles around the vehicle based on the perceived information comprises: Based on the perceived information, the outline and size of the obstacle are determined; as well as Based on the outline and size, the virtual object of the obstacle is determined from a predefined set of virtual objects.

3. The method (300) according to claim 1, further comprising: Based on the perceived information, the direction of the obstacle relative to the vehicle and the first distance between the obstacle and the vehicle are determined; And the virtual objects that display (308) the vehicle image, the environment image, and the obstacles on the display screen include: The vehicle image and the environment image are displayed on the screen. A virtual object of the obstacle is displayed in the direction of the vehicle image and at a second distance from the vehicle image, wherein the second distance is associated with the first distance.

4. The method (300) according to claim 3, further comprising: Based on the perceived information, the height of the obstacle is determined; as well as The first distance and the height of the obstacle are displayed on the screen.

5. The method (300) of claim 4, further comprising, before displaying the first distance and the height of the obstacle on the display screen: It is determined that the first distance is less than a predetermined distance threshold.

6. The method (300) according to claim 5, wherein displaying the first distance and the height of the obstacle on the display screen comprises: Based on the height of the obstacle, a second virtual object of the obstacle is determined, the second virtual object having height information associated with the height of the obstacle; as well as The target content is displayed on the display screen, and the target content includes the second virtual object and the first distance.

7. The method (300) according to claim 1, further comprising, before acquiring (304) the radar perception information of the vehicle: The environmental image is displayed on the display screen; as well as The system receives user input indicating that a virtual object should be displayed on the screen.

8. The method (300) according to claim 1, further comprising, before acquiring the radar perception information of the vehicle (304): Based on the environmental image, determine the perception performance level of the camera; as well as It is determined that the perception performance level is lower than the predetermined perception performance level.

9. The method (300) according to any one of claims 1 to 8, further comprising, before displaying (308) the vehicle image of the vehicle, the environment image, and the virtual object of the obstacle on the display screen: The vehicle is determined to be in a parking scenario; Furthermore, the environmental images captured by the camera of vehicle (302) include: Acquire panoramic images of the area surrounding the vehicle captured by the camera; The virtual objects that display (308) the vehicle image, the environment image, and the obstacles on the display screen include: The vehicle image, the panoramic image, and the virtual object are displayed on the screen.

10. A device (900) for displaying obstacles, comprising: The first acquisition unit (902) is configured to acquire environmental images captured by the vehicle's camera; The second acquisition unit (904) is configured to acquire the radar perception information of the vehicle; The image determination unit (906) is configured to determine virtual objects of obstacles around the vehicle based on the perceived information; as well as The display unit (908) is configured to display the vehicle image, the environment image, and the virtual object of the obstacle.

11. A controller (1000), comprising: At least one processor (1001); as well as A memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the controller (1000) to perform the method according to any one of claims 1 to 9.

12. A vehicle comprising a controller (1000) according to claim 11.

13. A computer program product comprising computer-executable instructions that, when executed, cause a computer to perform the steps of the method according to any one of claims 1 to 9.

Images