Display method, device and system

Abstract

A display method, device, and system, applied to fields such as autonomous driving or intelligent driving, can reduce the impact of A-pillar blind spots. The method includes: obtaining first information from a first camera device; determining driver information based on the first information; obtaining second information from at least one sensor device, the second information including external environment information; and obtaining third information based on the second information, the third information including information obtained by processing the second information based on the driver information. This solution can further be used to enhance the capabilities of autonomous driving or advanced driver assistance systems (ADAS), and can be applied to connected vehicles, such as vehicle-to-everything (V2X), long-term evolution (LTE-V), and vehicle-to-vehicle (V2V).

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technologies, and in particular to a display method, device, and system. Background Art

[0002] As the support pillar between the windshield and the front door, the A-pillar plays an irreplaceable role in ensuring safe driving. However, the stronger the A-pillar, the larger its area, resulting in a larger visual blind spot during driving. How to reduce the impact of the A-pillar blind spot is a technical problem to be solved in the embodiments of this application. Summary of the Invention

[0003] Embodiments of the present application provide a display method, device, and system to reduce the impact of the A-pillar blind spot.

[0004] In a first aspect, a display method is provided. The method is performed by a control device, which may optionally be a domain controller, a chip or integrated circuit within the domain controller, and includes: the control device acquiring first information from a first camera device; determining driver information based on the first information, the driver information including at least one of the driver's viewing angle and the relative position of the driver and the A-pillar; acquiring second information from at least one sensor device, the second information including information about the external environment; and acquiring third information based on the second information, the third information including information obtained by processing the second information based on the driver information. Optionally, the third information is output for display.

[0005] Through the above method, information about the external environment of the A-pillar can be obtained through the sensing device, and the external environment information of the A-pillar is processed according to information such as the driver's perspective and the relative position of the A-pillar, and finally information about the external environment blocked by the A-pillar is obtained, thereby reducing the impact of the A-pillar blind spot and improving driving safety.

[0006] In a possible design, the obtaining of the third information based on the second information includes: obtaining the third information through at least one of cropping and fusion based on the driver information and the location information of the obstacle area in the second information.

[0007] For example, the clipping process may include clipping the second information based on the driver's perspective. Specifically, the second information is clipped based on a clipping distance dx in the x-direction and a clipping distance dy in the y-direction, where the driver's perspective includes an x-direction mapping angle and a y-direction mapping angle, dx is determined based on the x-direction mapping angle, and dy is determined based on the y-direction mapping angle.

[0008] Since the driver's viewing angle is different, the area blocked by the A-pillar is also different. Through the above, according to the driver's viewing angle, the second information including the external environment blocked by the A-pillar is cut out, and the display of the A-pillar blocked area can be more accurate.

[0009] In a possible design, the fusion process may include: fusing the cropped second information with the location information of the obstacle area in the second information.

[0010] As mentioned above, if the area blocked by the A-pillar is an obstacle-free road area, the harm caused by the A-pillar obstruction is actually not very significant. The most worrying point is that if the A-pillar blocks an obstacle in the external environment, it may cause the driver's vehicle to collide with the obstacle, causing a traffic accident. In this embodiment of the application, to avoid the above-mentioned clipping process, the obstacles in the external environment are clipped out, and the clipped information is fused again with the location information including the obstacle. This ensures that the fused information always includes information about the obstacle, thereby ensuring driving safety.

[0011] In one possible design, when the relative position of the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and enlarging; or, when the relative position of the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and shrinking it.

[0012] As described above, when the driver's eyes are close to the A-pillar, the fused information is used as a basis for appropriate cropping, and the cropped area is enlarged for display. When the driver's eyes are farther away from the A-pillar, the fused information is used as a basis for adding the original information, and the overall information is scaled down for display, consistent with the human eye's characteristic of larger near-field vision and smaller far-field vision.

[0013] In one possible design, the at least one sensing device includes a second camera device and at least one detection device, and obtaining the second information from the at least one sensing device includes: obtaining fused information from the at least one sensing device, wherein the fused information incorporates the speed and / or distance information of the obstacle.

[0014] By adding information such as the speed and / or distance of the obstacle to the fused information, the driver can be reminded of the relative information between the current obstacle and the driver's vehicle, so that the driver has more time to avoid the obstacle and improve driving safety.

[0015] In one possible design, the method further includes: acquiring brightness information of an external environment from an ambient light sensor;

[0016] The brightness and / or contrast of the third information is adjusted.

[0017] Through the above, the brightness and / or contrast of the displayed third information is adjusted according to the brightness information of the external environment, thereby improving the driver's visual experience, and the brightness and / or contrast of the displayed third information can be adjusted at any time according to the external environment to ensure that the driver can clearly see the external area blocked by the A-pillar in any scenario, thereby ensuring driving safety.

[0018] In a second aspect, a device is provided, and the beneficial effects can be found in the description of the first aspect. The device has the function of implementing the behavior in the method embodiment of the first aspect. The function can be implemented by executing corresponding hardware or software. The hardware or software may include one or more units corresponding to the above functions. In one possible design, the device includes: a communication unit for obtaining first information from a first camera device; a processing unit for determining driver information based on the first information, the driver information including at least one of the driver's viewing angle and the relative position information between the driver and the A-pillar; the communication unit is also used to obtain second information from at least one sensor device, the second information including external environment information; the processing unit is also used to obtain third information based on the second information, the third information including information obtained by processing the second information based on the driver information. These units can perform the corresponding functions in the method example of the first aspect above. For details, please refer to the detailed description in the method example, which will not be repeated here.

[0019] In a third aspect, a device is provided, which may be the control device in the embodiment of the first aspect. The device includes a communication interface and a processor, and optionally, a memory. The memory is configured to store computer programs or instructions, and the processor is coupled to the memory and the communication interface. When the processor executes the computer program or instructions, the device executes the method performed by the control device in the embodiment of the method of the first aspect.

[0020] In a fourth aspect, a computer program product is provided, comprising: a computer program code, wherein when the computer program code is executed, the method performed by the control device in the first aspect is executed.

[0021] In a fifth aspect, a chip system is provided, comprising a processor configured to implement the functions of the control device in the method of the first aspect. In one possible design, the chip system further comprises a memory configured to store program instructions and / or data. The chip system may be composed solely of a chip or may include a chip and other discrete components.

[0022] In a sixth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed, the above-mentioned first aspect is implemented by the control device.

[0023] In the seventh aspect, a display system is provided, comprising: a first camera device for acquiring first information about the driver; at least one sensor device for acquiring second information, wherein the second information includes information about the external environment; a control device for determining driver information based on the first information, wherein the driver information includes at least one of the driver's viewing angle and relative position information between the driver and the A-pillar, and acquiring third information based on the second information, wherein the third information includes information obtained by processing the second information based on the driver information; and a display device for displaying the third information.

[0024] Through the above method, the sensor device can collect information about the external environment, while the first camera device can collect information about the driver. The external environment information is then combined with the driver information to generate third information, which can be considered to be information about the external environment in the area obstructed by the A-pillar. Finally, the display device displays the third information, thereby preventing traffic accidents caused by A-pillar obstruction.

[0025] In a possible design, the first camera device is a high-definition camera, and there are three high-definition cameras, which are located above the driver's seat, above the central control screen, and above the left door respectively.

[0026] In one possible design, the at least one sensing device includes a second camera device and at least one detection device, the second camera device is an infrared camera, there are two infrared cameras, which are located directly below the left A-pillar and directly below the right A-pillar respectively, and the detection device is a radar, there are two radars, which are located next to the left headlight and the right headlight of the vehicle body respectively.

[0027] In a possible design, the display device is a flexible display screen, and there are two flexible display screens, which are respectively attached to the left A-pillar and the right A-pillar.

[0028] Through the above, the flexible display screen can perfectly fit the A-pillar without affecting the structural strength of the A-pillar, and can display blind spots, protecting the driver's safety at low cost.

[0029] In one possible design, obtaining the third information based on the second information includes: obtaining the third information through at least one of cropping and fusion based on the driver information and the position of the obstacle in the second information.

[0030] In one possible design, the cropping process includes: cropping the second information according to the driver's perspective.

[0031] In one possible design, the second information is cropped according to the driver's perspective, including: cropping the second information according to the cropping distance dx in the x-direction and the cropping distance dy in the y-direction, the driver's perspective includes an x-direction mapping angle and a y-direction mapping angle, the dx is determined according to the x-direction mapping angle, and the dy is determined according to the y-direction mapping angle.

[0032] In a possible design, the fusion process includes: fusing the cropped second information with the location information of the obstacle area in the second information.

[0033] In one possible design, when the relative position of the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and enlarging; or, when the relative position of the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and shrinking it.

[0034] In a possible design, the acquiring of the second information of the external environment includes: acquiring fused information from the at least one sensing device, wherein the fused information is integrated with the speed and / or distance information of the obstacle.

[0035] In a possible design, it further includes: an ambient light sensor for collecting external ambient brightness; and the control device is further used to adjust the brightness and / or contrast of the third information.

[0036] In one possible design, the ambient light sensors are located at the bottom of the left A-pillar and the bottom of the right A-pillar respectively.

[0037] In a possible design, the above-mentioned display system may also include a sound alarm device for playing information about obstacles externally.

[0038] Through this, the obstacle information displayed can include obstacle type, relative distance from the obstacle to the current vehicle, or speed, thereby reminding the driver of the presence of obstacles on the current road and ensuring driving safety. Multi-dimensional prompts, combining audio and visual, make the reminder more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the A-pillar provided for the implementation of this application;

[0040] Figure 2a and Figure 2b A schematic diagram of the A-pillar blind spot provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a display method provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the fusion provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of cutting provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of the cutting process provided in an embodiment of the present application;

[0046] Figure 8 、 Figure 9 and Figure 10 A schematic diagram of the fusion provided in the embodiment of the present application;

[0047] Figure 11 A schematic diagram of cutting provided in an embodiment of the present application;

[0048] Figure 12 A schematic diagram of the final pre-cut area provided in an embodiment of the present application;

[0049] Figure 13a and Figure 13b A schematic diagram of the layout of each device provided in the embodiments of the present application;

[0050] Figure 14 A schematic diagram of the processing process of the display system provided in an embodiment of the present application;

[0051] Figure 15 Schematic diagram of the cockpit front view, cockpit left view and cockpit top view provided in an embodiment of the present application;

[0052] Figure 16 A schematic diagram of a processing device provided in an embodiment of the present application;

[0053] Figure 17 Another schematic diagram of the processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0055] In addition, the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] like Figure 1 As shown, the A-pillar is a connecting pillar on the left and right front that connects the roof and front cabin. It is located between the engine compartment and the cockpit, above the left and right rearview mirrors. For modern closed-door vehicles, the A-pillar is an indispensable structural element. Since most modern vehicles utilize a monocoque structure, the entire body serves as the frame and bears the entire load. The presence of the A-pillar ensures greater stability and rigidity for this monocoque structure, protecting the cockpit and minimizing deformation in the event of a collision, playing a crucial role in driver and passenger safety.

[0057] However, the presence of the A-pillar will block part of the driver's sight. When the distance is far, the blocked area is large enough to block an adult; especially when turning left, it will bring hidden dangers to driving safety. Figure 2a or Figure 2bAs shown, the pillar between the front windshield and the left door is the left A-pillar. Because it is closer to the driver, the blind spot angle is larger, approximately 6°. The pillar between the front windshield and the right door is the right A-pillar. Because it is farther from the driver, the blind spot angle is smaller, approximately 2°. How to reduce the impact of the A-pillar blind spot is a technical problem to be solved in the embodiments of this application.

[0058] One possible solution uses two cameras (one inside the vehicle, one outside), a display, and a central processing unit to comprehensively display the A-pillar blind spot. For example, the first camera (inside the vehicle) identifies the driver's line of sight; the second camera (outside the vehicle) is driven by a servo motor to align with the line of sight; and the display is controlled to display the blind spot image (aligned with the inner surface of the A-pillar). This solution has the following drawbacks: A single external camera can have inaccurate recognition; a single internal camera acquires viewpoint information slowly and with large errors; and controlling the external camera's angle (mechanical rotation) through the viewpoint results in significant delay and insufficient performance.

[0059] In another possible solution, the A-pillar blind spot information is detected by cameras and radars, and after processing, it is displayed on a display and announced by voice, allowing the driver to understand the road conditions in the A-pillar blind spot in real time while driving. For example, the first display is set on the left A-pillar, and the second display is set on the right A-pillar. The camera (outside the vehicle) detects the blind spot information; the radar (outside the vehicle) detects the blind spot information; the microprocessor processes the blind spot information and outputs it for display, and voice warnings are given. This solution has the following disadvantages: ordinary displays installed alone pose a risk of secondary injury; the information obtained by the camera and radar is not clearly processed for the driver, including fusion or cropping, and redundant information is not filtered out, affecting driving.

[0060] Another possible solution uses cameras placed inside and outside the vehicle to dynamically display the area obscured by the A-pillar from the driver's perspective. The exterior camera captures surround view information of the road ahead, while the interior camera captures the driver's line of sight. Based on the driver's line of sight, the image captured by the exterior camera is cropped to capture the driver's attention, and the image is displayed on the flexible display screen inside the A-pillar, aligned with its curvature. The image displayed on the flexible display screen inside the vehicle is then altered in real time based on the driver's line of sight angle captured by the interior camera. This solution has the following shortcomings: it crops the video based solely on the driver's perspective, failing to account for the varying display effects caused by the driver's distance from the A-pillar (same viewing angle, close up or further away); it fails to account for situations where videos cropped based on excessively large or small line of sight angles may not fully encompass obstacles; and the display brightness fails to take into account the effects of ambient light.

[0061] like Figure 3As shown, a schematic diagram of a possible application scenario is provided. The above-mentioned application scenarios may be scenarios such as unmanned driving, automatic driving, intelligent driving, and networked driving. The processing device or system in the embodiment of the present application may be installed in a motor vehicle (such as an unmanned vehicle, an intelligent vehicle, an electric vehicle, a digital vehicle, etc.), a drone, a rail vehicle, a bicycle, or a speed measuring device, etc. In addition, in addition to installing the processing device or system, the above-mentioned device may also be installed with a communication device, etc., without limitation. In addition, the above-mentioned processing device, display system, or communication device may be installed on a mobile device, for example, on a vehicle, etc., or may be installed on a fixed device, such as a road side unit, etc. In the embodiment of the present application, the installation location and function of the processing device, display system, or communication device are not limited.

[0062] like Figure 4 As shown, a display method is provided. The method can be performed by a control device. The control device can be a controller (e.g., a domain controller), or the control device can be a chip installed in a communication device, such as a controller, or other devices. The method includes but is not limited to the following steps:

[0063] In step 401, a control device receives first information from a first camera device. The first camera device can be used to capture images or video information of a driver. The first information can be image information or video information of the driver captured by the first camera device, without limitation.

[0064] In step 402 , the control device determines driver information based on the first information. The driver information may include at least one of the driver's viewing angle and relative position information between the driver and the A-pillar.

[0065] Step 403: The control device obtains second information from at least one sensing device, where the second information includes external environment information.

[0066] Optionally, the at least one sensing device includes at least one of a second camera and a detection device. The second camera can be used to obtain image or video information of the external environment, and the detection device can be used to detect information such as the speed and / or distance of obstacles in the external environment. In a possible implementation, the control device can obtain fusion information from the at least one sensing device, and the fusion information incorporates information such as the speed and / or distance of the obstacle. Figure 5As shown, the outer dashed box represents the raw information of the external environment captured by the second camera. The box outside the obstacle is the obstacle identification box, with marker information attached above it. This annotation information may include the obstacle type, the relative distance and / or relative speed between the obstacle and the vehicle, etc. The relative distance and relative speed can be detected by the detection device, and the obstacle type can be detected and identified by the second camera.

[0067] In step 404, the control device obtains third information based on the second information. The third information includes information obtained by processing the second information based on the driver information. For example, the control device may perform at least one of cropping and fusion based on the driver information and the location of the obstacle in the second information to obtain the third information. The term "cropping" here simply refers to a type of information processing and does not limit the specific cropping action. It simply refers to obtaining a smaller piece of information from a larger piece of information.

[0068] In a possible implementation, if the second information includes the driver's perspective, the control device may crop the second information according to the driver's perspective. Figure 6 As shown, the second information can be clipped according to the clipping distance dx in the x direction and the clipping distance dy in the y direction, wherein dx is determined according to the x-direction mapping in the driver's perspective, and dy is determined according to the y-direction mapping angle in the driver's perspective.

[0069] like Figure 7 As shown in the figure, due to the difference in focal length and image distance, the image size of the object on the film is different. Under the same viewing angle θ, the resulting video cropping portion d is different. The above process of determining the cropping distance dx in the x-direction based on the x-direction mapping angle can meet the following conditions:

[0070]

[0071] Among them, dx represents the clipping distance in the x direction, v represents the image distance, f represents the focal length, θ2 represents the mapping angle in the x direction, and D x Represents the width of the second video in the x direction.

[0072] The above process of determining the clipping distance dy in the y direction according to the y-direction mapping angle may satisfy the following conditions:

[0073]

[0074] Among them, dy represents the clipping distance in the y direction, v represents the image distance, f represents the focal length, θ3 represents the y-direction mapping angle, and D y Represents the width of the second video in the y direction.

[0075] Optionally, the control device may fuse the cropped second information with the location information of the obstacle area in the second information. The control device may obtain the location coordinates of the obstacle in the second information, and obtain the coordinates of the largest area around the obstacle based on the location coordinates of the obstacle. Figure 8 As shown, the control device can obtain the coordinates of the lower left corner (x1, y1) and the upper right corner (x2, y2) of the obstacle annotation box. The lower left corner (x1, y1) of the obstacle annotation box is extended downward and leftward (at a 45-degree angle) to obtain the intersection with the second information area and its coordinates (x0, y0). The upper right corner (x2, y2) of the obstacle annotation box is extended upward and rightward (at a 45-degree angle) to obtain the intersection with the second information area and its coordinates (x, y). The rectangular area formed by (x0, y0) and (x, y) is called the "maximum area around the obstacle." Simultaneously, the lower left corner coordinates of the clipped area obtained based on the driver's perspective are set to (x0', y0'), and the upper right corner coordinates are set to (x', y').

[0076] Case 1: If Figure 9 As shown, when the obstacle is within the clipping area, that is: x0' <x0<x1,x2<x<x’,y1> If y0'>y0,y>y'>y2, the final fusion area is the intersection of the clipping area and the largest area around the obstacle.

[0077] Case 2: If Figure 10 As shown, when the obstacle is outside the clipping area, that is, x0 <x1<x0’,x2<x<x’,y0<y0’> y1,y2>y'>y, then A = the intersection of the pre-cut area obtained according to the line of sight angle and the largest area where the obstacle is located, B = the obstacle annotation box area, C = AU B (the union of A and B), and the final fusion area D = C is completed as the minimum rectangle.

[0078] In another possible implementation, if the second information includes the relative position of the driver and the A-pillar, then when the relative position of the driver and the A-pillar is less than (or less than or equal to) a threshold, the third information is obtained by cropping and amplifying it. Alternatively, when the relative position of the driver and the A-pillar is greater than or equal to (or greater than) the threshold, the third information is obtained by adding the original information and then reducing it. The original information may refer to all or part of the original information of the external environment captured by the second camera device.

[0079] In the above description, when the driver's eyes are close to the A-pillar, the fused information is used as a basis for appropriate cropping, and the cropped area is enlarged for display. However, when the driver's eyes are farther away from the A-pillar, the fused information is used as a basis for the same viewing angle, and the original information is added to the original information. This is then scaled down to reflect the human eye's characteristic of larger near-field vision and smaller far-field vision.

[0080] Optional, such as Figure 11 As shown, when the driver's eyes are close to the A-pillar, the above-mentioned process of appropriate cropping based on the fusion information meets the following conditions, and the final cropped area can be seen in Figure 12 shown.

[0081]

[0082]

[0083] Wherein, dx represents the clipping distance in the x direction, dy represents the clipping distance in the y direction, D1 can be obtained by measuring the distance through the detection device, D3 is the distance the driver's eyes move from position B to position A, which can be detected by the first camera device, β is the video angle of the camera device of the obstacle observed by the first camera device when the driver's eyes are at point B, α is the vertical visual acuity of the human eye, and the comfort zone is approximately 20.

[0084] Optionally, the detection device may also obtain external environment brightness information from an ambient light sensor; adjust the brightness and / or contrast of the third information; and output the adjusted third information. The third information may be displayed or played on a display device located within the vehicle. Based on the external environment brightness, the brightness and / or contrast of the third information may be adjusted at any time, periodically, or as needed to ensure that the driver can clearly see the content displayed or played on the display device, thereby ensuring driving safety.

[0085] The present application also provides a display system, which can be cross-referenced with the description of the above-mentioned display method, including:

[0086] A first camera device is used to obtain first information; at least one sensor device is used to obtain second information, and the second information includes external environment information; a control device is used to determine driver information based on the first information, and the driver information includes at least one of the driver's viewing angle and the relative position information of the driver and the A-pillar, and obtain third information based on the second information, and the third information includes information obtained by processing the second information based on the driver information; a display device is used to display the third information.

[0087] See also Figure 13a or Figure 13bAs shown, the first camera device is a high-definition camera. The number of the high-definition cameras can be one or more, for example, three, which are respectively located above the driver's seat, above the central control screen, and above the left door. The at least one sensing device includes at least one of a second camera device and a detection device. The second camera device is an infrared camera. The number of the infrared cameras can be one or more, for example, two, which are respectively located directly below the left A-pillar and directly below the right A-pillar. The detection device is a radar, such as a millimeter-wave radar. The number of radars can be one or more, for example, two, which are respectively located next to the left headlight and the right headlight of the vehicle body. The display device is a flexible display screen. The number of the flexible display screens can be two, which are respectively attached to the left A-pillar and the right A-pillar. The flexible display screen can perfectly fit the A-pillar without affecting the structural strength of the A-pillar, and can display blind spots, protect the driver's safety, and is low-cost. The control device can be a domain controller in the vehicle or a chip or integrated circuit in the domain controller.

[0088] Optionally, the display system further includes an ambient light sensor for collecting external ambient brightness; and the controller is further configured to adjust the brightness and / or contrast of the third information. The ambient light sensors are located at the bottom of the left and right A-pillars, respectively.

[0089] like Figure 14 As shown, in a possible implementation, when the radar detects a close obstacle, the close obstacle may be a person, an animal or other, and the obstacle information and its relative distance and speed to the vehicle can be transmitted to the domain controller via 100M Ethernet. When the infrared camera captures an object in the external environment, the video information of the external environment is transmitted to the domain controller via 1000M Ethernet. The domain controller fuses the above two pieces of information to form a fused video with annotation information. The annotation information may include the type of obstacle, and / or the relative speed and relative distance between the obstacle and the vehicle. For the specific fusion process, please refer to the above Figure 5 The HD camera can collect driver information and transmit the driver information to the domain controller using 1000M Ethernet as the medium. The domain controller can identify the driver's perspective and the distance from the A-pillar based on the above driver information. Of course, the HD camera can also identify the driver's perspective and the distance from the A-pillar based on the collected driver information, and send the driver's perspective and the distance from the A-pillar directly to the domain controller without further identification by the domain controller. The domain controller can crop the fused information based on the driver's perspective and the annotation information in the fused video. For the process of cropping the above fused video, please refer to the above Figure 6Furthermore, in the above cropping process, in order to avoid cropping the obstacle area and ensure that the obstacle is always displayed on the A-pillar display screen, the cropped video can be fused again with the largest area around the obstacle. The specific process can be found in Figure 8 、 Figure 9 or Figure 10 Afterwards, the domain controller can use the above fused video to enhance character recognition, that is, to zoom in or out according to the distance between the driver and the A-pillar, so as to conform to the human eye's field of view, that is, objects that are larger when near and smaller when far away. The ambient light sensor transmits the brightness information of the external environment to the domain controller via the universal asynchronous receiver / transmitter (UART). The domain controller adjusts the brightness and contrast of the above enhanced video, and transmits it to the A-pillar flexible display screen via 1000M Ethernet for display. In the above Figure 14 In the solution, a universal communication architecture is adopted, integrating new communication links of cameras and radars, and effectively reducing the weight and cost of wiring harnesses through in-vehicle Ethernet.

[0090] From the above, it can be seen that when the radar detects an obstacle, the domain controller fuses the obstacle information detected by the radar with the video collected by the infrared camera to form a fused video with labeled information. The domain controller crops the fused information based on the driver's perspective information and the labeled information in the fused video. Combined with the distance between the driver and the A-pillar, the domain controller further selects a more interesting region on the pre-cropped region of interest (ROI) with the original fused video as a supplement, and scales it, and always ensures that the labeled box of the detected obstacle is displayed in the video. The ROI refers to the area to be processed in the image being processed in the form of a square, circle, ellipse or irregular polygon in the fields of machine vision or image processing. In the embodiment of the present application, the ROI may refer to the area blocked by the A-pillar. The domain controller receives the external environment information collected by the ambient light sensor, adjusts the brightness and contrast, and further processes the video. The domain controller outputs the processed final video to the A-pillar flexible display screen for display.

[0091] Optionally, the display system can also include an audio warning device for broadcasting obstacle information. For example, information such as the obstacle type, the relative distance from the vehicle, or the speed of the vehicle can be displayed. This alerts the driver to the presence of obstacles and ensures safe driving. This multi-dimensional warning, combining audio and visual information, makes the warning more effective.

[0092] In one possible implementation, the process of determining the driver's viewing angle and the distance between the driver and the A-pillar based on the driver information collected by the high-definition camera is as follows:

[0093] like Figure 15 As shown, the main view of the cockpit is obtained by using the image information collected by the high-definition camera located above the central control screen, and the left view of the cockpit is obtained by using the image information collected by the high-definition camera located above the left door; the top view of the cockpit is obtained by using the image information collected by the high-definition camera above the cockpit.

[0094] Distance between the driver's eyes and the A-pillar (Formula 1.5)

[0095] Among them, dx3 represents the horizontal distance between the eye and the A-pillar in the main view of the cockpit, dy3 represents the vertical distance between the eye and the A-pillar in the top view of the cockpit, and dy2 represents the vertical distance between the eye and the A-pillar in the left view of the cockpit.

[0096] Furthermore, the vertical angle θ2 of the human eye's viewing angle in the left view of the cockpit is the x-direction mapping angle in the above formula 1.1, and the horizontal angle θ3 of the human eye's viewing angle in the top view of the cockpit is the y-direction mapping angle in the above formula 1.1.

[0097] In this embodiment, radar and infrared camera information is fused to display and annotate identified obstacles, improving recognition. Dynamic ROI selection is performed on the fused video, combining the driver's perspective and distance from the A-pillar. This improves human-computer interaction, focuses on key information, and ensures that obstacles are always displayed on the A-pillar display.

[0098] Above, combined Figures 1 to 15 The method provided in the embodiment of the present application is described in detail. Figure 16 and Figure 17 The apparatus provided in the embodiments of the present application will be described in detail. It should be understood that the description of the apparatus embodiment corresponds to the description of the method embodiment. Therefore, for matters not described in detail, reference can be made to the description of the method embodiment above.

[0099] Figure 16It is a schematic block diagram of the device 1600 provided in an embodiment of the present application, which is used to implement the functions of the control device in the above method. For example, the device can be a software module or a chip system. The chip can be composed of chips, or can include chips and other discrete devices. The device 1600 includes a communication unit 1601 and a processing unit 1602. The communication unit 1601 can communicate with other devices and can also be called a communication interface, a transceiver unit or an input / output interface, etc. Optionally, the device 1600 can be a vehicle-mounted terminal, or a chip or circuit configured in a vehicle-mounted terminal. Alternatively, the device 1600 can be a vehicle-mounted central processing unit, or a chip or circuit configured in a vehicle-mounted central processing unit, etc. Alternatively, the device 1600 can be an intelligent cockpit domain controller (CDC), or a chip or circuit configured in a CDC, etc.

[0100] In a possible implementation, the communication unit 1601 is configured to execute the transmission and reception related operations of the control device in the above method embodiment, and the processing unit 1602 is configured to execute the processing related operations of the control device in the above method embodiment.

[0101] For example, the communication unit 1601 is used to obtain first information from a first camera device; the processing unit 1602 is used to determine driver information based on the first information, and the driver information includes at least one of the driver's viewing angle and the relative position information of the driver and the A-pillar; the communication unit 1601 is also used to obtain second information from at least one sensing device, and the second information includes external environment information; the processing unit 1602 is also used to obtain third information based on the second information, and the third information includes information obtained by processing the second information based on the driver information.

[0102] Optionally, obtaining the third information based on the second information includes: obtaining the third information through at least one of cropping and fusion based on the driver information and the location information of the obstacle area in the second information.

[0103] Optionally, the clipping process includes: clipping the second information according to the driver's perspective.

[0104] Optionally, the cropping of the second information according to the driver's perspective includes: cropping the second information according to a cropping distance dx in the x-direction and a cropping distance dy in the y-direction, the driver's perspective includes an x-direction mapping angle and a y-direction mapping angle, the dx is determined according to the x-direction mapping angle, and the dy is determined according to the y-direction mapping angle.

[0105] Optionally, the fusion process includes: fusing the cropped second information with the location information of the obstacle area in the second information.

[0106] Optionally, when the relative position of the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and enlarging; or, when the relative position of the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and shrinking it.

[0107] Optionally, the at least one sensing device includes at least one of a second camera device and a detection device, and obtaining the second information from the at least one sensing device includes: obtaining fusion information from the at least one sensing device, wherein the fusion information integrates the speed and / or distance information of the obstacle.

[0108] Optionally, the communication unit 1601 is further configured to obtain brightness information of the external environment from the ambient light sensor; and the processing unit 1602 is further configured to adjust the brightness and / or contrast of the third information.

[0109] Optionally, the processing unit 1602 is further configured to output the third information.

[0110] The division of units in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, in the embodiments of the present application, each functional unit may be integrated into a single processor, or may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0111] Figure 17 is a schematic block diagram of the apparatus 1700 provided in an embodiment of the present application, Figure 17 The device 1700 shown can be Figure 16 A hardware circuit implementation of the device shown. This device can perform the functions of the control device in the above method embodiment. For the convenience of explanation, Figure 17 Only the main components of the device are shown.

[0112] Figure 17 The illustrated device 1700 includes at least one processor 1701. Device 1700 may also include at least one memory 1702 for storing program instructions and / or data. Memory 1702 is coupled to processor 1701. Coupling in the embodiments of this application refers to information exchange between devices, units, or modules. Processor 1701 and memory 1702 can operate in conjunction with each other, executing program instructions stored in memory 1702. At least one of the at least one memory 1702 may be included in processor 1701.

[0113] Apparatus 1700 may further include a communication interface 1703 for communicating with other devices via a transmission medium, thereby enabling apparatus 1700 to communicate with other devices. In embodiments of the present application, the communication interface may be a transceiver, circuit, bus, module, or other type of communication interface. In embodiments of the present application, when the communication interface is a transceiver, the transceiver may include an independent receiver or an independent transmitter; a transceiver with integrated transceiver functions, or an interface circuit may also be provided. Optionally, the communication interface may be an antenna, comprising at least one receiving antenna and at least one transmitting antenna.

[0114] It should be understood that the connection medium between the processor 1701, the memory 1702 and the communication interface 1703 is not limited in the embodiment of the present application. Figure 17 The memory 1702, the processor 1701 and the communication interface 1703 are connected via a communication bus 1704. Figure 17 The connections between the other components are shown in bold, which is only for illustration and not for limitation. The bus may include an address bus, a data bus, a control bus, etc. Figure 17 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0115] In one example, the apparatus 1700 is used to implement the steps performed by the control apparatus in the above method embodiment. The communication interface 1703 is used to perform the transceiver-related operations of the control apparatus in the above method embodiment, and the processor 1701 is used to perform the processing-related operations of the control apparatus in the above method embodiment.

[0116] For example, the communication interface 1703 is used to obtain first information from a first camera device; the processor 1701 is used to determine driver information based on the first information, and the driver information includes at least one of the driver's viewing angle and the relative position information of the driver and the A-pillar; the communication interface 1703 is also used to obtain second information from at least one sensing device, and the second information includes external environment information; the processor 1701 is also used to obtain third information based on the second information, and the third information includes information obtained by processing the second information based on the driver information.

[0117] Optionally, obtaining the third information based on the second information includes: obtaining the third information through at least one of cropping and fusion based on the driver information and the location information of the obstacle area in the second information.

[0118] Optionally, the cropping process includes: cropping the second information according to the driver's perspective.

[0119] Optionally, the cropping of the second information according to the driver's perspective includes: cropping the second information according to a cropping distance dx in the x-direction and a cropping distance dy in the y-direction, the driver's perspective includes an x-direction mapping angle and a y-direction mapping angle, the dx is determined according to the x-direction mapping angle, and the dy is determined according to the y-direction mapping angle.

[0120] Optionally, the fusion process includes: fusing the cropped second information with the location information of the obstacle area in the second information.

[0121] Optionally, when the relative position of the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and enlarging; or, when the relative position of the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and shrinking it.

[0122] Optionally, the at least one sensing device includes at least one of a second camera device and a detection device, and obtaining the second information from the at least one sensing device includes: obtaining fusion information from the at least one sensing device, wherein the fusion information integrates the speed and / or distance information of the obstacle.

[0123] Optionally, the communication interface 1703 is further used to obtain brightness information of the external environment from the ambient light sensor;

[0124] The processor 1701 is further configured to adjust the brightness and / or contrast of the third information.

[0125] Optionally, processor 1701 is further configured to output the third information.

[0126] The embodiment of the present application also provides a terminal, which is a drone, an unmanned transport vehicle, a car, a vehicle, or a robot. In one design, the terminal includes at least one controller, which may include the above-mentioned Figure 16 or Figure 17 In another design, the terminal includes the above-mentioned Figure 16 or Figure 17 The device shown can be independently set or integrated into at least one controller included in the terminal, or integrated into an intelligent cockpit domain controller (CDC) or an on-board central processing unit included in the terminal.

[0127] Furthermore, an embodiment of the present application also provides a device, including a unit for implementing the above method embodiment. Alternatively, it includes a processor and an interface circuit, and the processor is used to communicate with other devices through the interface circuit and execute the method in the above method embodiment. Alternatively, the device includes a processor for calling a program stored in a memory to execute the method in the above method embodiment. An embodiment of the present application also provides a readable storage medium, including instructions, which, when run on a computer, enables the computer to execute the method in the above method embodiment. An embodiment of the present application also provides a chip system, which includes a processor and may also include a memory for implementing the method in the above method embodiment. The chip system can be composed of a chip, or it can include a chip and other discrete devices. An embodiment of the present application also provides a computer program product, including instructions, which, when run on a computer, enables the computer to execute the method in the above method embodiment.

[0128] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0129] In an embodiment of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in an embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0130] The methods provided in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in accordance with the embodiments of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, an SSD).

[0131] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A display method, characterized in that: include: Acquiring first information from a first camera device; determining driver information based on the first information, the driver information including at least one of a viewing angle of the driver and relative position information between the driver and an A-pillar; obtaining second information from at least one sensing device, the second information including external environment information and annotated information, the annotated information including a type, speed, and / or distance information of an obstacle, the at least one sensing device including a second camera device and a detection device, the second camera device being configured to capture images or video information of the external environment, and the detection device being configured to detect speed and / or distance information of obstacles in the external environment; acquiring third information according to the second information, the third information including information obtained by processing the second information based on the driver information; The acquiring of third information according to the second information includes: obtaining the third information by performing at least one of cropping and fusion based on the driver information and the location information of the obstacle area in the second information; The clipping process includes: clipping the second information according to the driver's perspective; The fusion process includes: fusing the cropped second information with the location information of the obstacle area in the second information.

2. The method according to claim 1, wherein The step of clipping the second information according to the driver's perspective includes: The second information is clipped according to the clipping distance dx in the x direction and the clipping distance dy in the y direction. The driver's perspective includes an x-direction mapping angle and a y-direction mapping angle. The dx is determined according to the x-direction mapping angle, and the dy is determined according to the y-direction mapping angle.

3. The method according to claim 1 or 2, wherein: When the relative position between the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and amplifying; or, When the relative position between the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and reducing it.

4. The method according to claim 1 or 2, wherein: The at least one sensing device includes at least one of a second camera device and a detection device, and obtaining second information from the at least one sensing device includes: Fused information is obtained from the at least one sensing device, where the fused information incorporates speed and / or distance information of the obstacle.

5. The method according to claim 1 or 2, wherein: The method further comprises: Obtaining brightness information of the external environment from the ambient light sensor; The brightness and / or contrast of the third information is adjusted.

6. The method according to claim 1 or 2, wherein: The method further comprises: The third information is output.

7. A processing device, characterized in that: include: a communication unit, configured to obtain first information from a first camera device; a processing unit, configured to determine driver information based on the first information, the driver information comprising at least one of a viewing angle of the driver and relative position information between the driver and an A-pillar; The communication unit is further configured to obtain second information from at least one sensing device, the second information including external environment information and annotation information, the annotation information including type, speed, and / or distance information of an obstacle, the at least one sensing device including a second camera device and a detection device, the second camera device being configured to capture images or video information of the external environment, and the detection device being configured to detect speed and / or distance information of obstacles in the external environment; The processing unit is further configured to obtain third information based on the second information, wherein the third information includes information obtained by processing the second information based on the driver information; The acquiring third information according to the second information includes: obtaining the third information by performing at least one of cropping and fusion based on the driver information and the location information of the obstacle area in the second information; The clipping process includes: clipping the second information according to the driver's perspective; The fusion process includes: fusing the cropped second information with the location information of the obstacle area in the second information.

8. The device according to claim 7, wherein The step of clipping the second information according to the driver's perspective includes: The second information is clipped according to the clipping distance dx in the x direction and the clipping distance dy in the y direction. The driver's perspective includes an x-direction mapping angle and a y-direction mapping angle. The dx is determined according to the x-direction mapping angle, and the dy is determined according to the y-direction mapping angle.

9. The device according to claim 7 or 8, characterized in that When the relative position between the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and amplifying; or, When the relative position between the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and reducing it.

10. The device according to claim 7 or 8, characterized in that The at least one sensing device includes at least one of a second camera device and a detection device, and obtaining second information from the at least one sensing device includes: Fused information is obtained from the at least one sensing device, where the fused information incorporates speed and / or distance information of the obstacle.

11. The device according to claim 7 or 8, characterized in that The communication unit is further configured to obtain brightness information of the external environment from the ambient light sensor; The processing unit is further configured to adjust the brightness and / or contrast of the third information.

12. The device according to claim 7 or 8, characterized in that The processing unit is further configured to control the display unit to output the third information.

13. A processing device, characterized in that: The device comprises at least one processor and at least one memory, wherein the at least one memory stores instructions, and when the at least one processor executes the instructions, the device causes the device to execute the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

15. A display system, characterized in that: include: A first camera device, configured to obtain first information; at least one sensing device, configured to acquire second information, the second information including external environment information and annotated information, the annotated information including the type, speed, and / or distance information of an obstacle, the at least one sensing device comprising a second camera device and a detection device, the second camera device being configured to capture images or video information of the external environment, and the detection device being configured to detect the speed and / or distance information of obstacles in the external environment; a control device configured to determine driver information based on the first information, the driver information including at least one of a viewing angle of the driver and information about a relative position of the driver and an A-pillar, and to obtain third information based on the second information, the third information including information obtained by processing the second information based on the driver information; a display device, configured to display the third information; The acquiring of the third information according to the second information includes: acquiring the third information by performing at least one of clipping and fusing according to the driver information and the position of the obstacle in the second information; The clipping process includes: clipping the second information according to the driver's perspective; The fusion process includes: fusing the cropped second information with the location information of the obstacle area in the second information.

16. The system according to claim 15, wherein: The first camera device is a high-definition camera, and there are three high-definition cameras, which are respectively located above the driver's seat, above the central control screen, and above the left door.

17. The system according to claim 15 or 16, characterized in that The at least one sensing device includes at least one of a second camera device and a detection device, wherein the second camera device is an infrared camera, and there are two infrared cameras, which are respectively located directly below the left A-pillar and directly below the right A-pillar; the detection device is a radar, and there are two radars, which are respectively located next to the left headlight and the right headlight of the vehicle body.

18. The system according to claim 15 or 16, characterized in that The display device is a flexible display screen, and there are two flexible display screens, which are respectively attached to the left A-pillar and the right A-pillar.

19. The system according to claim 15 or 16, wherein: The step of clipping the second information according to the driver's perspective includes: The second information is clipped according to the clipping distance dx in the x direction and the clipping distance dy in the y direction. The driver's perspective includes an x-direction mapping angle and a y-direction mapping angle. The dx is determined according to the x-direction mapping angle, and the dy is determined according to the y-direction mapping angle.

20. The system according to claim 15 or 16, wherein: When the relative position between the driver and the A-pillar is less than a threshold, the third information is obtained by cropping and amplifying; or, When the relative position between the driver and the A-pillar is greater than or equal to a threshold, the third information is obtained by adding the original information and reducing it.

21. The system according to claim 15 or 16, wherein: The obtaining of the second information includes: Fused information is obtained from the at least one sensing device, where the fused information is combined with speed and / or distance information of the obstacle.

22. The system according to claim 15 or 16, wherein: Also includes: Ambient light sensor, used to collect external ambient brightness; The control device is further configured to adjust the brightness and / or contrast of the third information.

23. The system of claim 22, wherein: The ambient light sensors are located at the bottom of the left A-pillar and the bottom of the right A-pillar respectively.

Images