Image fusion display method, device and system based on ADAS camera

Abstract

The embodiment of the invention provides an image fusion display method, device and system based on an ADAS camera. The method comprises the following steps: splicing images of two side view fields of a vehicle and images of a rear view field of the vehicle collected by an ADAS camera of the vehicle to obtain a target image, and displaying the target image in a preset display area of the vehicle. By means of the method, the technical scheme of the electronic rearview mirror capable of integrating view fields of multiple cameras and optimizing cost is provided, and the problems of system complexity, high cost and the like are solved. The perception ability of the vehicle is optimized, the smoothness and safety of driving experience are improved, and the requirements of modern vehicles for intelligence and integration are met.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to an image fusion display method, device and system based on an ADAS camera. Background Art

[0002] With the development of intelligent technology, vehicle streaming media interior rearview mirrors and electronic exterior rearview mirrors are gradually applied to the field of passenger cars. Compared with traditional physical rearview mirrors, electronic exterior rearview mirrors can better solve the rear vision problem in rainy and foggy weather and poor road lighting environment.

[0003] In the current mainstream electronic interior and exterior rearview mirror solutions, three display screens are usually installed, which are installed at the lower left and right of the A-Pillar and directly below the interior rearview mirror. Each screen only displays the field of view of a specific area around the vehicle, and due to the screen installation position, the driver needs to turn his head frequently to switch screens, affecting the experience and driving safety. The left rearview mirror, right rearview mirror and interior rearview mirror are each equipped with an independent control unit. Each control unit is responsible for processing the signal of its corresponding camera and transmitting the processed image to the corresponding display screen. The additional cameras also increase the cost of the entire vehicle.

[0004] In summary, providing an electronic rearview mirror technical solution that can further optimize the cost of the entire vehicle is an issue that needs to be urgently addressed. Summary of the invention

[0005] The present application provides an image fusion display method, device and system based on an ADAS camera, which are used to optimize user experience, improve driving safety, and reduce system complexity and cost, provide a more comprehensive and continuous field of view, reduce dependence on physical cameras and control units, and further optimize the cost of the electronic rearview mirror technology solution for the entire vehicle.

[0006] In a first aspect, the present application provides an image fusion display method based on an ADAS camera, which is applied to an autonomous driving domain controller or a cockpit domain controller of a vehicle, and the method comprises:

[0007] Acquire images of the fields of view on both sides of the vehicle and the field of view behind the vehicle captured by the ADAS camera of the vehicle;

[0008] The images of the fields of view on both sides of the vehicle are stitched together with the image of the field of view behind the vehicle to obtain a target image;

[0009] The target image is displayed in a preset display area in the vehicle.

[0010] In a specific implementation, displaying the target image in a preset display area in the vehicle includes:

[0011] Displaying the target image in a display area configured around a dashboard of the vehicle;

[0012] or,

[0013] Displaying the target image in a display area of ​​an interior rearview mirror of the vehicle;

[0014] or,

[0015] The target image is projected and displayed in a preset area on the windshield in front of the driving position of the vehicle.

[0016] In a specific implementation, the acquiring of the images of the fields of view on both sides of the vehicle and the images of the field of view behind the vehicle collected by the ADAS camera of the vehicle includes:

[0017] Acquire images of the fields of view on both sides of the vehicle through the side view cameras arranged on both sides of the vehicle in the ADAS camera;

[0018] An image of the rear view of the vehicle is acquired through a rear view camera arranged at the rear of the vehicle in the ADAS camera.

[0019] In a specific embodiment, the method further comprises:

[0020] If fogging of the camera in any field of view is detected, the camera is controlled to be heated.

[0021] In a specific implementation, the step of splicing the images of the fields of view on both sides of the vehicle with the image of the field of view behind the vehicle to obtain a target image includes:

[0022] The images of the fields of view on both sides of the vehicle are spliced ​​with the non-common view areas in the image of the field of view behind the vehicle, and the images of the fields of view on both sides of the vehicle are fused with the common view areas in the image of the field of view behind the vehicle to obtain the target image.

[0023] In a second aspect, the present application provides an image fusion display device based on an ADAS camera, the device comprising:

[0024] An acquisition module, used to acquire images of the fields of view on both sides of the vehicle and the field of view behind the vehicle collected by the ADAS camera of the vehicle;

[0025] A processing module, used for splicing the images of the fields of view on both sides of the vehicle and the image of the field of view behind the vehicle to obtain a target image;

[0026] A display module is used to display the target image in a preset display area in the vehicle.

[0027] In a third aspect, the present application provides a controller, including: a memory, a processor;

[0028] The memory stores computer-executable instructions;

[0029] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0030] In a specific implementation, the controller is an autonomous driving domain controller or a cockpit domain controller.

[0031] In a fourth aspect, the present application provides a vehicle, comprising: the controller described in the third aspect, an ADAS camera arranged outside the vehicle, and a display screen arranged inside the vehicle for displaying an image obtained by splicing an external field of view.

[0032] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementations of the first aspect.

[0033] The present application provides an image fusion display method, device and system based on an ADAS camera. In the solution, the images of the vehicle's two side fields of view and the image of the vehicle's rear field of view collected by the vehicle's ADAS camera are spliced ​​to obtain a target image, and the target image is displayed in a preset display area of ​​the vehicle. Through the above method, an electronic rearview mirror technical solution that can further optimize the cost of the entire vehicle is provided, solving problems such as system complexity and high cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] Figure 1 A schematic diagram of a flow chart of a first embodiment of an image fusion display method based on an ADAS camera provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of another specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application;

[0038] Figure 4A schematic diagram of another specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of another specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of an embodiment of an image fusion display device based on an ADAS camera provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of the structure of a controller provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of a vehicle provided in accordance with an embodiment of the present application.

[0043] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0045] Before introducing the embodiments of the present application, the application background of the embodiments of the present application is explained first:

[0046] An electronic rearview mirror is an indirect vision device that obtains a specified field of view through a system composed of a camera and a monitor. It includes electronic equipment such as a high-definition camera, a digital visual processing system, a safety system, and a liquid crystal display. It is a new type of rearview mirror that can replace traditional optical rearview mirrors. On July 1, 2023, the "Performance and Installation Requirements for Indirect Vision Devices for Motor Vehicles" was officially implemented. This new national standard added the technical requirements for the Camera Monitor System (CMS), i.e., the electronic rearview mirror, which means that cars can be equipped with electronic rearview mirrors to replace traditional optical rearview mirrors on the road.

[0047] Compared with traditional physical rearview mirrors, which are limited by the vehicle body structure and angle, and the driver can only see a limited picture, the electronic rearview mirror has a wider field of view through the combination of the main lens, the wide-angle lens and the display screen to present the image, eliminating blind spots and reducing the occurrence of accidents. In addition, the electronic rearview mirror performs well in adverse weather conditions. Its special waterproof design and coating effectively reduce the impact of rain and mud, while avoiding the problem of fogging of the windows. When driving at night, the electronic rearview mirror improves the visual effect by enhancing the brightness and contrast of the image, and reduces the glare caused by the high beam of the rear vehicle. In addition, since the electronic rearview mirror optimizes air resistance, it can also reduce fuel consumption. The electronic rearview mirror also has a variety of additional functions, such as automatic brightness adjustment, display of navigation information and driver assistance system warnings, which further improves the convenience of driving.

[0048] In the current mainstream electronic interior and exterior rearview mirror solutions, three display screens are usually installed, which are installed on the lower left and right of the A-pillar and directly below the interior rearview mirror. Each screen only displays the field of view of a specific area around the vehicle, and due to the screen installation position, the driver needs to frequently turn his head to view different screens in order to obtain complete environmental information, which not only affects the driving experience, but may also distract the driver's attention and bring potential safety hazards. In addition, in the current mainstream electronic interior and exterior rearview mirror solutions, the left rearview mirror, right rearview mirror and interior rearview mirror are each equipped with an independent control unit. Each control unit is responsible for processing the signal of its corresponding camera and transmitting the processed image to the corresponding display screen. The additional cameras also complicate the design and manufacturing process of the vehicle and increase the cost of the entire vehicle.

[0049] In summary, although electronic interior and exterior rearview mirrors offer many technical advantages, they still need to solve problems in field of vision integration, driver convenience and cost control in practical applications to achieve wider popularization and more efficient use. Therefore, providing an electronic rearview mirror technology solution that can further optimize vehicle costs and enhance driver experience is an urgent problem to be solved.

[0050] Based on the above technical problems, the inventors found that by combining the integrated internal and external rearview mirrors with the controller and the display screen, the system architecture can be effectively simplified, the manufacturing cost can be reduced, and a seamless panoramic view can be provided. It not only reduces the dependence on multiple independent display screens, but also integrates the perspectives of different cameras to provide the driver with a unified and intuitive display of environmental information, which improves the driver's driving experience and safety while reducing the cost of the whole vehicle. Based on this, the present application provides an image fusion display method, device and system based on an ADAS camera. In this solution, the images of the two sides of the vehicle's field of view and the image of the rear field of view of the vehicle collected by the vehicle's Advanced Driving Assistance System (ADAS) camera are spliced ​​to obtain a target image, and the target image is displayed in a preset display area of ​​the vehicle. Through the above method, a technical solution for an electronic rearview mirror that can integrate the field of view of multiple cameras and optimize costs is provided, which solves the problems of discontinuous field of view, poor user experience, system complexity and high cost. It not only optimizes the perception ability of the vehicle, but also improves the smoothness and safety of the driving experience, meeting the needs of modern automobiles for intelligence and integration.

[0051] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0052] Figure 1 The following is a flow chart of an embodiment 1 of an image fusion display method based on an ADAS camera provided in an embodiment of the present application. Figure 2 A schematic diagram of a specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application, such as Figure 1 As shown, the image fusion display method based on ADAS camera includes:

[0053] S101: Acquire images of the fields of view on both sides of the vehicle and the field of view on the rear of the vehicle collected by the ADAS camera of the vehicle.

[0054] In this step, for cost control reasons, this solution uses ADAS cameras to obtain images. ADAS cameras are key components in modern cars to support various driving assistance functions, such as Figure 2As shown, it is usually installed in front, behind or on the side of the vehicle, responsible for capturing real-time images of the vehicle's surroundings. The front-view monocular or binocular camera is usually installed on the front windshield with a viewing angle of about 45°, and is used for front vehicle collision warning, lane departure warning, traffic sign recognition, and pedestrian collision warning. The rear-view wide-angle camera is usually installed on the rear trunk to obtain the rear view of the vehicle. The side-view camera is usually installed below the rearview mirror for blind spot monitoring. The surround wide-angle camera is usually installed around the car and equipped with four cameras for image stitching to achieve a panoramic view, and an algorithm is added to achieve road perception. The interior wide-angle camera is usually installed outside the rearview mirror inside the car to monitor the status of the car, and is used for driver fatigue reminder, emotion recognition, gesture recognition, etc. ADAS cameras have high resolution and wide-angle field of view, and can recognize and detect a variety of objects and features such as road signs, lane lines, pedestrians, and other vehicles.

[0055] Specifically, the side view cameras arranged on both sides of the vehicle in the ADAS camera are used to obtain images of the fields of view on both sides of the vehicle, including the images of the left rear field of view and the right rear field of view of the vehicle; the rear view camera arranged at the rear of the vehicle in the ADAS camera is used to obtain images of the rear field of view of the vehicle. With the help of the versatility of the ADAS camera, the electronic rearview mirror system is optimized and improved without adding additional cameras. Through the above method, the existing hardware resources are fully utilized, and the complexity and cost of the system are reduced.

[0056] S102: splicing the images of the fields of view on both sides of the vehicle and the image of the field of view behind the vehicle to obtain a target image.

[0057] In this step, since each screen only displays the field of view of a specific area around the vehicle, and is limited by the screen installation position, the driver needs to frequently turn his head to switch screens, which affects the experience and driving safety. In this solution, the images of the field of view on both sides of the vehicle are spliced ​​with the images of the field of view behind the vehicle to obtain the target image. By integrating the side and rear views of the vehicle captured by the ADAS camera into a continuous panoramic image and displaying it in the preset display area of ​​the vehicle, a seamless field of view transition is provided, eliminating the blind spots and field of view faults that may exist in traditional multi-screen solutions. The driver can obtain complete information about the vehicle's surrounding environment without frequently turning his head to view different display screens, which greatly improves the driving experience and driving safety and reduces the driver's operating burden.

[0058] Specifically, the images of the fields of view on both sides of the vehicle and the images of the fields of view behind the vehicle are stitched together to obtain a target image, which specifically includes: stitching the images of the fields of view on both sides of the vehicle with the non-common view areas in the images of the fields of view behind the vehicle, and fusing the images of the fields of view on both sides of the vehicle with the common view areas in the images of the fields of view behind the vehicle to obtain a target image.

[0059] In a specific embodiment, the non-common view area refers to the part of the field of view covered by only one camera, and the common view area refers to the part of the field of view covered by multiple cameras at the same time. In order to ensure that the images can show consistency and continuity in the overlapping area covered by multiple cameras and provide a seamless visual experience, the images of the fields of view on both sides of the vehicle are fused with the common view area in the image of the field of view behind the vehicle. Specifically, the fusion process involves color correction, brightness adjustment and geometric correction to eliminate image differences caused by different camera positions and angles. In color correction, color mapping and histogram matching algorithms are used to adjust the color distribution of the image so that the images from different cameras are more consistent in color tone; in brightness adjustment, linear or nonlinear transformations are used to adjust the brightness level of the image so that the brightness of different images remains consistent in the overlapping area; in geometric correction, feature point matching methods are used to identify and match feature points in the image, and the transformation matrix between the images is calculated, so that the image is geometrically corrected so that the images of the fields of view on both sides of the vehicle and the common view area in the image of the field of view behind the vehicle are spatially aligned.

[0060] In addition, edge detection and feature matching techniques are used to ensure that objects in the common view area, such as vehicles and non-motor vehicles, maintain consistent shapes and positions in the stitched image. In edge detection, the goal is to identify the boundary or contour area of ​​objects with significant brightness changes in the image and extract important structural information in the image. First, the Canny algorithm uses a Gaussian filter to smooth the image through convolution operations to eliminate the interference of high-frequency noise. The standard deviation of the filter determines the degree of smoothing. The larger the standard deviation, the more obvious the smoothing effect. Then the gradient strength and direction of the image are calculated to identify potential edges. The gradient strength refers to the measure of the change in the grayscale value of the image, and the gradient direction indicates the direction of the change. By calculating the gradient components of the image in the horizontal and vertical directions, the gradient strength and direction of each pixel can be obtained. Areas with larger gradient strength usually correspond to edges in the image. Then, unnecessary edge responses are removed by non-maximum suppression, and the local maximum is retained. For each pixel in the image, the algorithm checks its neighboring pixels in the gradient direction and compares the gradient strength. If the gradient strength of the current pixel is not the local maximum, it is suppressed to zero, and only the most significant edge point in the gradient direction is retained. Finally, the double threshold method is used to connect the edges to ensure the continuity of the edges. The double threshold method involves a high threshold and a low threshold. The algorithm marks pixels with gradient strength higher than the high threshold as strong edges and suppresses pixels with gradient strength lower than the low threshold as non-edges. For pixels between the two, check whether they are connected to any strong edge. If the pixel is connected to any strong edge, it is marked as an edge, otherwise it is suppressed as a non-edge. The double threshold method can effectively connect edges, reduce breaks, and suppress false edges caused by noise.

[0061] In the feature matching method, the scale invariant feature transform (SIFT) algorithm is first used to detect key points in the image. The scale space is constructed by Gaussian blurring the image and repeating this process at different scales. Specifically, SIFT generates a series of image pyramids of different scales by Gaussian filtering the image. In these scale spaces, SIFT calculates Gaussian differences, that is, calculates difference images between adjacent scales to enhance the potential features in the image. By finding local extreme points in the difference image, SIFT can detect key points in the image that remain stable under scale changes and rotations. Then SIFT accurately locates and filters each key point. Specifically, in order to improve the stability of the key point, SIFT accurately locates the position and scale of the key point by fitting a three-dimensional quadratic function, and removes low-contrast key points and key points located at the edge to reduce the impact of noise. SIFT then generates a feature descriptor for each key point. Specifically, in order to achieve rotation invariance, SIFT first calculates the gradient direction histogram in the neighborhood of the key point and selects the main direction as the direction of the key point. Then, within the neighborhood of the key point, SIFT divides it into several sub-regions based on the main direction of the key point. In each sub-region, the gradient direction and amplitude are calculated, and this information is summarized into a feature vector. Finally, SIFT generates a 128-dimensional feature descriptor, which is robust to illumination changes, image noise, and affine transformations to a certain extent. Finally, SIFT identifies matching objects by comparing the feature descriptors of key points in different images. In order to improve the accuracy of matching, SIFT uses the nearest neighbor ratio test, that is, for each key point, find its two nearest neighbor points in the other image and calculate the ratio of the Euclidean distance of the descriptors of the two points. If the ratio is lower than a certain threshold, the match is considered reliable.

[0062] Through the above method, an image is obtained by fusion of the common viewing area in the image of the field of view on both sides of the vehicle and the image of the field of view behind the vehicle. This method not only improves the visual quality of the image, but also enhances the driver's perception of the surrounding environment.

[0063] Finally, the target image is obtained by merging the images of the common viewing area in the images of the two sides of the vehicle and the image of the rear view with the images of the non-common viewing area in the images of the two sides of the vehicle and the image of the rear view. This method effectively solves the problems of visual field faults and blind spots that may exist in traditional multi-screen solutions, improving driving safety and user experience.

[0064] S103: Displaying a target image in a preset display area in the vehicle.

[0065] In this step, the fused and stitched target image is displayed in a preset display area of ​​the vehicle, providing the driver with a complete and continuous field of view.

[0066] Figure 3 A schematic diagram of another specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application; Figure 4 This is a schematic diagram of another specific implementation of an image fusion display method based on an ADAS camera provided in an embodiment of the present application. Specifically, taking into account the sense of unity of the cockpit shape and the convenience of driver observation, such as Figure 3 As shown, the target image is displayed in a display area configured around the dashboard of the vehicle; or, the target image is projected and displayed in a preset area on the windshield in front of the driver's seat of the vehicle. Figure 4 As shown, the target image is displayed in the display area of ​​the vehicle's rearview mirror. Through the above display method, the target image information is centrally displayed within the driver's natural field of vision, and the driver does not need to frequently turn his head to switch back and forth to view different screens or rearview mirrors, thereby reducing possible distraction and fatigue during driving, focusing more on road conditions and driving operations, reducing the driver's reaction time in emergency situations, and improving the smoothness and safety of the overall driving experience.

[0067] In a specific implementation, if fogging of the camera in any field of view is detected, the camera is controlled to be heated.

[0068] Specifically, the ADAS camera generates heat during the operation of the mainboard during video recording. When the ambient temperature is low, the lens or internal components of the camera will generate fog, affecting normal video recording, resulting in a decrease in the quality of the images of the vehicle's side view obtained by the side view cameras set on both sides of the vehicle in the ADAS camera and the image of the vehicle's rear view obtained by the rear view camera set at the rear of the vehicle in the ADAS camera, and the image pixels are lower than the preset clarity threshold. It should be noted that this threshold is set by comprehensively considering the basic visual requirements of the vehicle in different driving environments, such as day and night, sunny and rainy days, etc., as well as the technical specifications and performance of the camera, including the resolution of the sensor, the optical quality of the lens, and the capabilities of the image processing algorithm and other actual needs.

[0069] If fogging is detected on the camera in any field of view, the control will heat the camera until the image pixels taken by the camera are higher than the preset clarity threshold, thereby completing the defogging process on the fogged camera and ensuring that the image quality can meet the human eye's perception of details and dynamic changes.

[0070] The image fusion display method based on ADAS camera provided in the embodiment of the present application, in which the image of the field of view on both sides and the rear of the vehicle is obtained by using the existing ADAS camera of the vehicle, and the images are spliced ​​and fused to generate a complete panoramic image. The panoramic image is displayed in the preset display area of ​​the vehicle, such as around the instrument panel, the projection area on the windshield or the display area of ​​the rearview mirror, so as to provide the driver with a continuous and seamless field of view. In addition, the system also has an automatic defog function to ensure that the image quality of the camera in a low temperature environment is always maintained above the preset clarity threshold, thereby improving the reliability of the ADAS system and the visual quality of the image. Through the above method, the existing hardware resources are fully utilized, the cost and complexity of adding additional cameras are avoided, and the blind spots and faults in the traditional multi-screen solution are eliminated through image splicing and fusion technology. The need for the driver to frequently turn his head to view different screens is reduced, the distraction and fatigue during driving are reduced, the concentration on road conditions is improved, and the driver's visual experience and driving safety are significantly improved. Overall, it not only optimizes the perception ability of the vehicle, but also improves the smoothness and safety of the driving experience, and meets the needs of modern automobiles for intelligence and integration.

[0071] Figure 5 A schematic diagram of another specific implementation method of an ADAS camera-based image fusion method provided in the present application embodiment is provided. Based on the above embodiment, the ADAS camera-based image fusion display method is applied to the vehicle's automated driving control unit (ADCU) or cockpit domain controller (DHU), aiming to enhance the driver's perception of the vehicle's surrounding environment while reducing system complexity and cost. Figure 5 As shown, the specific implementation process of this method is:

[0072] First, for cost control considerations, this solution uses the ADAS camera installed on the vehicle to obtain the field of view images on the left, right and rear of the vehicle. In traditional rearview mirror image display solutions, each screen only displays the field of view of a specific area around the vehicle, and is limited by the screen installation position. The driver needs to frequently turn his head to switch screens, which affects the experience and driving safety. The autonomous driving domain controller or the cockpit domain controller receives the three-way field of view images collected by the ADAS. This solution splices the three-way field of view images to obtain a continuous panoramic image. Specifically, the splicing process includes fusing the images of the field of view on both sides of the vehicle with the common viewing areas in the image of the rear field of view of the vehicle through color correction, brightness adjustment and geometric correction, as well as edge detection and feature matching methods, and splicing the images of the field of view on both sides of the vehicle with the non-common viewing areas in the image of the rear field of view of the vehicle to obtain the target image, providing a seamless visual experience. The autonomous driving domain controller or cockpit domain controller outputs the target image to the display screen. Taking into account the sense of integration of the cockpit shape and the convenience of driver observation, the integrated display screen can be installed in the display area configured around the vehicle's instrument panel or the preset area on the windshield in front of the vehicle's driver's seat for projection display or in the rearview mirror display area of ​​the vehicle to display the wide-angle target image. This enables the driver to switch back and forth to view different screens or rearview mirrors without frequently turning his head, reducing possible distraction and fatigue during driving, and improving the smoothness and safety of the overall driving experience.

[0073] In addition, to ensure image quality, the system also has a camera fogging detection function. When the camera in any field of view is detected to be fogged, the control module eliminates the fog by heating the lens until the image pixels are above the preset clarity threshold.

[0074] In summary, the image fusion display method based on ADAS camera optimizes and improves the electronic rearview mirror system by making full use of existing hardware resources, reducing the complexity and cost of the system. Through seamless image splicing and display technology, it provides the driver with a complete and continuous field of view, greatly improving the driving experience and driving safety.

[0075] Figure 6 A structural diagram of an embodiment of an image fusion display device based on an ADAS camera provided in an embodiment of the present application, such as Figure 6 As shown, the image fusion display device 60 based on the ADAS camera includes:

[0076] An acquisition module 601 is used to acquire images of the fields of view on both sides of the vehicle and the field of view behind the vehicle collected by the ADAS camera of the vehicle;

[0077] A processing module 602 is used to splice the images of the field of view on both sides of the vehicle and the image of the field of view behind the vehicle to obtain a target image;

[0078] The display module 603 is used to display the target image in a preset display area in the vehicle.

[0079] The image fusion display device based on the ADAS camera provided in this embodiment is used to execute the technical solution in the aforementioned method embodiment, and its implementation principle and technical effect are similar and will not be repeated here.

[0080] In a specific implementation, the display module 603 is specifically used for:

[0081] A display area configured around a dashboard of a vehicle displays a target image;

[0082] or,

[0083] Displaying a target image in a display area of ​​an interior rearview mirror of the vehicle;

[0084] or,

[0085] The target image is projected and displayed in a preset area on the windshield in front of the driving seat of the vehicle.

[0086] In a specific implementation, the acquisition module 601 is specifically used to:

[0087] The side view cameras installed on both sides of the vehicle in the ADAS camera are used to obtain images of the field of view on both sides of the vehicle;

[0088] The rear view image of the vehicle is acquired through the rear view camera arranged at the rear of the vehicle in the ADAS camera.

[0089] In a possible implementation, the ADAS camera-based image fusion display device 600 further includes: a fourth processing module 604, configured to:

[0090] If fogging of the camera in any field of view is detected, the camera is controlled to be heated.

[0091] In a specific implementation, the processing module 602 is specifically used to:

[0092] The images of the fields of view on both sides of the vehicle are spliced ​​with the non-common view areas in the image of the field of view behind the vehicle, and the images of the fields of view on both sides of the vehicle are fused with the common view areas in the image of the field of view behind the vehicle to obtain the target image.

[0093] The image fusion display device based on the ADAS camera provided in any of the above embodiments is used to execute the technical solution in the above method embodiments, and its implementation principle and technical effect are similar, which will not be repeated here.

[0094] Figure 7A schematic diagram of the structure of the controller provided in the embodiment of the present application, such as Figure 7 As shown, the controller 70 is an autonomous driving domain controller or a cockpit domain controller, including: at least one processor 701 and a memory 702. Optionally, the controller 70 also includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0095] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0096] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0097] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0098] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0099] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0100] Figure 8 A schematic diagram of a vehicle is provided for an embodiment of the present application, such as Figure 8As shown, the vehicle 800 includes: a controller, an ADAS camera arranged outside the vehicle, and a display screen arranged inside the vehicle for displaying an image obtained by splicing the external field of view.

[0101] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0102] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0103] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0104] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0107] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0108] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0109] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image fusion display method based on an ADAS camera, characterized in that: The method is applied to an autonomous driving domain controller or a cockpit domain controller of a vehicle, and comprises: Acquire images of the fields of view on both sides of the vehicle and the field of view behind the vehicle captured by an advanced driving assistance system (ADAS) camera of the vehicle; The images of the fields of view on both sides of the vehicle are stitched together with the image of the field of view behind the vehicle to obtain a target image; The target image is displayed in a preset display area in the vehicle.

2. The method according to claim 1, characterized in that The displaying of the target image in a preset display area in the vehicle includes: Displaying the target image in a display area configured around a dashboard of the vehicle; or, Displaying the target image in a display area of ​​an interior rearview mirror of the vehicle; or, The target image is projected and displayed in a preset area on the windshield in front of the driving position of the vehicle.

3. The method according to claim 1 or 2, characterized in that: The obtaining of images of the fields of view on both sides of the vehicle and the field of view on the rear of the vehicle collected by the ADAS camera of the vehicle includes: Acquire images of the fields of view on both sides of the vehicle through the side view cameras arranged on both sides of the vehicle in the ADAS camera; An image of the rear view of the vehicle is acquired through a rear view camera arranged at the rear of the vehicle in the ADAS camera.

4. The method according to claim 3, characterized in that The method further comprises: If fogging of the camera in any field of view is detected, the camera is controlled to be heated.

5. The method according to claim 1 or 2, characterized in that: The step of splicing the images of the fields of view on both sides of the vehicle with the image of the field of view behind the vehicle to obtain a target image includes: The images of the fields of view on both sides of the vehicle and the non-common view areas in the image of the field of view behind the vehicle are spliced, and the images of the fields of view on both sides of the vehicle and the common view areas in the image of the field of view behind the vehicle are fused to obtain the target image.

6. An image fusion display device based on an ADAS camera, characterized in that: The device comprises: An acquisition module, used to acquire images of the fields of view on both sides of the vehicle and the field of view behind the vehicle collected by the ADAS camera of the vehicle; A processing module, used for splicing the images of the fields of view on both sides of the vehicle and the image of the field of view behind the vehicle to obtain a target image; A display module is used to display the target image in a preset display area in the vehicle.

7. A controller, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 5.

8. The controller according to claim 7, characterized in that: The controller is an autonomous driving domain controller or a cockpit domain controller.

9. A vehicle, characterized in that: The vehicle comprises: the controller according to claim 7 or 8, an ADAS camera arranged outside the vehicle, and a display screen arranged inside the vehicle for displaying an image obtained by splicing an external field of view.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

Images