Vehicle-mounted auxiliary display method, display system, display device and readable storage medium

Abstract

This invention relates to an in-vehicle auxiliary display method, display system, display device, and readable storage medium. The in-vehicle auxiliary display method includes: S1, obtaining first image information, which is the original image in the A-pillar blind spot; S2, capturing the origin of the driver's line-of-sight coordinate system in real time; S3, converting the first image information into second image information in the driver's line-of-sight coordinate system; S4, projecting the outer contour of the A-pillar curved screen onto the driver's line-of-sight coordinate system and cropping it to obtain third image information; S5, performing parallax compensation on the third image information to obtain fourth image information; and S6, displaying the fourth image information on the A-pillar curved screen in real time. This invention proposes an in-vehicle auxiliary display method, display system, display device, and readable storage medium to facilitate drivers' viewing of road and pedestrian information behind the A-pillar, avoid unnecessary visual jumps, eliminate the A-pillar blind spot, and improve driving safety.

Description

Technical Field

[0001] This invention relates to the field of vehicle autonomous driving technology, and in particular to an in-vehicle auxiliary display method, display system, display device, and readable storage medium. Background Technology

[0002] The A-pillar blind spot is a common problem encountered by drivers. The A-pillar is a crucial supporting structure in the vehicle body, with very high safety requirements. Its main material is typically high-strength metal to ensure structural strength, while the exterior is painted for aesthetic purposes, and the interior is fitted with materials. During normal driving, the driver cannot see objects in the blind spot behind the A-pillar, leading to accidents. Existing technologies offer several solutions to address the A-pillar blind spot. These existing technical solutions are as follows:

[0003] 1. When designing the vehicle, high-strength materials should be used as much as possible to minimize the blind spot area while ensuring the strength of the A-pillar. However, this solution cannot completely eliminate the A-pillar blind spot.

[0004] 2. Blind Spot Monitoring System: Many modern cars are equipped with blind spot monitoring systems, which use sensors such as radar or cameras to detect objects in the A-pillar blind spot and warn the driver through visual or audible alerts. However, current technology mostly involves mounting a display screen on the instrument panel, requiring the driver to shift their gaze to the central control screen to confirm the information, which still poses a certain driving risk.

[0005] 3. Augmented Reality Technology: Some automakers are developing augmented reality technologies, such as using sensors to detect the A-pillar area and projecting the information onto the windshield using an AR-HUD to alert the driver to potential objects in the A-pillar blind spot. However, this solution relies on overlaying information onto the existing windshield area, which may interfere with the normal windshield field of vision and pose a certain driving risk.

[0006] 4. Panoramic Camera: Many cars are equipped with a 360-degree panoramic camera system, which provides a panoramic view, displaying information about the vehicle's surroundings as a specific image on the central control panel. This helps the driver avoid collisions with objects in blind spots when changing lanes or reversing. However, this system requires the driver to shift their attention to the central control display screen, which can pose a certain driving risk. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, this invention proposes an in-vehicle auxiliary display method, display system, display device, and readable storage medium, which facilitates drivers in seeing visual information such as the road and pedestrians behind the A-pillar, avoids unnecessary visual jumps, eliminates the blind spot of the A-pillar, and improves driving safety.

[0008] Specifically, the present invention proposes an in-vehicle auxiliary display method, comprising the following steps:

[0009] S1, obtain the first image information, which is the original image of the blind spot of the A-pillar captured by the external camera;

[0010] S2 captures the driver's binocular position in real time and defines the center of the binoculars as the origin of the driver's line of sight coordinate system;

[0011] S3, based on the vehicle external camera coordinate system, the first image information is transformed to the driver's line of sight coordinate system to obtain the second image information that simulates the visibility through the blind spot of the A-pillar;

[0012] S4, Project the outer contour of the curved screen of the A-pillar onto the driver's line of sight coordinate system, and crop the second image information based on the outer contour of the curved screen to obtain the third image information;

[0013] S5, perform parallax compensation on the third image information to obtain the fourth image information;

[0014] S6, the fourth image information is displayed on the curved screen of the A-pillar in real time.

[0015] According to an embodiment of the present invention, in step S1, the original image is preprocessed, and the preprocessing includes at least image denoising, image enhancement, white balance and contrast adjustment.

[0016] According to one embodiment of the present invention, in step S2, the Eyenet algorithm is used to track the changes in the driver's head posture in real time and update the origin of the driver's line of sight coordinate system.

[0017] According to an embodiment of the present invention, in step S3, based on the pose information of the origin of the line-of-sight coordinate system in the carrier coordinate system and the installation position of the external camera in the carrier coordinate system, a rotation and translation transformation matrix from the external camera coordinate system to the driver's line-of-sight coordinate system is obtained. This rotation and translation transformation matrix is ​​then used to transform the first image information into second image information in the driver's line-of-sight coordinate system. The calculation formula is as follows:

[0018]

[0019] Where P eye,image T represents the image pixel position in the driver's line-of-sight coordinate system. eye,cam_in It is the driver's binocular posture captured by the in-car camera. It's the installation location of the in-car camera, T car,cam_out This is the installation location for the exterior camera. It refers to the pixel position of the image captured by the vehicle's external camera.

[0020] According to an embodiment of the present invention, in step S4, based on the pose information of the origin of the viewpoint coordinate system in the carrier coordinate system and the installation pose of the curved screen in the carrier coordinate system, a rotation and translation transformation matrix from the curved screen coordinate system to the driver's viewpoint coordinate system is obtained. This rotation and translation transformation matrix is ​​used to project the outer contour of the curved screen onto the driver's viewpoint coordinate system. Based on the outer contour of the curved screen, the second image information is cropped to obtain the third image information. The calculation formula is as follows:

[0021] S display,eye =T eye,cam_in T display S display,car ;

[0022] Where S display,eye T is the projection of the outer contour of the curved screen in the driver's line-of-sight coordinate system. display S represents the installation pose of the curved screen in the carrier coordinate system. display,car This refers to the point string information of the outer contour of the curved screen.

[0023] According to an embodiment of the present invention, in step S5, disparity compensation is performed on the third image information using a linear fitting method to obtain the fourth image information, and the corresponding compensation calculation formula is as follows:

[0024] x' = a*(x-cx)*f;

[0025] y' = c*(y-cy)*f;

[0026] Where a and c are the magnification coefficients obtained from linear fitting, cx and cy are the corrected offsets, f is the focal length of the camera, and x' and y' are the optimized image pixels.

[0027] The present invention also provides an in-vehicle auxiliary display system, applicable to the aforementioned in-vehicle auxiliary display method. The in-vehicle auxiliary display system includes an exterior camera, an interior camera, an A-pillar covering a curved screen, and a controller. The controller includes:

[0028] The acquisition module is used to obtain the first image information, which is the original image of the A-pillar blind spot captured by the external camera; the acquisition module also obtains the driver's binocular position captured by the internal camera.

[0029] The first calculation module calculates the origin of the driver's line-of-sight coordinate system based on the binocular position;

[0030] The second calculation module transforms the first image information into the driver's line-of-sight coordinate system based on the vehicle's external camera coordinate system to obtain second image information that simulates visibility through the blind spot of the A-pillar.

[0031] The cropping module projects the outer contour of the curved screen on the A-pillar onto the driver's line-of-sight coordinate system, and crops the second image information based on the outer contour of the curved screen to obtain the third image information.

[0032] The compensation module performs parallax compensation on the third image information to obtain the fourth image information;

[0033] The execution module is used to send the fourth image information to the A-pillar curved screen so that the fourth image information is displayed on the A-pillar curved screen in real time.

[0034] According to one embodiment of the present invention, the in-vehicle camera is mounted on the sunroof switch panel or the rearview mirror.

[0035] According to one embodiment of the present invention, the curved screen is a flexible liquid crystal screen, which covers the blind spot of the A-pillar interior of the vehicle.

[0036] The present invention also provides an in-vehicle auxiliary display device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the aforementioned in-vehicle auxiliary display methods.

[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the preceding vehicle-mounted auxiliary display methods.

[0038] This invention provides an in-vehicle auxiliary display method, display system, display device, and readable storage medium. It captures images of the A-pillar blind spot using an external camera, displays the captured external images in real-time on a curved screen in the driver's blind spot, and tracks the driver's binocular positions using an in-vehicle camera. The images are then processed using a parallax compensation algorithm to optimize the visual information captured by the external camera and displayed on the curved screen. This information is then stitched together with the windshield's field of vision, allowing the driver to see road and pedestrian information behind the A-pillar, avoiding unnecessary visual jumps, thus eliminating the A-pillar blind spot and improving driving safety.

[0039] It should be understood that the above general description and the following detailed description of the invention are exemplary and illustrative, and are intended to provide further explanation of the invention as described in the claims. Attached Figure Description

[0040] The accompanying drawings are included to provide further explanation of the invention. They are incorporated into and constitute a part of this application. The drawings illustrate embodiments of the invention and, together with this specification, serve to explain the principles of the invention.

[0041] In the attached image:

[0042] Figure 1 A flowchart of an embodiment of the in-vehicle auxiliary display method of the present invention is shown.

[0043] Figure 2 A schematic diagram of the structure of an in-vehicle auxiliary display system according to an embodiment of the present invention is shown.

[0044] Figure 3 A schematic diagram of the controller according to an embodiment of the present invention is shown.

[0045] Figure 4 The diagram illustrates the effect of using an in-vehicle auxiliary display method according to an embodiment of the present invention. Detailed Implementation

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0049] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0050] In the description of this application, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is usually based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0051] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.

[0052] Figure 1 A flowchart of an embodiment of the in-vehicle auxiliary display method of the present invention is shown. As shown in the figure, the present invention provides an in-vehicle auxiliary display method, including the following steps:

[0053] S1, Obtain first image information, which is the original image of the A-pillar blind spot captured by the exterior camera. The exterior cameras on the left and right sides of the vehicle are mounted on the housings of the exterior rearview mirrors, facing the A-pillar blind spot area, and are used to capture image information in the A-pillar blind spot.

[0054] S2 captures the driver's binocular position in real time, defines the center of binoculars as the origin of the driver's line of sight coordinate system, and determines the driver's line of sight coordinate system.

[0055] S3, based on the coordinate system of the external camera, transforms the first image information to the driver's line-of-sight coordinate system to obtain a second image information that simulates visibility through the blind spot of the A-pillar. The first image information is converted into the second image information through coordinate transformation.

[0056] S4. Project the outer contour of the curved screen of the A-pillar onto the driver's line of sight coordinate system, and crop the second image information based on the outer contour of the curved screen to obtain the third image information.

[0057] S5, because the surface shape of the curved screen will vary depending on the actual design, it will also cause some deviation in the driver's line of sight. Therefore, it is necessary to perform parallax compensation on the third image information to obtain the fourth image information.

[0058] S6 displays the fourth image information in real time on the curved screen on the A-pillar. This allows the driver to easily and intuitively observe the road conditions in the blind spot of the vehicle's A-pillar, avoiding unnecessary visual jumps, thus eliminating the blind spot and improving driving safety.

[0059] Preferably, in step S1, the original image is preprocessed, and the preprocessing steps include at least image denoising, image enhancement, white balance, and contrast adjustment.

[0060] Preferably, in step S2, the Eyenet algorithm is used to track the driver's head pose changes in real time and update the origin of the driver's gaze coordinate system. An in-vehicle camera is used to capture the driver's eye position in real time. To achieve the effect that the image displayed on the A-pillar curved screen is consistent with the real image seen by the human eye through the A-pillar, the image information captured by the external camera needs to be transformed and displayed on the curved screen. This requires real-time capture of the driver's gaze information to transform the image information in the external camera coordinate system to the driver's gaze coordinate system. Since the driver's head pose changes according to the driver's height or seat adjustment, and the same driver's gaze also changes as needed during driving, it is necessary to obtain the driver's gaze pose information in real time. In addition to obtaining the pose information of the in-vehicle camera through calibration, a facial detection algorithm is also needed to obtain head and even eye movements. The detection of gaze position and direction is an active research area, and some excellent deep learning-based solutions have emerged in recent years. In this invention, existing algorithms are used to complete the field of view detection and tracking. Algorithms that can be used include the Gaze360 model proposed by Microsoft Research Asia in 2021. This is a multi-task deep neural network model that can simultaneously perform face recognition and gaze direction estimation. This model utilizes image enhancement techniques to increase the amount of data, enabling it to more accurately estimate head pose and eye movements. The Gaze360 model can also improve its accuracy and stability by adding more data, thus exhibiting excellent performance in both accuracy and real-time performance. Besides the Gaze360 model, there are other technical solutions, such as Pupil Labs' Pupil Invisible, Google's AutoML Vision Edge, and GazeSense. These solutions can be selected based on the specific application requirements. After obtaining the position and direction of the gaze, using an appropriate tracking algorithm can improve accuracy and stability. Because the driver's head and body are constantly moving during driving, a tracking algorithm is needed to track the driver's head position and pose, and update the position and direction of the gaze in real time to maintain accuracy and stability. Tracking algorithms can be based on traditional computer vision techniques, such as Kalman filters, particle filters, segmentation, and matching, or on deep learning techniques, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and attention mechanisms. The preferred algorithm in this invention is EyeNet, a real-time eye position and gaze tracking algorithm based on deep neural networks, capable of achieving high-precision gaze tracking in various scenarios.

[0061] Preferably, in step S3, based on the pose information of the origin of the line-of-sight coordinate system in the carrier coordinate system (pose includes position and orientation; in a three-dimensional coordinate system, pose information is a set of 6-dimensional information), and the installation position of the external camera in the carrier coordinate system, a rotation and translation transformation matrix from the external camera coordinate system to the driver's line-of-sight coordinate system is obtained. This rotation and translation transformation matrix is ​​then used to transform the first image information into the second image information in the driver's line-of-sight coordinate system. The calculation formula is as follows:

[0062]

[0063] Where P eye,image T represents the image pixel position in the driver's line-of-sight coordinate system. eye,cam_in It is the driver's binocular posture captured by the in-car camera. It's the installation location of the in-car camera, T car,cam_out This is the installation location for the exterior camera. It refers to the pixel position of the image captured by the vehicle's external camera.

[0064] Preferably, in step S4, based on the pose information of the origin of the viewpoint coordinate system in the carrier coordinate system and the installation pose of the curved screen in the carrier coordinate system, a rotation and translation transformation matrix from the curved screen coordinate system to the driver's viewpoint coordinate system is obtained. This rotation and translation transformation matrix is ​​then used to project the outer contour of the curved screen onto the driver's viewpoint coordinate system, thereby matching the image displayed on the curved screen with the image information obtained normally through the windshield and front door glass. The second image information is then cropped based on the outer contour of the curved screen to obtain the third image information. The projection calculation formula is as follows:

[0065] s display,eye =T eye,cam_in T display S display,car ;

[0066] Where s display,eye T is the projection of the outer contour of the curved screen in the driver's line-of-sight coordinate system. display For the installation pose of the curved screen in the carrier coordinate system, s display,car This refers to the point string information of the outer contour of the curved screen.

[0067] Preferably, in step S5, a linear fitting method is used to perform disparity compensation on the third image information. This invention selects a relatively simple compensation method: by calibrating and testing the changes of multiple sets of marker points, the visual difference and scaling factor are calculated through data fitting, and then the pixel positions are optimized accordingly to obtain the fourth image information. The corresponding compensation calculation formula is as follows:

[0068] x' = a*(x-cx)*f;

[0069] y' = c*(y-cy)*f;

[0070] Where a and c are the magnification coefficients obtained from linear fitting, cx and cy are the corrected offsets, f is the focal length of the camera, x' and y' are the image pixel coordinates of the optimized fourth image information, and x and y are the image pixel coordinates of the third image information.

[0071] Figure 2 A schematic diagram of the structure of an in-vehicle auxiliary display system according to an embodiment of the present invention is shown. Figure 3 A schematic diagram of the controller according to an embodiment of the present invention is shown. As shown, the present invention also provides an in-vehicle auxiliary display system 100, applicable to the aforementioned in-vehicle auxiliary display method. The in-vehicle auxiliary display system 100 includes an external camera 101, an internal camera 102, an A-pillar 103 covering a curved screen, and a controller 104. (Reference) Figure 2 The blind spots on both sides of the vehicle's A-pillars are shown in the diagram. The dashed lines represent the field of view of the external cameras, while the solid lines represent the driver's field of view. Referring to the diagram, controller 104 includes:

[0072] The acquisition module 1041 is used to acquire first image information, which is the original image in the blind spot of the A-pillar 103 captured by the external camera 101; the acquisition module also acquires the driver's binocular position captured by the internal camera 102.

[0073] The first calculation module 1042 calculates the origin of the driver's line-of-sight coordinate system based on the binocular position;

[0074] The second calculation module 1043 converts the first image information to the driver's line-of-sight coordinate system based on the coordinate system of the external camera 101 to obtain second image information that simulates the visibility of the blind spot through the A-pillar 103.

[0075] The cropping module 1044 is used to project the outer contour of the curved screen of the A-pillar 103 onto the driver's line-of-sight coordinate system, and crop the second image information based on the outer contour of the curved screen to obtain the third image information.

[0076] The compensation module 1045 performs parallax compensation on the third image information to obtain the fourth image information;

[0077] The execution module 1046 is used to send the fourth image information to the curved screen of the A-pillar 103 so that the fourth image information is displayed on the curved screen of the A-pillar 103 in real time.

[0078] It should be noted that the controller 104 can be located in the trunk.

[0079] Preferably, the in-vehicle camera 102 is mounted on the sunroof control panel or the interior rearview mirror. Mounting it on the sunroof control panel is preferred because this position is relatively fixed, and the mounting position of the in-vehicle camera 102 can be relatively easily determined by referring to the outer casing of the exterior rearview mirror. If mounted on a movable component of the interior rearview mirror, a real-time camera calibration algorithm is required to obtain the in-cabin camera's mounting pose information. Compared to mounting on a fixed component, the calibration algorithm used for a movable component requires more computational resources to achieve the same pose accuracy.

[0080] Preferably, the curved screen is a flexible LCD screen, covering the blind spot area of ​​the A-pillar 103 in the vehicle's interior. Compared to rigid LCD screens, flexible LCD screens offer better flexibility. They can be bent and customized to fit the curvature and shape of the A-pillar 103, better conforming to its shape and reducing blind spots caused by its curvature and shape. Flexible LCD screens also offer a wider field of view, as they can be integrated with the surface of the A-pillar 103, providing a larger viewing area compared to traditional screens. Installing the flexible LCD screen on the A-pillar 103 reduces driver obstruction and distraction compared to mounting it on the dashboard, minimizing distractions and improving driver safety. Furthermore, the flexible LCD screen can be customized to the shape and color of the A-pillar 103, more easily integrating into the vehicle's interior design and enhancing the overall aesthetics and comfort. Figure 4 The diagram illustrates the effect of using an in-vehicle auxiliary display method according to an embodiment of the present invention. The driver can intuitively observe the road conditions in the blind spot of the vehicle's A-pillar 103 through the flexible liquid crystal screen. As liquid crystal display technology matures, future flexible liquid crystal screens will be able to cover the entire A-pillar 103, completely eliminating the blind spot.

[0081] The present invention also provides an in-vehicle auxiliary display device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the aforementioned in-vehicle auxiliary display methods.

[0082] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the aforementioned vehicle-mounted auxiliary display methods.

[0083] The specific implementation methods and technical effects of the vehicle auxiliary display system, display device, and computer-readable storage medium can be found in the embodiments of the vehicle auxiliary display method provided by the present invention, and will not be repeated here.

[0084] This invention provides an in-vehicle auxiliary display method, display system, display device, and readable storage medium, which combines a flexible screen and image processing technology to achieve real-time monitoring and compensation of the driver's A-pillar blind spot. This avoids the A-pillar blind spot obstructing the driver's view of road conditions, improving driving safety and driving experience. Compared with existing technologies, this patent application has the following advantages:

[0085] 1. Compared to ordinary screens, flexible screen technology is introduced and installed in the A-pillar. With greater plasticity and flexibility, the flexible screen can bend to a certain extent, thus better adapting to the curvature of the A-pillar, reducing the gap between the screen and the A-pillar, and improving the field of vision. It also avoids the problem of traditional external displays occupying interior space, affecting the vehicle's aesthetics and operating space. Flexible screens are also thinner and lighter, reducing the size of the A-pillar blind spot. Furthermore, flexible screens typically have lower power consumption, use more durable materials, and can maintain high reliability and performance for a longer period.

[0086] 2. By combining in-vehicle camera technology to capture the driver's eye position in real time, the system can dynamically adjust the displayed image, effectively reducing visual fatigue and discomfort caused by changes in viewing angle. At the same time, the system's functions are not affected by the driver's posture or head movement.

[0087] 3. Taking into account the installation location and surface curvature of the flexible screen as well as the installation location of the camera, a distortion correction algorithm is used to compensate for parallax in the image, which can ensure the image quality and accuracy seen by the driver, achieving an effect similar to a transparent A-pillar.

[0088] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0089] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0090] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0091] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0092] It will be apparent to those skilled in the art that various modifications and variations can be made to the exemplary embodiments described above without departing from the spirit and scope of the invention. Therefore, it is intended that this invention cover modifications and variations falling within the scope of the appended claims and their equivalents.

Claims

1. A vehicle-mounted auxiliary display method, comprising the following steps: S1, obtain the first image information, which is the original image of the blind spot of the A-pillar captured by the external camera; S2 captures the driver's binocular position in real time and defines the center of the binoculars as the origin of the driver's line of sight coordinate system; S3, based on the vehicle external camera coordinate system, the first image information is transformed to the driver's line of sight coordinate system to obtain the second image information that simulates the visibility through the blind spot of the A-pillar; S4, Project the outer contour of the curved screen of the A-pillar onto the driver's line of sight coordinate system, and crop the second image information based on the outer contour of the curved screen to obtain the third image information; S5, perform parallax compensation on the third image information to obtain the fourth image information; S6, the fourth image information is displayed in real time on the curved screen of the A-pillar; In step S3, based on the pose information of the origin of the line-of-sight coordinate system in the carrier coordinate system and the installation position of the external camera in the carrier coordinate system, the rotation and translation transformation matrix from the external camera coordinate system to the driver's line-of-sight coordinate system is obtained. This rotation and translation transformation matrix is ​​then used to transform the first image information into second image information in the driver's line-of-sight coordinate system. The calculation formula is as follows: ； The image pixel position in the driver's line-of-sight coordinate system. It is the driver's binocular posture captured by the in-car camera. This is the installation location for the in-car camera. This is the installation location for the exterior camera. It refers to the pixel position of the image captured by the vehicle's external camera; In step S4, based on the pose information of the origin of the viewpoint coordinate system in the carrier coordinate system and the installation pose of the curved screen in the carrier coordinate system, the rotation and translation transformation matrix from the curved screen coordinate system to the driver's viewpoint coordinate system is obtained. This rotation and translation transformation matrix is ​​used to project the outer contour of the curved screen onto the driver's viewpoint coordinate system. The second image information is then cropped based on the outer contour of the curved screen to obtain the third image information. The calculation formula is as follows: ； This is the projection of the outer contour of the curved screen onto the driver's line-of-sight coordinate system. This represents the installation pose of the curved screen in the carrier coordinate system. This refers to the string of points representing the outer contour of the curved screen. In step S5, disparity compensation is performed on the third image information using a linear fitting method to obtain the fourth image information. The corresponding compensation calculation formula is as follows: ； ； in , The magnification factor is obtained from linear fitting. , This is the corrected offset. It's the camera's focal length. , These are the optimized image pixels.

2. The vehicle-mounted auxiliary display method as described in claim 1, characterized in that, In step S1, the original image is preprocessed, and the preprocessing includes at least image denoising, image enhancement, white balance, and contrast adjustment.

3. The in-vehicle auxiliary display method according to claim 1, characterized by, In step S2, the Eyenet algorithm is used to track the changes in the driver's head posture in real time and update the origin of the driver's line of sight coordinate system.

4. A vehicle auxiliary display system, characterized by, The in-vehicle auxiliary display method as described in any one of claims 1-3, wherein the in-vehicle auxiliary display system includes an exterior camera, an interior camera, an A-pillar covering a curved screen, and a controller, the controller comprising: The acquisition module is used to obtain the first image information, which is the original image of the A-pillar blind spot captured by the external camera; the acquisition module also obtains the driver's binocular position captured by the internal camera. The first calculation module calculates the origin of the driver's line-of-sight coordinate system based on the binocular position; The second calculation module transforms the first image information into the driver's line-of-sight coordinate system based on the vehicle external camera coordinate system to obtain second image information that simulates the visibility through the blind spot of the A-pillar. The cropping module is used to project the outer contour of the curved screen of the A-pillar onto the driver's line of sight coordinate system, and crop the second image information based on the outer contour of the curved screen to obtain the third image information. The compensation module performs parallax compensation on the third image information to obtain the fourth image information; The execution module is used to send the fourth image information to the A-pillar curved screen so that the fourth image information is displayed on the A-pillar curved screen in real time.

5. The vehicle auxiliary display system of claim 4, wherein, The in-vehicle camera is installed on the sunroof switch panel or the rearview mirror.

6. The vehicle auxiliary display system of claim 4, wherein, The curved screen is a flexible LCD screen that covers the blind spot area of ​​the A-pillar interior.

7. A vehicle-mounted auxiliary display device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the in-vehicle auxiliary display method as described in any one of claims 1-3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the vehicle auxiliary display method as described in any one of claims 1-3.

Images