A method for eliminating the blind area of ​​the AB column of a vehicle

Abstract

The present invention relates to the field of automobile safety, and specifically discloses a method for eliminating the blind area of ​​the AB pillar of a vehicle. The present invention collects external images of the AB pillar of the vehicle, facial contour information and head posture information of the driver, determines the driver's line of sight direction through the facial contour information and head posture information of the driver, and then determines the feature vector of the external image of the AB pillar of the vehicle according to the driver's line of sight. The collected external image of the AB pillar of the vehicle is processed according to the feature vector to obtain the AB pillar blind area image corresponding to the driver's line of sight, and finally projects the AB pillar blind area image on the AB pillar of the vehicle to fill the AB pillar blind area field of vision that the driver cannot directly obtain, and helps the driver obtain more information about the surrounding area of ​​the vehicle, so as to make accurate driving operations.

Description

Technical Field

[0001] The invention belongs to the technical field of automobile safety, and in particular relates to a method for eliminating the blind area of ​​the AB column of a vehicle. Background Art

[0002] The A-pillar of a car is an important component supporting the front windshield, but due to its structure and position, it often forms a blind spot when turning, changing lanes or approaching an intersection, making it impossible for the driver to clearly see pedestrians, other vehicles or obstacles in the blind spot. The B-pillar of a car is an important component supporting the roof in the middle of the car body, and is located between the front and rear window glass. When reversing or turning, the B-pillar of the car will block part of the driver's field of vision, forming a B-pillar blind spot, which makes it impossible for the driver to obtain the visual information inside the blind spot, and unable to make timely driving operations based on the visual information, which in turn leads to traffic accidents.

[0003] Some existing technical solutions reduce the impact of blind spots by improving the materials or structures of the A-pillar and the B-pillar, such as using transparent materials, reducing the volume of the A-pillar and the B-pillar, etc. However, these methods of changing the material or structure of the vehicle often not only cause the problem of unclear vision, but also lead to a reduction in the structural strength of the vehicle body, and cannot effectively ensure the safety of the people in the cabin. Therefore, it is necessary to design a solution that can help the driver obtain the field of vision of the A-pillar and B-pillar blind spots without reducing the structural strength of the vehicle body. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a method for eliminating the blind spot of the AB pillar of a vehicle, so as to solve the problem that the method of replacing the AB pillar with a transparent material to obtain the blind spot field of view not only reduces the strength of the vehicle body structure, but also the obtained blind spot field of view is not clear enough.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for eliminating the blind area of ​​the AB column of a vehicle, comprising:

[0007] Collect the driver's facial profile information and head posture information to determine the driver's line of sight direction;

[0008] Real-time collection of external images of the vehicle's AB pillars;

[0009] Determine the feature vector of the external image of the AB pillar of the vehicle according to the direction of the driver's sight;

[0010] Processing the external image of the AB pillar of the vehicle according to the feature vector to determine the blind spot image in the direction of the driver's line of sight;

[0011] The blind spot image is displayed on the inner surface of the AB pillar of the vehicle.

[0012] Preferably, determining the driver's sight direction includes:

[0013] Use OpenCV, Dilb or a deep learning framework to determine the driver's facial contour information, including eye information, nose information and mouth information, wherein the eye information includes the driver's eye boundary information and inner and outer eye corner position information;

[0014] Based on the eye information, nose information and mouth information, the pitch angle, yaw angle and roll angle of the driver's head are calculated to determine the driver's head posture;

[0015] Use Hough transform or threshold segmentation method to process facial contour information, obtain pupil position, and obtain sight direction based on eye information and pupil position information;

[0016] The relative spatial position of the eyes is determined according to the driver's head posture information, the line of sight direction of the eyes is adjusted based on the pitch angle, yaw angle and roll angle of the head, and the three-dimensional direction vector of the line of sight direction is determined.

[0017] Preferably, the external images of the vehicle's AB pillars are recorded by two cameras disposed on both sides of the A pillar and two cameras disposed on both sides of the B pillar respectively.

[0018] Preferably, the initial vehicle AB pillar external images recorded by the camera are preprocessed, and the preprocessing includes:

[0019] Use mean filtering, Gaussian filtering or median filtering to smooth the image for denoising;

[0020] Convert color video frames to grayscale images;

[0021] The grayscale images of the video images corresponding to the two cameras on both sides of the A-pillar and the B-pillar are cropped, and the images of the A-pillar and the B-pillar in the video image are removed to obtain two video frame images containing the blind spot image of the A-pillar and two video frame images containing the blind spot of the B-pillar.

[0022] Preferably, image registration is performed on the two video frame images containing the A-pillar blind spot image and the two video frame images containing the B-pillar blind spot, respectively. The image registration methods include an image registration method based on grayscale information, an image registration method based on a transform domain, and an image registration method based on features. Image fusion is performed on the registered video frame images. The image fusion methods include an average method, a maximum method, and a fade-in and fade-out method. A video frame image containing the A-pillar blind spot image and a video frame image containing the B-pillar blind spot are respectively obtained, and then a screen containing the A-pillar blind spot and a video containing the B-pillar blind spot are obtained.

[0023] Preferably, a screen for displaying the blind area including the A-pillar and a screen for displaying the blind area including the B-pillar are installed on the inner surface of the AB pillar of the vehicle.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention collects external images of the vehicle's AB pillars, the driver's facial contour information and head posture information, determines the driver's line of sight direction through the driver's facial contour information and head posture information, and then determines the feature vector of the vehicle's AB pillar external images according to the driver's line of sight direction, processes the collected vehicle's AB pillar external images according to the feature vector to obtain an AB pillar blind spot image corresponding to the driver's line of sight, and finally projects the AB pillar blind spot image onto the vehicle's AB pillar to fill the AB pillar blind spot field of vision that the driver cannot directly obtain, thereby helping the driver obtain more vehicle surrounding information and further making accurate driving operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A block diagram of the method of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Example:

[0029] See also Figure 1 As shown, a method for eliminating the blind area of ​​the AB pillar of a vehicle comprises:

[0030] Collect the driver's facial profile information and head posture information to determine the driver's line of sight direction;

[0031] Real-time collection of external images of the vehicle's AB pillars;

[0032] Determine the feature vector of the external image of the AB pillar of the vehicle according to the direction of the driver's sight;

[0033] Processing the external image of the AB pillar of the vehicle according to the feature vector to determine the blind spot image in the direction of the driver's line of sight;

[0034] The blind spot image is displayed on the inner surface of the AB pillar of the vehicle.

[0035] From the above, it can be seen that by collecting the external images of the vehicle's AB pillars, the driver's facial contour information and head posture information, the driver's line of sight direction is determined through the driver's facial contour information and head posture information, and then the feature vector of the vehicle's AB pillar external images is determined according to the driver's line of sight direction, and the collected vehicle AB pillar external images are processed according to the feature vector to obtain the AB pillar blind spot image corresponding to the driver's line of sight, and finally the AB pillar blind spot image is projected onto the vehicle's AB pillar to fill the AB pillar blind spot field of vision that the driver cannot directly obtain, and help the driver obtain more information about the vehicle's surroundings, and then make accurate driving operations.

[0036] Determining the driver's line of sight includes:

[0037] 1. Use OpenCV, Dilb or deep learning framework to determine the driver's facial contour information, including eye information, nose information and mouth information. The eye information includes the driver's eye boundary information and inner and outer eye corner position information. OpenCV, Dilb or deep learning framework are relatively mature facial key point detection libraries. In addition to using ready-made detection libraries, you can also independently design related algorithms to obtain the driver's facial information;

[0038] 2. Based on the eye information, nose information and mouth information, the pitch angle, yaw angle and roll angle of the driver's head are calculated to determine the driver's head posture. The pitch angle indicates the up and down rotation of the head, the yaw angle indicates the left and right rotation of the head, and the roll angle indicates the rotation of the head around its own axis. The driver's head posture can be accurately judged through these three angles. The angle judgment can be inferred by using image analysis algorithms combined with the spatial positions of the eyes, nose and chin. In addition, a depth camera or a stereo camera can be used to directly calculate the three-dimensional spatial position of the head, and then the head posture angle is estimated. Considering the cost of use, the former method is used, that is, using an algorithm to judge the driver's head posture angle;

[0039] 3. Use Hough transform or threshold segmentation method to process facial contour information to obtain the approximate position of the eyes, obtain pupil position information by processing the eye position information, and obtain the sight direction based on the eye information and pupil position information. In addition, an eye movement model can be established. After determining the static position of the eyes, an eye rotation movement model is established. The eye rotation matrix is ​​used to describe the changes in the eye sight. In the specific use process, relevant personnel can select a suitable algorithm to determine the driver's sight direction as needed;

[0040] 4. Determine the relative spatial position of the eyes based on the driver's head posture information, adjust the line of sight direction based on the pitch, yaw and roll angles of the head, and determine the three-dimensional direction vector of the line of sight, which represents the angle at which the driver's eyes acquire the image, as a basis for subsequent adjustments to the external images of the vehicle's AB pillars.

[0041] The external images of the vehicle's AB pillars are recorded by two cameras respectively arranged on both sides of the A pillar and two cameras respectively arranged on both sides of the B pillar, that is, two cameras are installed on both sides of the A pillar, and two cameras are installed on both sides of the B pillar. The cameras on both sides of the A pillar or the B pillar can obtain the blind spot field of view behind the A pillar or the B pillar, and the images obtained by each group of two cameras will have repeated parts, which need to be further processed. In order to reduce the processing amount of the recorded image, the camera can be installed as close to the driver's head as possible, such as installed at the head of the front seat, but it is necessary to avoid the driver's head blocking the camera.

[0042] Preprocessing the initial vehicle AB column external images recorded by the camera, the preprocessing includes:

[0043] 1. Use mean filtering, Gaussian filtering or median filtering to smooth the image for denoising, eliminate image noise, deformity and distortion, and prepare for subsequent image registration;

[0044] 2. Converting color video frames into grayscale images can effectively reduce the amount of image processing, speed up image processing, and reduce the sense of separation between the image displayed on the AB column and the real picture;

[0045] 3. Crop the grayscale images of the video images corresponding to the two cameras on both sides of the A-pillar and the B-pillar, remove the images of the A-pillar and the B-pillar in the video image, further reduce the processing amount, and only process the blind spot field of view blocked by the AB pillars and the connected partial field of view, so as to obtain two video frame images containing the A-pillar blind spot image and two video frame images containing the B-pillar blind spot.

[0046] Image registration is performed on the two video frame images containing the A-pillar blind spot image and the two video frame images containing the B-pillar blind spot respectively. The image registration methods include image registration methods based on grayscale information, image registration methods based on transform domain and image registration methods based on features. The image registration method based on grayscale information directly uses some statistical information of the grayscale of the image itself to represent the points that reach the maximum or minimum value between images, so as to determine the transformation relationship between the images. The most common image registration method based on transform domain is Fourier transform. The feature-based image registration method uses the unchanged internal features in the image for registration. The registration speed is fast, and the user can choose the registration method independently according to the actual situation. In the process of acquiring images, due to the existence of lighting, field of view, and angle in the image acquisition environment, If the acquired images are simply superimposed, there will be very obvious stitching gaps in the acquired stitched images. Therefore, the video frame images after registration are fused so that the final image obtained by stitching the registered images has no obvious stitching gaps. Image fusion methods include the average method, the maximum method and the fade-in and fade-out method, which are common processing methods within the field. Users can choose one to use according to their needs, and obtain a video frame image containing the A-pillar blind spot image and a video frame image containing the B-pillar blind spot, and then obtain a screen containing the A-pillar blind spot and a video containing the B-pillar blind spot, and then display the screen containing the A-pillar blind spot and the video containing the B-pillar blind spot on the A-pillar and the B-pillar respectively, so that the video is integrated with the real environment to help the driver judge the driving environment.

[0047] A screen for displaying the blind area of ​​the A-pillar and a screen for displaying the blind area of ​​the B-pillar are installed on the inner surfaces of the AB pillars of the vehicle. Compared with the image projection method, the images displayed on the screens are clearer, and the screen opening time can be specially set. When driving in a straight line at high speed, the screen can be turned off. In addition, some special conditions for turning on the screen are set. For example, when the car starts, turns, reverses or parks, the screen can be turned on to obtain the surrounding field of view of the vehicle. On the one hand, it can avoid the driver's misjudgment caused by long-term flickering of the screen, and on the other hand, it can also reduce the usage time of devices such as screens and cameras and extend their service life.

[0048] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0049] In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other.

Claims

1. A method for eliminating the blind area of ​​the AB pillar of a vehicle, characterized in that: include: Collect the driver's facial profile information and head posture information to determine the driver's line of sight direction; Real-time collection of external images of the vehicle's AB pillars; Determine the feature vector of the external image of the AB pillar of the vehicle according to the direction of the driver's sight; Processing the external image of the AB pillar of the vehicle according to the feature vector to determine the blind spot image in the direction of the driver's line of sight; The blind spot image is displayed on the inner surface of the AB pillar of the vehicle.

2. The method for eliminating the blind spot of the AB pillar of a vehicle according to claim 1, characterized in that: Determining the driver's line of sight includes: Use OpenCV, Dilb or a deep learning framework to determine the driver's facial contour information, including eye information, nose information and mouth information, wherein the eye information includes the driver's eye boundary information and inner and outer eye corner position information; Based on the eye information, nose information and mouth information, the pitch angle, yaw angle and roll angle of the driver's head are calculated to determine the driver's head posture; Use Hough transform or threshold segmentation method to process facial contour information, obtain pupil position, and obtain sight direction based on eye information and pupil position information; The relative spatial position of the eyes is determined according to the driver's head posture information, the line of sight direction of the eyes is adjusted based on the pitch angle, yaw angle and roll angle of the head, and the three-dimensional direction vector of the line of sight direction is determined.

3. The method for eliminating the blind spot of the AB pillar of a vehicle according to claim 1, characterized in that: The external images of the vehicle's AB pillars are recorded by two cameras disposed on both sides of the A pillar and two cameras disposed on both sides of the B pillar respectively.

4. The method for eliminating the blind spot of the AB pillar of a vehicle according to claim 3, characterized in that: Preprocessing the initial vehicle AB column external images recorded by the camera, the preprocessing includes: Use mean filtering, Gaussian filtering or median filtering to smooth the image for denoising; Convert color video frames to grayscale images; The grayscale images of the video images corresponding to the two cameras on both sides of the A-pillar and the B-pillar are cropped, and the images of the A-pillar and the B-pillar in the video image are removed to obtain two video frame images containing the blind spot image of the A-pillar and two video frame images containing the blind spot of the B-pillar.

5. The method for eliminating the blind area of ​​the AB pillar of a vehicle according to claim 4, characterized in that: Image registration is performed on the two video frame images containing the A-pillar blind spot image and the two video frame images containing the B-pillar blind spot respectively, and the image registration methods include an image registration method based on grayscale information, an image registration method based on a transform domain, and an image registration method based on features. Image fusion is performed on the registered video frame images, and the image fusion methods include an average method, a maximum method, and a fade-in and fade-out method. A video frame image containing the A-pillar blind spot image and a video frame image containing the B-pillar blind spot are respectively obtained, and then a screen containing the A-pillar blind spot and a video containing the B-pillar blind spot are obtained.

6. The method for eliminating the blind area of ​​the AB pillar of a vehicle according to claim 5, characterized in that: A screen for displaying the blind area of ​​the A-pillar and a screen for displaying the blind area of ​​the B-pillar are installed on the inner surface of the AB pillar of the vehicle.

Images