Image processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses an image processing method, device and equipment and a storage medium, and the method comprises the steps: obtaining a first image shot by at least one main camera and a second image shot by at least one auxiliary camera, the main camera comprises fisheye cameras located in the front, back, left and right directions of a vehicle, and the auxiliary camera comprises fisheye cameras located in the front, back, left and right directions of the vehicle; the auxiliary camera comprises charge-coupled device telescopic cameras located on the right front portion, the left front portion, the right rear portion and the left rear portion of the vehicle. Carrying out distortion correction on the first image and the second image to respectively obtain a third image and a fourth image; and fusing the third image and the fourth image to obtain a final panoramic looking-around image. According to the technical scheme, the visual blind area of the vehicle surround view system can be reduced, and the collision rate of the vehicle is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of automobile panoramic surround view camera, and in particular to an image processing method, device, equipment and computer-readable storage medium. Background Art

[0002] Around View Monitor (AVM) is part of the automatic parking system. It is a highly practical function that can greatly improve user experience and intelligent driving safety. In related technologies, a panoramic image around the vehicle body is mainly obtained by stitching four-way surround fisheye lens images; however, this method has limited effect on reducing blind spots, thus affecting user experience and driving safety.

[0003] Application Contents

[0004] In view of the above problems, the present application provides an image processing method for solving the visual blind spot problem at the four corners of a car in a car surround view system existing in the related art.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, the method comprising: acquiring a first image taken by at least one main camera, and a second image taken by at least one auxiliary camera, wherein the main camera comprises a fisheye camera located in four directions of front, rear, left, right and left of a vehicle, and the auxiliary camera comprises a charge-coupled device retractable camera located in the right front, left front, right rear and left rear of the vehicle; performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; and fusing the third image and the fourth image to obtain a final panoramic surround view image.

[0006] In an optional manner, the auxiliary camera includes a CDC retractable camera located below the right front headlight, below the left front headlight, below the right rear brake light, and below the left rear brake light of the vehicle.

[0007] In an optional manner, the performing distortion correction on the first image and the second image to obtain dedistorted third and fourth images respectively includes: determining the normalized distance of each camera based on the distortion table and focal length corresponding to the main camera and the auxiliary camera; performing polynomial fitting on each camera based on the light incident angle and normalized distance corresponding to each camera to obtain the distortion correction coefficient of the main camera and the auxiliary camera; performing distortion correction on the first image and the second image based on the distortion correction coefficient to obtain coordinates corresponding to the third image and the fourth image.

[0008] In an optional manner, performing polynomial fitting on each camera based on the light incident angle and normalized distance corresponding to each camera to obtain the distortion correction coefficient of the main camera and the auxiliary camera includes: acquiring a data sequence to be processed, the data sequence including a sequence sequence composed of the light incident angles and a sequence composed of the normalized distances, the sequence composed of the light incident angles is an independent variable sequence, and the sequence composed of the normalized distances is a dependent variable sequence; substituting the independent variable sequence and the dependent variable sequence into a preset ninth-order polynomial, fitting the coefficients of the ninth-order polynomial using a least squares method, and obtaining the distortion correction coefficient of the corresponding camera, wherein the coefficients of the even-order ninth-order polynomial are zero.

[0009] In an optional manner, fusing the third image and the fourth image to obtain a final panoramic surround image includes: obtaining an intrinsic parameter matrix and an extrinsic parameter matrix corresponding to a main camera and an auxiliary camera, respectively; calculating a projection transformation matrix corresponding to each camera based on the intrinsic parameter matrix and the extrinsic parameter matrix; and projecting the third image and the fourth image onto a bird's-eye view based on the projection transformation matrix to obtain the panoramic surround image.

[0010] In an optional manner, before calculating the projection transformation matrix of each camera based on the intrinsic parameter matrix and the extrinsic parameter matrix, the method includes: obtaining the projection of the translation vector of the main camera and the auxiliary camera in the unit vector direction; calculating the normalized device coordinates of each camera based on the projection of the translation vector of each camera in the unit vector direction and the distance from the corresponding camera to the projection plane; and determining the extrinsic parameter matrix of each camera based on the sum of the normalized device coordinates of each camera and the rotation matrix.

[0011] In an optional manner, before acquiring a first image taken by at least one main camera and a second image taken by at least one auxiliary camera, the method includes: acquiring driving data of the user, the driving data including a steering wheel angle; if an absolute value of the steering wheel angle is greater than a first preset threshold value and greater than zero, determining that the cameras located at the left front and the right front are auxiliary cameras; if the absolute value of the steering wheel angle is greater than a first preset threshold value and less than zero, determining that the camera located at the left front is an auxiliary camera.

[0012] In an optional manner, before acquiring a first image taken by at least one main camera and a second image taken by at least one auxiliary camera, the method includes: acquiring driving data of the user, the driving data including a steering wheel angle; if the absolute value of the steering wheel angle is greater than a second preset threshold and greater than zero, determining that the auxiliary camera located at the right rear is the auxiliary camera; if the absolute value of the steering wheel angle is greater than the second preset threshold and less than zero, determining that the camera located at the left rear is the auxiliary camera.

[0013] According to another aspect of an embodiment of the present application, a vehicle assisted driving method is provided, the method comprising: if it is detected that the detection distance of the ultrasonic radar and the distance from the obstacle is less than a preset threshold and the vehicle speed is less than a preset threshold, issuing a request to raise the camera; and executing the above-mentioned image processing method in response to the request to raise the camera.

[0014] According to another aspect of an embodiment of the present application, a 360-degree surround view system for a vehicle is provided, comprising: at least four main cameras for capturing a first image, the main cameras comprising fisheye cameras located in four directions of the front, rear, left, right and left of the vehicle; at least four auxiliary cameras for capturing a second image, the auxiliary cameras comprising CDC retractable cameras located in the right front, left front, right rear and left rear of the vehicle; a processor for performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; and fusing the third image and the fourth image to obtain a final panoramic surround view image.

[0015] According to another aspect of an embodiment of the present application, an image processing device is provided, the device comprising: an acquisition module, acquiring a first image taken by at least one main camera, and a second image taken by at least one auxiliary camera, wherein the main camera comprises a fisheye camera located in four directions of front, rear, left, right and left of a vehicle, and the auxiliary camera comprises a charge-coupled device retractable camera located in the right front, left front, right rear and left rear of the vehicle; a distortion correction module, performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; and a fusion module, fusing the third image and the fourth image to obtain a final panoramic surround view image.

[0016] According to another aspect of an embodiment of the present application, there is provided an image processing device, including:

[0017] Controller;

[0018] The memory stores one or more programs, and when the one or more programs are executed by the controller, the controller implements the above-mentioned image processing method.

[0019] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes at least one executable instruction. When the executable instruction is executed on an image processing apparatus / device, the image processing apparatus / device performs the operation of the above-mentioned image processing method.

[0020] In order to solve the risks caused by the visual blind spots that may exist in the right front, left front, right rear and left rear of the vehicle, the embodiment of the present application adds at least four auxiliary cameras, namely CDC retractable cameras, at the above four positions, and performs distortion conversion on the first image and the second image taken by the main camera and the auxiliary camera, and then fuses the images after distortion conversion to obtain the final panoramic surround image. In this way, the blind spots around the vehicle are greatly reduced, and the risk of collision during vehicle driving is reduced.

[0021] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0023] Figure 1 is an exemplary structural schematic diagram of a vehicle blind spot;

[0024] Figure 2 It is an exemplary structural diagram of the installation position of the vehicle camera of the present application;

[0025] Figure 3 A flowchart of a first embodiment of an image processing method of the present application is shown;

[0026] Figure 4 is a structural diagram of a communication bus link shown in an exemplary embodiment;

[0027] Figure 5 Shows Figure 3 A flowchart of an embodiment of step S120;

[0028] Figure 6 is a structural schematic diagram of a camera distortion model shown in an exemplary embodiment;

[0029] Figure 7 Shows Figure 3 A flowchart of an embodiment of step S130;

[0030] Figure 8 It is a structural schematic diagram of a projection transformation model shown in an exemplary embodiment;

[0031] Fig. 9 A schematic diagram showing the structure of an embodiment of the image processing device of the present application is shown;

[0032] Fig.10 A schematic structural diagram of an embodiment of the image processing device of the present application is shown. DETAILED DESCRIPTION

[0033] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0036] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0037] In automotive technology, 360-degree panoramic imaging technology has become an important means to improve driving safety. The 360-degree panoramic imaging technology is to install a surround-view camera in the middle of the front bumper, the middle of the rear bumper, under the left rearview mirror, and under the right rearview mirror. For example, an external wide-angle fisheye camera collects image data around the vehicle, and through virtual reality technology and computer processing, renders a panoramic image of the vehicle starting from the roof perspective, and can manually trigger the perspective of each lens individually, which effectively reduces the driver's blind spot and improves driving safety.

[0038] Although the 360-degree surround view system can significantly reduce the visual blind spots during driving, it cannot completely eliminate all blind spots. Since the surround view image is composed of four images provided by four wide-angle cameras, there are still blind spots in the four corners of the car. For example, Figure 1FIG. 1 is a schematic diagram of an exemplary structure of a vehicle blind spot, such as Figure 1 As shown in the figure, there are blind spots at the four corners of the vehicle. In addition, the area above the camera position (such as the roof, engine hood, etc.) is also a blind spot that cannot be photographed.

[0039] Limited by the position of the camera, when the vehicle is on an extremely narrow road, affected by the field of vision, if the vehicle is leaving the warehouse at this moment, or the vehicle is reversing into the warehouse, if it is not an experienced driver, it is impossible to accurately judge the distance from the right front (such as the right front headlight) or the right rear (such as the right rear brake light), the left front (such as the left front headlight), and the left rear (such as the left rear brake light) to other vehicles, which poses a great risk of scratches. In open roads at low speeds, most of the scratches come from these positions. These problems limit the further development of the application of existing technologies in this field, and a new solution is urgently needed to solve these problems.

[0040] like Figure 2 As shown, Figure 2 This is an exemplary structural diagram of the installation position of the vehicle camera of the present application. It can be seen that if at least one camera is set on the left front, right front, left rear and right rear of the vehicle, in conjunction with the vehicle's original surround-view camera, all cameras can cover the entire perimeter of the vehicle and there are no blind spots.

[0041] Figure 3 A flow chart of a first embodiment of an image processing method of the present application is shown, and the method is executed by a vehicle body controller. Figure 4 is a structural diagram of a communication bus link shown in an exemplary embodiment, such as Figure 4 As shown in the figure, the vehicle control (also known as the body controller) is connected to the T-box and the gateway through a bus, and the T-box and the gateway are connected to the ADAS domain through a bus. The vehicle's multiple cameras, including the auxiliary camera and the main camera, obtain images around the vehicle through image acquisition, so that the body controller obtains the images and then processes the multiple image information. Figure 3 , the method comprises the following steps:

[0042] Step S110: Acquire a first image taken by at least one main camera and a second image taken by at least one auxiliary camera.

[0043] Among them, the main camera includes fisheye cameras located in the front, back, left and right directions of the vehicle, and the auxiliary camera includes retractable wide-angle CDC cameras located in the right front, left front, right rear and left rear.

[0044] Among them, in step S110), the installation positions of the four main fisheye cameras are respectively the outermost ends of the left and right rearview mirrors of the car, the center of the intersection of the car hood and the air intake grille, and the center of the lower edge of the rear license plate.

[0045] The main cameras of some embodiments of the present application include a front-view camera, a rear-view camera, a left-view camera, and a right-view camera; the front-view camera is tilted downward by 5-15 degrees relative to the lateral direction of the vehicle body; the rear-view camera is tilted downward by 20-35 degrees relative to the longitudinal direction of the vehicle body; the left-view camera and the right-view camera are tilted backward by 5 degrees relative to the lateral direction of the vehicle body, and tilted outward by 45 degrees relative to the longitudinal direction of the vehicle body.

[0046] In some embodiments of the present application, in order to solve the risks caused by the visual blind spots that may exist in the right front (below the right front headlight), left front (below the left front headlight), right rear (below the right rear brake light), and left rear (below the left rear brake light) of the vehicle, 4 auxiliary cameras are added at the above four positions, such as retractable wide-angle CCD cameras. This camera is small in size, light in weight, low in cost, not affected by magnetic fields, and has good anti-vibration and anti-impact performance. The camera image pixel is about 500,000, supports color scene recognition, and automatically adjusts the imaging according to the lighting conditions to adapt to most application scenarios.

[0047] In some embodiments of the present application, after the installation positions of each camera are fixed, the four fisheye cameras are calibrated respectively. Exemplarily, a two-stage calibration method is adopted. First, the fisheye camera is calibrated by using a spherical perspective projection method to obtain a distortion correction coefficient. The fisheye image after distortion correction is then calibrated again by using the Zhang Zhengyou calibration method to obtain the internal and external parameters of the fisheye camera.

[0048] In some embodiments of the present application, the installation positions of the four auxiliary cameras can be determined by calibration. For example, the visual blind spots near the four corners of the vehicle are counted, and the setting positions of the corresponding auxiliary cameras are calibrated based on the positions of the visual blind spots.

[0049] Step S120: performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively.

[0050] Due to its special lens design, fisheye cameras often produce images with significant distortion, which can cause straight lines in the image to become curved and circles to become ellipses, seriously affecting the visual effect and quality of the image. This application can restore these distorted images to a more real-world appearance through distortion correction, thereby improving the overall image quality.

[0051] Generally, the methods for correcting the distortion of images taken by fisheye cameras mainly include polynomial distortion model, chessboard calibration method, lateral expansion method and longitude and latitude method. In general, there are many methods for correcting the distortion of images taken by fisheye cameras, and each method has its own unique advantages and limitations. When choosing a specific method, it is necessary to weigh and choose according to the actual application scenario and needs.

[0052] Step S130: Fusing the third image and the fourth image to obtain a final panoramic image.

[0053] In some embodiments of the present application, if there are multiple third images and multiple fourth images, all the third images and fourth images are fused.

[0054] 360-degree surround view imaging technology is an advanced vehicle-assisted driving technology that captures images around the vehicle by installing multiple cameras around the car and stitches these images into a seamless panoramic view through image processing algorithms.

[0055] In some embodiments of the present application, a variety of methods can be used to fuse all third images and fourth images, including but not limited to image fusion based on deep learning, such as the yolov3 network model, that is, a target detection and image segmentation model based on deep learning. This method is conducive to evaluating the quality of stitched images of 360-degree panoramic surround view devices. For example, image fusion based on image registration, SURF registration algorithm and SIFT algorithm; and image fusion based on perspective transformation. Usually, since the car surround view camera is usually installed at an oblique downward angle, the image output by the original camera is not a top view. In order to generate a bird's-eye view, it is necessary to perform a perspective transformation on the image and project the image to a new top-down perspective; and based on weighted image fusion, in the process of image stitching, for the overlapping area, the weight value can be calculated according to the distance. The closer to the center, the higher the weight, so that the image is superimposed according to the weight value to reduce the ghosting phenomenon.

[0056] In some embodiments of the present application, in order to solve the risks caused by the visual blind spots that may exist in the right front, left front, right rear and left rear of the vehicle, at least four auxiliary cameras, namely CDC retractable cameras, are added at the above four positions, and the first image and the second image taken by the main camera and the auxiliary camera are distorted and transformed, and then the distorted images are fused to obtain the final panoramic surround image. In this way, the blind spots around the vehicle are greatly reduced, and the risk of collision during vehicle driving is reduced.

[0057] Figure 5 Shows Figure 3 For a flowchart of an embodiment of step S120 in FIG. Figure 2 As shown, step S120 includes steps S210 to S230:

[0058] S210: Determine the normalized distance of each camera based on the distortion table and focal length corresponding to the main camera and the auxiliary camera.

[0059] In some embodiments of the present application, the normalized distance is determined using the distortion table and focal length corresponding to each camera.

[0060] The distortion table is obtained from the camera manufacturer. The table provides the distance between the actual imaging point and the imaging center corresponding to different incident angles, while the focal length of the camera is the actual physical focal length of the camera, which is used to convert the actual distance to the distance relative to the imaging plane. Figure 6 As shown, Figure 6 is a schematic structural diagram of a camera distortion model shown in an exemplary embodiment. Figure 6 As shown, f is the focal length of the camera, PO C is the incident light, θ is the incident angle of the light, and Od is the distance from the actual imaging point corresponding to the incident angle of the light to the imaging center.

[0061] The normalized distance is calculated based on the distortion table and the camera focal length. That is, the actual distance in the distortion table is divided by the focal length to obtain the normalized distance. This step is to eliminate the influence of the actual physical size and make the distortion data under different cameras or different settings comparable.

[0062] S220: Performing polynomial fitting on each camera based on the light incident angle and normalized distance corresponding to each camera to obtain distortion correction coefficients of the main camera and the auxiliary camera.

[0063] In some embodiments of the present application, polynomial fitting is performed based on the normalized distance and the incident angle of the light. The polynomial coefficients obtained by fitting can be used to describe the distortion model and used to correct the distortion in subsequent image processing. It is convenient to apply the polynomial coefficients obtained by fitting as distortion correction coefficients in the image processing algorithm to correct the distorted image to generate a distortion-free panoramic image.

[0064] Optionally, step S220 may include:

[0065] 1. Obtain the data sequence to be processed, the data sequence includes a sequence composed of light incident angles and a sequence composed of normalized distances, the sequence composed of light incident angles is the independent variable sequence, and the sequence composed of normalized distances is the dependent variable sequence.

[0066] 2. Substitute the independent variable sequence and the dependent variable sequence into the ninth-order polynomial, use the least squares method to fit the coefficients of the polynomial, and obtain the distortion correction coefficients of the corresponding camera, where the coefficients of the even-order ninth-order polynomial are zero.

[0067] In practice, the fitting is based on the following polynomial:

[0068] r(θ)=k0(θ)+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9

[0069] Among them, r(θ) is the distance between the outgoing light ray in the normalized plane of the camera and the origin, θ is the incident angle of the light ray, and k0, k1, k2, k3, and k4 are all distortion correction coefficients to be fitted. In some embodiments of the present application, the normalized plane refers to a concept introduced to facilitate mathematical processing, so that the actual physical distance is converted into a relative value during the calculation process, thereby simplifying the calculation process of distortion correction. Through polynomial fitting, the distortion model can be accurately described and the distortion in the image can be effectively corrected. The above optional embodiment uses a ninth-order odd-order polynomial for data fitting, which can effectively describe the nonlinear characteristics of the distortion and has a better fitting effect.

[0070] S230: Perform distortion correction on the first image and the second image respectively based on the distortion correction coefficient to obtain coordinates corresponding to the third image and the fourth image.

[0071] In some embodiments of the present application, based on the following formula

[0072] x=x d r / r(θ)

[0073] y=y d r / r(θ)

[0074] r = tan(θ)

[0075] Among them, (x, y) is the undistorted pixel coordinate, (xd, yd) is the distorted coordinate, r(θ) is the distance between the outgoing light ray in the camera normalized plane and the origin, and θ is the incident angle of the light ray.

[0076] In some embodiments of the present application, Figure 7 As shown, Figure 3 Shows Figure 1 For a flowchart of an embodiment of step S130, please refer to Figure 7 As shown, step S130 includes steps S310 to S330:

[0077] S310: Obtain the intrinsic parameter matrix and the extrinsic parameter matrix corresponding to the main camera and the auxiliary camera respectively.

[0078] Since the car surround view camera is usually installed at an oblique downward angle, the image output by the original camera is not a top view. Therefore, in order to achieve the effect of a bird's-eye view, the image output by the camera must be projected to a new top-down perspective.

[0079] In some embodiments of the present application, the intrinsic parameter matrix K of the camera is determined, including parameters such as focal length, camera center, etc. These parameters describe the geometric characteristics inside the camera and are crucial for projecting a three-dimensional point onto a two-dimensional image plane.

[0080] According to the camera's posture (rotation matrix R) and position (translation vector t), we can get the camera's external parameter matrix [R|t]. This matrix describes the transformation relationship between the camera coordinate system and the world coordinate system.

[0081] The camera's rotation matrix and translation vector are key parameters that describe the transformation relationship between the camera coordinate system and the world coordinate system.

[0082] S320: Calculate the projection transformation matrix of each camera based on the intrinsic parameter matrix and the extrinsic parameter matrix.

[0083] Exemplarily, the camera projection matrix is ​​obtained by multiplying the intrinsic parameter matrix and the extrinsic parameter matrix, and this projection matrix can be used to project points in the three-dimensional space onto the two-dimensional image plane.

[0084] Figure 8 FIG. 1 is a schematic diagram of a structure of a projection transformation model shown in an exemplary embodiment. Figure 8 As shown, two cameras shoot the X plane at two different positions A and B with different postures, generating two images m and m'. In this embodiment, it is necessary to perform a projection transformation between the two images, where d is the distance from the shooting position to the X plane, and n is the translation vector of the corresponding camera.

[0085] Exemplarily, in order to obtain the extrinsic parameter matrix corresponding to each camera, first obtain the projection of the camera's translation vector in the unit vector direction; then, the normalized device coordinates are calculated based on the projection of the translation vector in the unit vector direction and the distance from the corresponding camera to the projection plane; finally, the extrinsic parameter matrix of the corresponding camera is determined based on the sum of the normalized device coordinates and the camera's rotation matrix.

[0086] Exemplarily, the projection transformation matrix H between images is obtained based on the following formula:

[0087]

[0088] Among them, K is the camera intrinsic parameter matrix, R is the rotation matrix, T is the translation vector, d is the distance from the corresponding camera to the projection plane, and N is the unit matrix.

[0089] S330: Projecting the third image and the fourth image onto the bird's-eye view based on the projection transformation matrix to obtain a panoramic surround image.

[0090] The image processing method provided in this embodiment is applicable to multiple application scenarios and achieves multiple beneficial effects. For example, it can improve the safety of driving on narrow roads. When a vehicle is on an extremely narrow road and when the vehicle is turning, the vehicle can trigger the image processing method provided in the embodiment of this application, and can promptly report the risk of scratching the left front / left rear headlight and the right / left rear brake light caused by the turn. This solution can effectively avoid the problem in the prior art that the driver cannot be aware of the risks caused by turning in a timely manner, thereby reducing the occurrence of scratching accidents and improving the safety of driving on narrow roads.

[0091] In an exemplary embodiment, a vehicle assisted driving method is provided. The method is performed by a vehicle body controller and includes the following steps:

[0092] S410: If it is detected that the detection distance of the ultrasonic radar and the distance from the obstacle is less than a preset threshold and the vehicle speed is less than a preset threshold, a request to raise the camera is issued.

[0093] S420: Execute the image processing method provided in the above embodiment in response to the request to raise the camera.

[0094] In some embodiments of the present application, when the vehicle enters an extremely narrow road, the ultrasonic radar USS, an off-board sensing element, emits ultrasonic waves, detects echoes, and detects target objects in real time through time difference ranging, and outputs the distance to the object being measured. Before the triggering process, the body control unit BCM continuously sends an Available signal to the ADAS domain. When the ADAS domain receives the detection distance of the sensing element ultrasonic radar and the object being measured <= 30cm, and the vehicle speed ICSpeed ​​<= 10KM / H (both events are met at the same time), the ADAS domain controller sends a request to BCM to raise the camera, and performs fusion image output through the previous example one. The body control unit BCM executes and replies to the camera raising event status. When the ultrasonic radar USS detects the target object at a distance lower than the 10cm threshold (which can be changed), the ADAS domain controller can send an audible and visual alarm request to the cockpit domain CDC. The cockpit domain CDC then executes, such as Figure 4 As shown;

[0095] When the vehicle leaves an extremely narrow road, the ultrasonic radar USS, an external sensing element, emits ultrasonic waves, detects echoes, and detects target objects in real time through time difference distance measurement, and outputs the distance to the object being measured. Before the triggering process, the body control unit BCM continuously sends an Available signal to the ADAS domain. When the ADAS domain receives that the detection distance of the sensing element ultrasonic radar and the object being measured is >30cm, and the vehicle speed ICSpeed>10KM / H, the ADAS domain controller sends a request to BCM to retract the camera. The body control unit BCM executes and responds to the camera retraction event status. Figure 5 As shown;

[0096] When the ADAS domain receives the vehicle speed information ICSpeed>25KM / H, the ADAS domain requests the cockpit domain CDC to exit the panoramic imaging mode.

[0097] The driver can manually select whether the assisted driving panoramic image should automatically rise or fall on narrow roads in the cockpit large screen CDC, or call up the panoramic image in the cockpit and select the corresponding viewing angle, such as the left front, left rear, right front, right rear, front, rear, left and right viewing angles of the vehicle body. The vehicle-side T-Box can support remote interconnection with mobile phones, and can also be viewed when the vehicle is not in Ready state to monitor in real time whether there are potential risks in the outside view of the vehicle.

[0098] In a specific embodiment of the present application, in order to make the vehicle's 360-degree surround view system more energy-efficient and power-saving, when the present application detects that the vehicle enters a curve, at least one main camera is raised, and the image captured by the main camera is used to identify whether the vehicle is entering a curve to the left or to the right. If the vehicle is entering a curve to the left, there is a higher possibility that an obstacle will appear on the left side of the curve. In this case, the left front auxiliary camera is raised; conversely, if the vehicle is entering a curve to the right, there is a higher possibility that an obstacle will appear on the right side of the curve. In this case, the right front auxiliary camera is raised.

[0099] In another specific embodiment of the present application, in order to make the vehicle's 360-degree surround view system more energy-efficient and power-saving, during the process of detecting vehicle parking, if the vehicle is detected to be in a reversing state and the steering wheel is rotated to the right, there may be an obstacle in the lane near the left brake light of the vehicle, and therefore, the auxiliary camera under the left brake light is raised at this time; conversely, if the vehicle is detected to be in a reversing state and the steering wheel is rotated to the left, there may be an obstacle in the lane near the right brake light of the vehicle, and therefore, the auxiliary camera under the right brake light is raised at this time; if the vehicle is detected to be in a state of exiting a parking space and the steering wheel is rotated to the left, there may be an obstacle in the lane near the left front headlight of the vehicle, and therefore, the auxiliary camera under the left front headlight is raised at this time; if the vehicle is detected to be in a state of exiting a parking space and the steering wheel is rotated to the right, there may be an obstacle in the lane near the right front headlight of the vehicle, and therefore, the auxiliary camera under the right front headlight is raised at this time. This embodiment forms a 360-degree surround-view camera system by adding a main camera and an auxiliary camera. In different states of vehicle parking or driving, the auxiliary camera in the corresponding direction is raised. In this way, energy saving and power saving can be achieved.

[0100] The present application also provides a 360-degree surround view system for a vehicle, comprising at least four main cameras for capturing a first image, the main cameras comprising fisheye cameras located in four directions of the front, rear, left, right and sides of the vehicle; at least four auxiliary cameras for capturing a second image, the auxiliary cameras comprising CDC retractable cameras located in the right front, left front, right rear and left rear of the vehicle; a processor for performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; and fusing the third image and the fourth image to obtain a final panoramic surround view image.

[0101] In some embodiments of the present application, the possible blind spots of the vehicle are determined based on the road condition information of the vehicle, and the auxiliary cameras at the corresponding positions are turned on based on the positions of the blind spots, so as to achieve the purpose of improving driving safety and saving energy and power. Specifically, the inventors of the present application have found through statistics that several areas are prone to become blind spots when the vehicle turns, and these blind spots may increase the risk of accidents. The following are the main blind spots:

[0102] Front lateral blind spot:

[0103] When the vehicle turns right, the right front area near the sidewalk is likely to become a blind spot. This area is usually near the vehicle's A-pillar (that is, the pillar between the windshield and the cab), and the line of sight is easily blocked. When the vehicle turns left, the left front area near the central isolation strip of the road or the opposite lane is also likely to become a blind spot.

[0104] Therefore, in some embodiments of the present application, the auxiliary camera that needs to work is determined by the following steps:

[0105] 1. Obtain the user's driving data, which includes the steering wheel angle.

[0106] In this embodiment, the size and positive and negative sign of the steering wheel angle can determine whether the road condition on which the vehicle is located is a curve. If the vehicle is on a curve and is turning, the blind spot of the vehicle at this time will change accordingly. Therefore, this embodiment utilizes the working condition of the auxiliary camera set at the corresponding position of the blind spot of the vehicle when turning to achieve energy-saving effect.

[0107] 2. If the absolute value of the steering wheel angle is greater than a first preset threshold and greater than zero, the cameras located at the left front and the right front are determined to be auxiliary cameras.

[0108] In this embodiment, if the absolute value of the steering wheel angle is greater than the first preset threshold and greater than zero, it means that the vehicle is turning right. When the vehicle turns right, the area near the sidewalk in the front right side is likely to become a blind spot. This area is usually near the A-pillar of the vehicle (that is, the pillar between the windshield and the cab), and the line of sight is easily blocked. Therefore, the cameras located in the left front and right front are determined as auxiliary cameras. In this case, only the cameras in the left front and right front of the vehicle take the second picture, while the cameras in the left rear and right rear of the vehicle do not work to achieve energy saving.

[0109] 3. If the absolute value of the steering wheel angle is greater than a first preset threshold and less than zero, it is determined that the camera located in the left front is an auxiliary camera.

[0110] In this embodiment, if the absolute value of the steering wheel angle is greater than the first preset threshold and less than zero, it means that the vehicle is turning left. When the vehicle turns left, the area in the left front near the central isolation strip of the road or the oncoming lane is also likely to become a blind spot, so the camera located in the left front is determined to be an auxiliary camera. In this case, only the camera in the left front of the vehicle takes the second picture, while the cameras in the left rear, right rear and right front of the vehicle do not work to achieve energy saving.

[0111] In some other embodiments of the present application, the auxiliary camera that needs to be operated is determined by the following steps:

[0112] 1. Obtain the user's driving data, which includes the steering wheel angle.

[0113] 2. If the absolute value of the steering wheel angle is greater than a second preset threshold and greater than zero, it is determined that the auxiliary camera located at the right rear is the auxiliary camera.

[0114] If the absolute value of the steering wheel angle is greater than the second preset threshold and greater than zero, it means that the vehicle is turning right at this time. The inventor has found through research that if the steering wheel angle is too large, there may be a blind spot in the rear outer blind spot. When turning right, the area behind the right side of the vehicle is likely to become a blind spot. This area is between the rearview mirror and the side mirror and is not easily visible to the driver. Therefore, the auxiliary camera located at the right rear is determined as the auxiliary camera, so that the auxiliary camera located at the right rear can shoot real-time video.

[0115] 3. If the absolute value of the steering wheel angle is greater than a second preset threshold and less than zero, it is determined that the camera located at the left rear is an auxiliary camera.

[0116] If the absolute value of the steering wheel angle is greater than the second preset threshold and less than zero, it means that the vehicle is turning left at this time. The inventor has found through research that if the vehicle turns left and the steering wheel angle is too large, there may be a blind spot in the rear outer blind spot. When turning left, the area near the central isolation belt of the road behind the vehicle is also likely to become a blind spot. Therefore, the camera located at the left rear is determined to be an auxiliary camera, so that the auxiliary camera located at the left rear can shoot real-time video. In the above two exemplary embodiments, the second preset threshold is greater than the first preset threshold.

[0117] Fig. 9 FIG. 1 is a schematic diagram showing the structure of an embodiment of the image processing device of the present application. Fig. 9 As shown, the device 500 includes: an acquisition module 510, a distortion correction module 520 and a fusion module 530.

[0118] Among them, the acquisition module 510 is used to acquire a first image taken by at least one main camera and a second image taken by at least one auxiliary camera, wherein the main camera includes fisheye cameras located in four directions of the front, back, left and right of the vehicle, and the auxiliary cameras include charge-coupled device retractable cameras located in the right front, left front, right rear and left rear of the vehicle; the distortion correction module 520 is used to perform distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; the fusion module 530 is used to fuse the third image and the fourth image to obtain a final panoramic surround view image.

[0119] In an optional manner, the auxiliary camera includes a CDC retractable camera located below the right front headlight, below the left front headlight, below the right rear brake light, and below the left rear brake light of the vehicle.

[0120] In an optional manner, the distortion correction module 520 includes a determination submodule, a polynomial fitting submodule and a distortion correction submodule, wherein the determination submodule is used to determine the normalized distance of the corresponding camera based on the distortion table and focal length corresponding to the main camera and the auxiliary camera; the polynomial fitting submodule is used to perform polynomial fitting based on the incident angle of light and the normalized distance to obtain the distortion correction coefficient of the corresponding camera; the distortion correction submodule is used to perform distortion correction based on the distortion coordinates and distortion correction coefficients of the first image and the second image to obtain the coordinates corresponding to the third image and the fourth image.

[0121] In an optional manner, the fusion module 530 includes an acquisition submodule, a calculation submodule and a projection submodule, wherein the acquisition submodule is used to acquire the intrinsic parameter matrix and the extrinsic parameter matrix corresponding to the main camera and the auxiliary camera respectively; the calculation submodule is used to calculate the projection transformation matrix of the corresponding camera based on the intrinsic parameter matrix and the extrinsic parameter matrix; the projection submodule is used to project the third image and the fourth image onto a bird's-eye view based on the projection transformation matrix to obtain a panoramic surround image.

[0122] In order to solve the risks that may be caused by the visual blind spots that may exist in the right front, left front, right rear and left rear of the vehicle, at least four auxiliary cameras, namely charge-coupled device retractable cameras, are added at the above four positions. The image processing device provided in this embodiment performs distortion conversion on the first image and the second image taken by the main camera and the auxiliary camera, and then fuses the images after distortion conversion to obtain the final panoramic surround image. In this way, the blind spots around the vehicle are greatly reduced, and the risk of collision during vehicle driving is reduced.

[0123] The image processing device provided in the above embodiment and the image processing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0124] Fig.10 A structural schematic diagram of an embodiment of the image processing device of the present application is shown, which shows a structural schematic diagram of a computer system suitable for implementing the image processing device of the embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the image processing device.

[0125] See also Fig.10 As shown, the image processing device includes: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the above-mentioned image processing method is executed.

[0126] Please continue reading Fig.10 As shown, the computer system 500 of the image processing device includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503, such as executing the method in the above embodiment. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502 and the RAM 503 are connected to each other through a bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0127] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0128] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the system of the present application are executed.

[0129] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the image processing method as described above is implemented. The computer-readable storage medium may be included in the image processing device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0130] Another aspect of the present application further provides a computer program product or a computer program, which includes at least one executable instruction. When the executable instruction runs on an image processing apparatus / device, the image processing apparatus / device executes the above image processing method.

[0131] The executable instructions may be specifically used to enable the image processing device / apparatus to perform the following operations:

[0132] Acquire a first image taken by at least one main camera and a second image taken by at least one auxiliary camera, wherein the main camera includes fisheye cameras located in four directions of the front, rear, left and right of the vehicle, and the auxiliary cameras include CDC retractable cameras located in the right front, left front, right rear and left rear of the vehicle; perform distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; fuse the third image and the fourth image to obtain a final panoramic surround view image.

[0133] The computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0134] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0136] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0137] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0138] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image captured by at least one main camera and a second image captured by at least one auxiliary camera, wherein the main camera comprises fisheye cameras located in four directions of front, rear, left, and right of the vehicle, and the auxiliary cameras comprise charge-coupled device retractable cameras located in front right, front left, right rear, and left rear of the vehicle; Performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; The third image and the fourth image are fused to obtain a final panoramic image.

2. The method according to claim 1, characterized in that The performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively comprises: Determine the normalized distance of each camera based on the distortion table and focal length corresponding to the main camera and the auxiliary camera; Performing polynomial fitting on each camera based on the light incident angle corresponding to each camera and the normalized distance, to obtain distortion correction coefficients of the main camera and the auxiliary camera; Distortion correction is performed on the first image and the second image based on the distortion correction coefficient to obtain coordinates corresponding to the third image and the fourth image.

3. The method according to claim 2, characterized in that The polynomial fitting is performed on each camera based on the incident angle of light corresponding to each camera and the normalized distance to obtain the distortion correction coefficients of the main camera and the auxiliary camera, including: Acquire a data sequence to be processed, wherein the data sequence includes a sequence composed of the light incident angles and a sequence composed of the normalized distances, wherein the sequence composed of the light incident angles is an independent variable sequence, and the sequence composed of the normalized distances is a dependent variable sequence; Substitute the independent variable sequence and the dependent variable sequence into a preset ninth-order polynomial, and use the least squares method to fit the coefficients of the ninth-order polynomial to obtain the distortion correction coefficients of the corresponding camera, wherein the coefficients of the even-order ninth-order polynomial are zero.

4. The method according to claim 1, characterized in that The fusing the third image and the fourth image to obtain a final panoramic image includes: Obtaining the intrinsic parameter matrix and the extrinsic parameter matrix corresponding to the main camera and the auxiliary camera respectively; The projection transformation matrix corresponding to each camera is calculated based on the intrinsic parameter matrix and the extrinsic parameter matrix; The third image and the fourth image are projected onto a bird's-eye view based on the projection transformation matrix to obtain the panoramic surround image.

5. The method according to claim 4, characterized in that Before calculating the projection transformation matrix of each camera based on the intrinsic parameter matrix and the extrinsic parameter matrix, the method includes: Obtaining projections of the translation vectors of the main camera and the auxiliary camera in the direction of the unit vector; The normalized device coordinates of each camera are obtained based on the projection of the translation vector of each camera in the unit vector direction and the distance from the corresponding camera to the projection plane; The extrinsic matrix of each camera is determined based on the sum of the normalized device coordinates of each camera and the rotation matrix.

6. The method according to claim 1, characterized in that Before acquiring a first image taken by at least one main camera and a second image taken by at least one auxiliary camera, the method includes: Acquiring driving data of a user, wherein the driving data includes a steering wheel angle; If the absolute value of the steering wheel angle is greater than a first preset threshold and greater than zero, determining that the cameras located at the left front and the right front are auxiliary cameras; If the absolute value of the steering wheel angle is greater than the first preset threshold and less than zero, it is determined that the camera located at the left front is an auxiliary camera.

7. The method according to claim 1, characterized in that Before acquiring a first image taken by at least one main camera and a second image taken by at least one auxiliary camera, the method includes: Acquiring driving data of a user, wherein the driving data includes a steering wheel angle; If the absolute value of the steering wheel angle is greater than a second preset threshold and greater than zero, determining that the auxiliary camera located at the right rear is an auxiliary camera; If the absolute value of the steering wheel angle is greater than the second preset threshold and less than zero, it is determined that the camera located at the left rear is an auxiliary camera.

8. An image processing device, characterized in that: The image processing device comprises: an acquisition module, for acquiring a first image captured by at least one main camera and a second image captured by at least one auxiliary camera, wherein the main camera comprises a fisheye camera located in four directions of the front, rear, left, and right of the vehicle, and the auxiliary camera comprises a charge-coupled device retractable camera located in the right front, left front, right rear, and left rear of the vehicle; a distortion correction module, performing distortion correction on the first image and the second image to obtain a third image and a fourth image respectively; The fusion module fuses the third image and the fourth image to obtain a final panoramic image.

9. An image processing device, characterized in that: include: Controller; A memory storing one or more programs, which, when executed by a controller, enables the controller to implement the image processing method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which includes at least one executable instruction. When the executable instruction is executed on the image processing apparatus / device, the image processing apparatus / device performs the operation of the image processing method according to any one of claims 1 to 7.

Images