Image processing method and device, electronic equipment and vehicle

Abstract

The application provides an image processing method and device, electronic equipment and vehicle. The method comprises: acquiring lane line pictures captured by multiple cameras in the vehicle at the same time, and performing bird's eye view conversion on the lane line pictures captured by the multiple cameras to obtain multiple lane line bird's eye view pictures; performing preprocessing and Hough transformation on each lane line bird's eye view picture respectively to obtain a lane fitting straight line corresponding to each lane line bird's eye view picture; sequentially calculating the offset of the two lane fitting straight lines corresponding to any two lane line bird's eye view pictures with a splicing area in the corresponding splicing area until the offsets of all splicing areas corresponding to the multiple lane line bird's eye view pictures are obtained; correcting the homography matrix previously calibrated for the vehicle according to the offsets of all splicing areas to obtain a corrected homography matrix; and splicing the pictures captured subsequently by the multiple cameras in the vehicle into a bird's eye view picture by using the corrected homography matrix. The application can solve the problem of lane line splicing mismatch existing in the BEV function.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image processing method, apparatus, electronic device, and vehicle. Background Technology

[0002] AVM (Around View Monitor), also known as a panoramic monitoring system, is widely used in fields such as driver assistance and autonomous driving. An AVM system consists of multiple fisheye cameras that stitch together images to create a panoramic overhead view, providing the driver with more information about the vehicle's surroundings and enabling functions such as assisted parking and automatic lane changing.

[0003] In the AVM system, the most frequently used function by the driver is BEV (Bird's Eye View), also known as the "God's View." However, due to calibration errors, the bird's-eye view generated by the BEV function has uneven lane line stitching, which greatly reduces the user's driving experience. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and vehicle to solve the problem of uneven lane line stitching in bird's-eye view images obtained by BEV function stitching.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] The system acquires lane line images taken by multiple cameras in the vehicle at the same time, and converts these images into bird's-eye view images to obtain multiple lane line bird's-eye view images.

[0007] Each lane line bird's-eye view is preprocessed and subjected to Hough transform to obtain the lane fitting line corresponding to each lane line bird's-eye view.

[0008] Calculate the offset of the two lane fitting lines corresponding to any two lane line bird's-eye views with splicing areas in the corresponding splicing areas, until the offset of all splicing areas corresponding to multiple lane line bird's-eye views is obtained.

[0009] The pre-calibrated homography matrix of the vehicle is corrected based on the offset of all splicing areas to obtain the corrected homography matrix;

[0010] Using the corrected homography matrix, bird's-eye view images were stitched together from multiple subsequent images taken by cameras in the vehicle.

[0011] Secondly, embodiments of this application provide an image processing apparatus, including:

[0012] The first acquisition module is used to acquire lane line images taken by multiple cameras in the vehicle at the same time, and to convert the lane line images taken by multiple cameras into bird's-eye view images to obtain multiple lane line bird's-eye view images.

[0013] The second acquisition module is used to preprocess and perform Hough transform on each lane line bird's-eye view to obtain the lane fitting line corresponding to each lane line bird's-eye view.

[0014] The calculation module is used to sequentially calculate the offset of the two lane fitting lines corresponding to any two lane line bird's-eye views with splicing areas in the corresponding splicing areas, until the offset of all splicing areas corresponding to multiple lane line bird's-eye views is obtained.

[0015] The correction module is used to correct the pre-calibrated homography matrix of the vehicle based on the offset of all splicing areas, and obtain the corrected homography matrix.

[0016] The stitching module is used to stitch together bird's-eye view images from multiple cameras in the vehicle using the corrected homography matrix.

[0017] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect above.

[0018] Fourthly, embodiments of this application provide a vehicle including the electronic equipment described in the third aspect.

[0019] This application provides an image processing method, apparatus, electronic device, and vehicle. It utilizes images with uneven lane line stitching issues for reverse calibration to correct the homography matrix. This corrected homography matrix allows for the stitching of bird's-eye view images subsequently captured by multiple cameras within the vehicle. Since the corrected homography matrix is ​​based on reverse calibration, calibration errors are eliminated. Therefore, the problem of uneven lane line stitching in the bird's-eye view image obtained by BEV (Battery on the Vehicle) function, caused by calibration errors, is solved, providing the driver with a better visual experience and significantly enhancing the user's driving experience. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating an example of uneven lane line splicing provided in this application embodiment;

[0022] Figure 2 This is a flowchart illustrating the implementation of an image processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a binarized version provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a polygon filtering process provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of an edge detection process provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram illustrating an embodiment of the present application for eliminating uneven lane line splicing;

[0027] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0032] As described in related technologies, in an AVM system, the most frequently used function by the driver is the BEV (Battery EV) function. This function calibrates the pose relationships between multiple fisheye cameras, then projects the images captured by these cameras onto a bird's-eye view, and finally stitches them together to obtain the final bird's-eye view. Because camera parameters and calibration tools often have calibration errors, the bird's-eye view obtained by the BEV function often suffers from uneven lane line stitching, for example... Figure 1The issue of misaligned lane lines is shown in the middle circle area. Figure 1 The black blocks in the image represent vehicles.

[0033] To address the problems of the prior art, embodiments of this application provide an image processing method, apparatus, electronic device, and vehicle. The image processing method provided in this application embodiment will be described first below.

[0034] The entity executing the image processing method can be an image processing device, such as a vehicle control unit (VCU), electronic control unit (ECU), or any electronic device capable of performing related processing of the image processing method. This application does not specifically limit it.

[0035] The applicant discovered that the principle behind using BEV (Battery Electric Vehicle) functionality to stitch together a bird's-eye view is as follows: first, a homography matrix is ​​obtained based on calibration data; then, the homography matrix is ​​used to stitch together images taken by various cameras to obtain the final bird's-eye view. Analysis shows that the key to solving the problems of existing technologies is to use an accurate homography matrix, and accurate calibration data is needed to obtain an accurate homography matrix. Therefore, the image processing method provided in this application adopts the following technical concept: using images with uneven lane line stitching for reverse calibration to eliminate calibration errors, thereby correcting the homography matrix.

[0036] See Figure 2 The document illustrates a flowchart of the image processing method provided in an embodiment of this application, which is described in detail below:

[0037] Step 210: Obtain lane line images taken by multiple cameras in the vehicle at the same time, and convert the lane line images taken by multiple cameras into bird's-eye view images to obtain multiple lane line bird's-eye view images.

[0038] In some embodiments, the vehicle can be any type of vehicle equipped with BEV functionality, such as traditional fuel vehicles like gasoline vehicles and diesel vehicles, or new energy vehicles such as EV (Electric Vehicle), HEV (Hybrid Electric Vehicle), and PHEV (Plug-in Hybrid Electric Vehicle).

[0039] In some embodiments, the lane line image can be an image including lane lines. Specifically, lane line images captured simultaneously by multiple cameras in the vehicle can be obtained in various ways. For example, lane line images captured simultaneously by multiple cameras can be retrieved from the vehicle's cache, or lane line images captured simultaneously by multiple cameras can be obtained in real time when lane lines are detected during vehicle operation.

[0040] Step 220: Perform preprocessing and Hough transform on each lane line bird's-eye view to obtain the lane fitting line corresponding to each lane line bird's-eye view.

[0041] In some embodiments, the lane fitting line is a straight line that reflects the actual position of the lane lines in the bird's-eye view. To obtain the lane fitting line, each lane line bird's-eye view can be preprocessed, including binarization, distortion removal, and quadrilateral filtering, and then Hough transform can be used to obtain the lane fitting line corresponding to each lane line bird's-eye view.

[0042] The following section uses a bird's-eye view of the target lane line as an example. This target lane line bird's-eye view refers to any one of the multiple lane line bird's-eye views obtained in step 210. The process for obtaining the lane fitting line is described below. First, the target lane line bird's-eye view is sequentially binarized, distortion-removing, and quadrilateral-removing processed to obtain the lane graphic corresponding to the target lane line bird's-eye view. Next, edge detection processing is performed on the lane graphic to generate a lane edge mask. Then, a Hough transform is performed on the lane edge mask to obtain the corresponding straight line coordinate data. Finally, straight line fitting is performed on the straight line coordinate data corresponding to the lane edge mask to obtain the lane fitting line corresponding to the target lane line bird's-eye view.

[0043] In some embodiments, a brightness threshold can be used to binarize the bird's-eye view of the target lane lines to separate the lane lines. For example... Figure 3 As shown, a binarized image is provided.

[0044] In some embodiments, considering that bird's-eye view images often contain some distorted areas, distortion removal processing can be performed after obtaining the binarized image to remove distorted areas from the image, thereby improving the accuracy of subsequent reverse calibration.

[0045] In some embodiments, considering that lane lines are typically quadrilaterals, after distortion removal processing, the image can be subjected to quadrilateral filtering to obtain lane graphics. Specifically, firstly, polygon detection is performed on the image to identify existing polygons. Then, polygon fitting is performed on the extra polygons to obtain polygons with a specific number of sides, such as quadrilaterals, pentagons, hexagons, etc. Next, quadrilaterals are filtered out from all the fitted polygons. Finally, a second filtering is performed on the filtered quadrilaterals using an area threshold to obtain lane graphics that meet the requirements. For the area threshold, considering that lane lines often have a large area, quadrilaterals that do not belong to lane lines can be removed using an area threshold. This area threshold can be an empirical value, such as 1000 square centimeters. Figure 4 As shown, a method for... Figure 3 Lane images obtained by polygon filtering.

[0046] In some embodiments, edge detection processing can be Sobel edge detection. For example... Figure 5 As shown, a method for... Figure 4 Lane edge mask obtained by edge detection processing of the middle lane image.

[0047] In some embodiments, after obtaining the lane edge mask, a Hough transform can be performed on the lane edge mask, that is, the image is transformed into the Hough domain on the lane edge mask, and then the straight line data is extracted, such as by using the HoughLinesP Hough transform function in OpenCV to obtain the straight line data of the coordinate pairs of the start and end points of the straight line.

[0048] In some embodiments, considering the possibility of interfering straight lines, they can be eliminated using certain thresholds. Specifically, lane width is typically 3-4 meters, vehicle width is 2 meters, and the scale relationship between the pixels in the bird's-eye view image and the actual length is 1:1, i.e., 1 pixel represents 1 cm, etc. Thresholds for lane line detection ranges can be set, such as 3 meters, and then data exceeding the threshold can be excluded. In this way, some interfering straight lines can be eliminated, thereby improving the accuracy of subsequent reverse calibration.

[0049] In some embodiments, the coordinates of the points on the detected multiple straight lines can be fitted with a straight line to fit a line that reflects the actual position of the lane lines in the bird's-eye view.

[0050] Step 230: Calculate the offset of the two lane fitting lines corresponding to any two lane line bird's-eye views with splicing areas in the corresponding splicing areas, until the offset of all splicing areas corresponding to multiple lane line bird's-eye views is obtained.

[0051] It should be noted that misaligned lane lines often occur in the stitching area, which is the overlapping area when two bird's-eye view images are stitched together. The stitching area is key to reverse calibration, as it provides the parameters that ensure neat lane line stitching: the offset of the fitted straight lines of the two lanes within the stitching area.

[0052] In some embodiments, the offset of the two lane fitting lines in the splicing area can be obtained through the intersection point of the lane fitting lines and the splicing seam of the splicing area, which is the boundary of the splicing area. Specifically, for ease of description, the first fitting line and the second fitting line are used below to refer to the two lane fitting lines corresponding to any two bird's-eye view images of lane lines with splicing areas. First, the first intersection point of the first fitting line and the splicing seam of the target splicing area, and the second intersection point of the second fitting line and the splicing seam of the target splicing area are obtained, where the target splicing area is the splicing area of ​​any two bird's-eye view images of lane lines with splicing areas. Then, the distance between the first intersection point and the second intersection point is calculated, and the calculated distance is determined as the offset of the target splicing area.

[0053] It should be noted that, based on the relevant parameters of the bird's-eye view, we can accurately determine the actual length represented by a pixel in the bird's-eye view in reality. In other words, the bird's-eye view can provide rich prior information related to scale. Therefore, the distance between the first intersection point and the second intersection point can accurately reflect the degree of unevenness in lane line splicing, thus providing accurate calibration data for subsequent correction of the homography matrix.

[0054] Step 240: Correct the pre-calibrated homography matrix of the vehicle based on the offset of all splicing areas to obtain the corrected homography matrix.

[0055] In some embodiments, the projection point coordinates of multiple cameras can be adjusted first according to a preset adjustment method to eliminate offsets in all stitching areas. Then, based on the adjusted projection point coordinates of the multiple cameras, the pre-calibrated homography matrix of the vehicle is corrected to obtain the corrected homography matrix. Specifically, the pre-calibrated homography matrix can be labeled as H_front, H_back, H_left, and H_right.

[0056] Taking a vehicle equipped with four cameras as an example, which can be referred to as the front camera, rear camera, left camera, and right camera, the process of adjusting the coordinates of the projection points of multiple cameras is introduced.

[0057] In some embodiments, the projection point coordinates of the front and rear cameras can be kept unchanged, and the offset of all stitching areas can be added to the corresponding projection point coordinates of the left and right cameras. That is, the positions of the front and rear bird's-eye views are fixed, and the positions of the left and right bird's-eye views are finely adjusted.

[0058] Specifically, the offset of the stitched area between the lane line bird's-eye view images captured by the left and front cameras, denoted as offset_left_front, can be added to the coordinates of the left front projection point of the left camera; the offset of the stitched area between the lane line bird's-eye view images captured by the left and rear cameras, denoted as offset_left_back, can be added to the coordinates of the left rear projection point of the left camera; the offset of the stitched area between the lane line bird's-eye view images captured by the right and front cameras, denoted as offset_right_front, can be added to the coordinates of the right front projection point of the right camera; and the offset of the stitched area between the lane line bird's-eye view images captured by the right and rear cameras, denoted as offset_right_back, can be added to the coordinates of the right rear projection point of the right camera.

[0059] In some embodiments, the projection point coordinates of the left and right cameras can be kept unchanged, and the offset of all stitching areas can be added to the corresponding projection point coordinates of the front and rear cameras. That is, the positions of the left and right bird's-eye view are fixed, and the positions of the front and rear bird's-eye view are finely adjusted.

[0060] Specifically, the offset of the stitched area of ​​the lane line bird's-eye view images captured by the front camera and the left camera can be added to the coordinates of the front left projection point of the front camera; the offset of the stitched area of ​​the lane line bird's-eye view images captured by the front camera and the right camera can be added to the coordinates of the front right projection point of the front camera; the offset of the stitched area of ​​the lane line bird's-eye view images captured by the rear camera and the left camera can be added to the coordinates of the rear left projection point of the rear camera; and the offset of the stitched area of ​​the lane line bird's-eye view images captured by the rear camera and the right camera can be added to the coordinates of the rear right projection point of the rear camera.

[0061] Step 250: Using the corrected homography matrix, perform bird's-eye view stitching on images subsequently captured by multiple cameras in the vehicle.

[0062] Thus, after obtaining the corrected homography matrix, the pre-calibrated homography matrix of the vehicle can be replaced with the corrected homography matrix. Since the corrected homography matrix is ​​based on reverse calibration, the calibration error is eliminated. Therefore, the problem of uneven lane line stitching in the bird's-eye view obtained by BEV function stitching caused by calibration error is solved, which can provide the driver with a good visual effect and greatly improve the user's driving experience.

[0063] like Figure 6 As shown, a method is provided that utilizes the corrected homography matrix to... Figure 1 A diagram illustrating the reassembly process, from... Figure 6 It can be seen that the problem of uneven lane line splicing in the circled area has been well resolved.

[0064] This application provides a scheme for correcting the homography matrix by performing reverse calibration on images with uneven lane line stitching. This corrected homography matrix allows for the stitching of bird's-eye view images subsequently captured by multiple cameras within the vehicle. Since the corrected homography matrix is ​​based on reverse calibration, calibration errors are eliminated. This solves the problem of uneven lane line stitching in the bird's-eye view image obtained by BEV (Battery on the Vehicle) function due to calibration errors, providing the driver with a better visual experience and significantly enhancing the user's driving experience.

[0065] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0066] The following are device embodiments of this application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0067] Figure 7 A schematic diagram of the image processing apparatus provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiments of this application are shown, and are described in detail below:

[0068] like Figure 7 As shown, the image processing apparatus includes:

[0069] The first acquisition module 710 is used to acquire lane line images captured by multiple cameras in the vehicle at the same time, and to convert the lane line images captured by multiple cameras into bird's-eye view images to obtain multiple lane line bird's-eye view images.

[0070] The second acquisition module 720 is used to preprocess and perform Hough transform on each lane line bird's-eye view to obtain the lane fitting line corresponding to each lane line bird's-eye view.

[0071] The calculation module 730 is used to sequentially calculate the offset of the two lane fitting lines corresponding to any two lane line bird's-eye views with splicing areas in the corresponding splicing areas, until the offset of all splicing areas corresponding to multiple lane line bird's-eye views is obtained.

[0072] The correction module 740 is used to correct the pre-calibrated homography matrix of the vehicle based on the offset of all splicing areas, so as to obtain the corrected homography matrix.

[0073] The stitching processing module 750 is used to stitch together bird's-eye view images taken by multiple cameras in the vehicle using the corrected homography matrix.

[0074] In one possible implementation, the second acquisition module is also used for:

[0075] The target lane line bird's-eye view is sequentially binarized, distorted, and quadrilateralized to obtain the lane graphic corresponding to the target lane line bird's-eye view; the target lane line bird's-eye view is any one of multiple lane line bird's-eye views.

[0076] Perform edge detection processing on the lane image to generate a lane edge mask;

[0077] Perform a Hough transform on the lane edge mask to obtain the linear coordinate data corresponding to the lane edge mask;

[0078] By performing line fitting on the linear coordinate data corresponding to the lane edge mask, the fitted straight line corresponding to the target lane line in the bird's-eye view is obtained.

[0079] In one possible implementation, the computation module is also used for:

[0080] Obtain the first intersection point of the first fitted line and the splicing seam of the target splicing area, and the second intersection point of the second fitted line and the splicing seam of the target splicing area; wherein, the first fitted line is any one of the two lane fitted lines, the second fitted line is a lane fitted line that is different from the first fitted line among the two lane fitted lines, and the target splicing area is the splicing area of ​​any two bird's-eye view images of lane lines with splicing areas.

[0081] Calculate the distance between the first intersection point and the second intersection point, and determine the calculated distance as the offset of the target stitching area.

[0082] In one possible implementation, the correction module is also used for:

[0083] Based on the offset of all stitched areas, the coordinates of the projection points of multiple cameras are adjusted according to a preset adjustment method; the preset adjustment method is used to eliminate the offset of all stitched areas.

[0084] Based on the projection point coordinates of multiple cameras obtained through adjustment, the pre-calibrated homography matrix of the vehicle is corrected to obtain the corrected homography matrix.

[0085] In one possible implementation, the multiple cameras are a front camera, a rear camera, a left camera, and a right camera arranged around the vehicle.

[0086] Correspondingly, the calibration module is also used for:

[0087] Keeping the projection point coordinates of the front and rear cameras unchanged, the offset of all stitching areas is added to the corresponding projection point coordinates of the left and right cameras.

[0088] Alternatively, keep the projection point coordinates of the left and right cameras unchanged, and add the offset of all stitched areas to the corresponding projection point coordinates of the front and rear cameras.

[0089] In one possible implementation, the correction module is also used for:

[0090] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the left camera and the front camera to the coordinates of the left front projection point of the left camera.

[0091] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the left camera and the rear camera to the coordinates of the left rear projection point of the left camera.

[0092] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the right camera and the front camera to the coordinates of the right front projection point of the right camera.

[0093] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the right camera and the rear camera to the coordinates of the right rear projection point of the right camera.

[0094] In one possible implementation, the correction module is also used for:

[0095] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the front camera and the left camera to the coordinates of the front left projection point of the front camera.

[0096] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the front camera and the right camera to the coordinates of the front right projection point of the front camera.

[0097] Add the offset of the stitched area of ​​the lane line bird's-eye view captured by the rear camera and the left camera to the coordinates of the rear left projection point of the rear camera.

[0098] The offset of the stitched area of ​​the lane line bird's-eye view images captured by the rear camera and the right camera is added to the coordinates of the rear right projection point of the rear camera.

[0099] In this embodiment, since the corrected homography matrix is ​​obtained based on reverse calibration, the calibration error is eliminated. Therefore, the problem of uneven lane line stitching in the bird's-eye view obtained by BEV function stitching caused by calibration error is solved, which can provide the driver with a good visual effect and greatly improve the user's driving experience.

[0100] This application also provides a computer program product having program code that, when run in a corresponding processor, controller, computing device, or terminal, executes the steps in any of the above-described image processing method embodiments, for example... Figure 2Steps 210 to 250 are shown. Those skilled in the art will understand that the methods and apparatus proposed in the embodiments of this application can be implemented in various forms, including hardware, software, firmware, dedicated processors, or combinations thereof. Dedicated processors may include application-specific integrated circuits (ASICs), reduced instruction set computers (RISCs), and / or field-programmable gate arrays (FPGAs). The proposed methods and apparatus are preferably implemented as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. This is typically based on a machine with a computer platform, such as one or more central processing units (CPUs), random access memory (RAM), and one or more input / output (I / O) interfaces. An operating system is also typically installed on the computer platform. The various processes and functions described herein may be part of an application program, or a portion thereof may be executed by an operating system.

[0101] Figure 8 This is a schematic diagram of the electronic device 8 provided in an embodiment of this application. For example... Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the various image processing method embodiments described above, for example... Figure 2 Steps 210 to 250 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module in the above-described device embodiments, for example... Figure 7 The functions of modules 710 to 750 are shown.

[0102] For example, the computer program 82 can be divided into one or more modules, which are stored in the memory 81 and executed by the processor 80 to complete this application. The one or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 82 in the electronic device 8. For example, the computer program 82 can be divided into... Figure 7 Modules 710 to 750 are shown.

[0103] The electronic device 8 may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0104] The processor 80 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0105] The memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or memory. The memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 81 can include both internal and external storage units of the electronic device 8. The memory 81 is used to store the computer program and other programs and data required by the electronic device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0106] This application also provides a vehicle, such as... Figure 9 As shown, the vehicle 9 includes the aforementioned electronic equipment 8.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0108] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various image processing method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0114] Furthermore, the features of the embodiments shown in the accompanying drawings or the various embodiments mentioned in this specification should not be construed as independent embodiments. Rather, each feature described in one example of an embodiment can be combined with one or more other desired features from other embodiments to produce other embodiments not described in words or with reference to the accompanying drawings.

[0115] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: The system acquires lane line images captured by multiple cameras in the vehicle at the same time, and performs bird's-eye view conversion on the lane line images captured by the multiple cameras to obtain multiple lane line bird's-eye view images. Each of the above-view images of the lane lines is preprocessed and subjected to Hough transform to obtain the lane fitting line corresponding to each of the above-view images of the lane lines. Calculate the offset of the two lane fitting lines corresponding to any two lane line bird's-eye views with splicing areas in the corresponding splicing areas, until the offset of all splicing areas corresponding to the multiple lane line bird's-eye views is obtained. The pre-calibrated homography matrix of the vehicle is corrected based on the offset of all the splicing regions to obtain the corrected homography matrix; Using the corrected homography matrix, bird's-eye view images subsequently captured by multiple cameras in the vehicle are stitched together. The step of sequentially calculating the offset of the two lane fitting lines corresponding to any two bird's-eye views of the lane lines with splicing areas in the corresponding splicing areas includes: Based on the intersection points of the two lane fitting lines corresponding to any two bird's-eye views of the lane lines with splicing areas and the splicing seams of the corresponding splicing areas, calculate the offset of the corresponding splicing area. The step of correcting the pre-calibrated homography matrix of the vehicle based on the offset of all spliced ​​regions to obtain the corrected homography matrix includes: Based on the offset of all the stitched areas, the projection point coordinates of the multiple cameras are adjusted according to a preset adjustment method; wherein, the preset adjustment method is used to eliminate the offset of all the stitched areas; Based on the adjusted projection point coordinates of the multiple cameras, the pre-calibrated homography matrix of the vehicle is corrected to obtain the corrected homography matrix.

2. The image processing method according to claim 1, characterized in that, The step of preprocessing and performing Hough transform on each of the lane line bird's-eye views to obtain the lane fitting line corresponding to each lane line bird's-eye view includes: The target lane line bird's-eye view is sequentially subjected to binarization, distortion removal, and quadrilateral filtering to obtain the lane graphic corresponding to the target lane line bird's-eye view; wherein, the target lane line bird's-eye view is any one of the multiple lane line bird's-eye views. The lane pattern is subjected to edge detection processing to generate a lane edge mask; Perform a Hough transform on the lane edge mask to obtain the linear coordinate data corresponding to the lane edge mask; By performing line fitting on the linear coordinate data corresponding to the lane edge mask, the lane fitting line corresponding to the bird's-eye view of the target lane line is obtained.

3. The image processing method according to claim 1, characterized in that, The step of calculating the offset of the corresponding splicing area based on the intersection points of the two lane fitting lines corresponding to any two bird's-eye views of the lane lines with splicing areas and the splicing seams of the corresponding splicing areas includes: Obtain the first intersection point of the first fitted line and the seam of the target splicing area, and the second intersection point of the second fitted line and the seam of the target splicing area; wherein, the first fitted line is any one of the two lane fitted lines, the second fitted line is a lane fitted line different from the first fitted line among the two lane fitted lines, and the target splicing area is the splicing area of ​​any two bird's-eye views of the lane lines with splicing areas. Calculate the distance between the first intersection point and the second intersection point, and determine the calculated distance as the offset of the target splicing area.

4. The image processing method according to claim 1, characterized in that, The multiple cameras are front cameras, rear cameras, left cameras and right cameras arranged around the vehicle; The step of adjusting the projection point coordinates of the multiple cameras according to a preset adjustment method based on the offset of all the stitched areas includes: While keeping the projection point coordinates of the front camera and the rear camera unchanged, the offset of all stitching areas is added to the corresponding projection point coordinates of the left camera and the right camera; Alternatively, while keeping the projection point coordinates of the left and right cameras unchanged, the offsets of all stitched areas are added to the corresponding projection point coordinates of the front and rear cameras.

5. The image processing method according to claim 4, characterized in that, The step of adding the offsets of all the stitched areas to the corresponding projection point coordinates of the left and right cameras includes: The offset of the stitching area of ​​the lane line bird's-eye view captured by the left camera and the front camera respectively is added to the coordinates of the left front projection point of the left camera. The offset of the stitching area of ​​the lane line bird's-eye view captured by the left camera and the rear camera respectively is added to the coordinates of the left rear projection point of the left camera. The offset of the stitching area of ​​the lane line bird's-eye view captured by the right camera and the front camera respectively is added to the coordinates of the right front projection point of the right camera. The offset of the stitched area of ​​the lane line bird's-eye view captured by the right camera and the rear camera is added to the coordinates of the right rear projection point of the right camera.

6. The image processing method according to claim 4, characterized in that, The step of adding the offset of all the stitched areas to the corresponding projection point coordinates of the front camera and the rear camera includes: Add the offset of the stitching area of ​​the lane line bird's-eye view captured by the front camera and the left camera to the coordinates of the front left projection point of the front camera. Add the offset of the stitching area of ​​the lane line bird's-eye view captured by the front camera and the right camera to the coordinates of the front right projection point of the front camera. The offset of the stitching area of ​​the lane line bird's-eye view captured by the rear camera and the left camera is added to the coordinates of the rear left projection point of the rear camera. The offset of the stitching area of ​​the lane line bird's-eye view captured by the rear camera and the right camera is added to the coordinates of the rear right projection point of the rear camera.

7. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire lane line images captured by multiple cameras in the vehicle at the same time, and to convert the lane line images captured by the multiple cameras into bird's-eye view images to obtain multiple lane line bird's-eye view images. The second acquisition module is used to preprocess and perform Hough transform on each of the lane line bird's-eye views to obtain the lane fitting line corresponding to each of the lane line bird's-eye views. The calculation module is used to sequentially calculate the offset of the two lane fitting lines corresponding to any two lane line bird's-eye views with splicing areas in the corresponding splicing areas, until the offset of all splicing areas corresponding to the multiple lane line bird's-eye views is obtained. The correction module is used to correct the pre-calibrated homography matrix of the vehicle based on the offset of all the splicing areas, so as to obtain the corrected homography matrix. The stitching processing module is used to stitch together bird's-eye view images subsequently captured by multiple cameras in the vehicle using the corrected homography matrix. The computing module is specifically used for: Based on the intersection points of the two lane fitting lines corresponding to any two bird's-eye views of the lane lines with splicing areas and the splicing seams of the corresponding splicing areas, calculate the offset of the corresponding splicing area. The correction module is specifically used for: Based on the offset of all the stitched areas, the projection point coordinates of the multiple cameras are adjusted according to a preset adjustment method; wherein, the preset adjustment method is used to eliminate the offset of all the stitched areas; Based on the adjusted projection point coordinates of the multiple cameras, the pre-calibrated homography matrix of the vehicle is corrected to obtain the corrected homography matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6 above.

9. A vehicle, characterized in that, Including the electronic device as described in claim 8.

Images