A time synchronization-based vehicle-mounted side rear view image generation method

Abstract

The application provides a kind of vehicle-mounted side rear view image generation method based on time synchronization, at least comprising: time synchronization is carried out to the time of image acquisition of multiple vehicle-mounted Ethernet cameras, secondary distortion correction is carried out to the collected images, automatically judge the redundant cameras, select the uncontaminated camera as the working camera, establish the target coordinate system, and fuse the images taken by multiple different working cameras to obtain the final image. By using vehicle-mounted Ethernet cameras, Ethernet synchronization technology and CMOS passive trigger acquisition are combined, the acquisition of each camera at the same time is realized, thereby solving the root cause of the uncontrollable error of the fusion boundary image caused by the fusion of multiple camera images. Through the algorithm, the camera with more picture information is selected as the working camera in real time, so that the camera can provide as much image information as possible for driving, and the safety of driving is improved.

Description

Technical Field

[0001] This invention relates to the automotive field, and more particularly to a method for generating vehicle side and rear view images based on time synchronization. Background Technology

[0002] The development of side-view and rear-view applications is rapid, with most using in-vehicle cameras to directly replace rearview mirrors and displaying fisheye images captured by cameras at various locations on the corresponding areas of the screen. However, from a visual perspective, the images from each camera are independent and discontinuous, resulting in a poor subjective experience for the driver. Furthermore, since cameras are electronic products, they are subject to certain failures. External cameras can still be contaminated by dust and mud, causing blurring or obstruction in certain areas of the image, reducing the driving experience and potentially leading to accidents. Currently, camera contamination is still judged by the human eye; there is no automatic method for detecting camera contamination. This means that when contamination affects image quality, human judgment and correction are required. If the driver does not notice and address the issue promptly, relying on judgments based on a contaminated camera could lead to safety accidents. Therefore, existing technology still has many shortcomings and needs improvement. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a time-synchronized method for generating vehicle side and rear view images, comprising at least:

[0004] Step S1: Synchronize the image acquisition time of multiple vehicle-mounted Ethernet cameras and perform distortion correction on the acquired images;

[0005] Step S2: Obtain images captured by redundant vehicle-mounted Ethernet cameras, and obtain a fused image after image fusion. Based on the original distorted image corresponding to the fused image, select the uncontaminated vehicle-mounted Ethernet camera as the working camera.

[0006] Step S3: Obtain images captured by working cameras at different installation positions and perform differential homography matrix correction in distortion correction. Correction and fusion homography matrix The image captured by the working camera is mapped to obtain the image of the fusion plane corresponding to the working camera at the installation position.

[0007] Step S4: The pixel values ​​of the overlapping areas of the images of working cameras at different installation positions on the fused image are fused according to the information weights.

[0008] A method for generating vehicle side and rear view images based on time synchronization is further described, wherein the vehicle Ethernet camera transmits image content and control information through the Ethernet EAVB protocol, and has a built-in RTC clock. The intelligent host corrects the RTC clock of each vehicle Ethernet camera at regular intervals through the vehicle Ethernet bus.

[0009] A method for generating vehicle side and rear view images based on time synchronization, further comprising time synchronization of the image acquisition time of the vehicle Ethernet camera, specifically including:

[0010] The vehicle-mounted Ethernet cameras calculate their respective link propagation delays with the main processor. ;

[0011] When the main processor periodically sends local time messages to the vehicle Ethernet camera, the vehicle Ethernet camera parses the local timestamp tm recorded in the message, and then adds the link propagation delay. Get the local RTC timestamp tc of the main processor after the message is received;

[0012] Meanwhile, the vehicle-mounted Ethernet cameras modify their local RTC clock timestamps to tc.

[0013] A method for generating vehicle side and rear view images based on time synchronization, further comprising at least a first distortion correction, wherein the first distortion correction comprises at least: acquiring a distorted image after imaging a preset target image perpendicular to the ground through a redundant vehicle Ethernet camera, and obtaining a first distortion-corrected image after performing a first distortion correction on the distorted image using an optical distortion parameter mapping table.

[0014] A method for generating vehicle side and rear view images based on time synchronization, further comprising at least a second distortion correction, wherein the second distortion correction includes at least: establishing a difference correction plane coordinate system, obtaining the first coordinate points of the feature points of the preset target image corresponding to the first coordinate points of the first distortion-corrected image and the second coordinate points corresponding to the feature points of the first distortion-corrected image in the difference correction plane, and solving the homography matrix using perspective transformation. Using homography matrix Perform an inverse transformation to perform a second distortion correction on the first distortion-corrected image;

[0015] Obtain the homography matrix Then, by using inverse perspective transformation, all the first coordinate points corresponding to the difference correction plane are established as the second coordinate points in the first distortion correction image. The pixel values ​​of the second coordinate points in the first distortion correction image are obtained and the pixel values ​​are assigned to the first coordinate points.

[0016] A method for generating vehicle side and rear view images based on time synchronization further includes the following step S2: For redundant vehicle Ethernet cameras installed at the same location, the distorted images of the original cameras after synchronous acquisition are divided into several regions, and the sharpness score of each region is calculated independently. A set of sharpness scores for each region is obtained for each camera. Each redundant camera Each corresponds to a set of sharpness scores .

[0017] A method for generating vehicle side and rear view images based on time synchronization further includes, in step S2, mapping the corrected images of redundant vehicle Ethernet cameras installed at the same location onto the fused image of the actual display application, thereby obtaining the corrected fusion homography matrix of the fused images corresponding to the installation position differences of each vehicle Ethernet camera. And set the regional information weighting function for the merged image. And based on the installation location, each redundant vehicle-mounted Ethernet camera Sharpness score and its corresponding difference-corrected homography matrix Obtain information weights Among the redundant cameras, the camera with the highest information weight is the working camera, and the rest are standby cameras.

[0018] A method for generating vehicle side and rear view images based on time synchronization, further comprising: a fused image of the coverage areas of the right-side and rear-side working cameras. Images fused from the coverage areas of the left and rear working cameras ;

[0019] The fused image of the camera coverage area includes the weighted average of the images and information weights of the working cameras in the corresponding area on the final fusion plane. The fusion calculation method includes:

[0020]

[0021]

[0022] Among them, the information weights of the left, rear, and right working cameras.

[0023] These are images from the right, rear, and left working cameras on the final fusion plane, respectively.

[0024] A method for generating vehicle side and rear view images based on time synchronization, further comprising the information weights of the left, rear, and right working cameras. The calculation formula is as follows:

[0025]

[0026]

[0027]

[0028] in The sharpness score is given to the rear working camera; The sharpness score is given to the working camera on the left. Score the clarity of the working camera on the right.

[0029] This is the information weighting function.

[0030] A time-synchronized method for generating vehicle side and rear view images, further comprising obtaining a sharpness score specifically including: dividing the acquired original distorted image into VNUM×HNUM small blocks with a side length of b1. Calculate the sharpness score within each small block. The sharpness score is obtained by summing the sharpness scores within each small block of n consecutive frames of images. :

[0031]

[0032] in, This represents the current image frame number, RESH and RESV represent the horizontal and vertical resolutions of the image, respectively, and bl is the side length of the small block. , where HNUM = VNUM = .

[0033] A method for generating vehicle side and rear view images based on time synchronization, further, a sharpness score. The calculation methods include:

[0034]

[0035] in, This represents a small block with vertical index v and horizontal index h, f is the frame number of the current image, bl is the side length of the small block, and i and j represent the pixel coordinates within the small block image, i∈[1,bl], j∈[1,bl];

[0036] Set the resolution score threshold , The settings are based on the bl size and the number of frames counted, n. The range is 8 to 12;

[0037] Sharpness score threshold

[0038]

[0039] The sharpness score of each small block is compared with the sharpness score threshold. If the score is greater than or equal to the sharpness score threshold, the image sharpness of that area is considered to meet the requirements.

[0040] A method for generating vehicle side and rear view images based on time synchronization, further comprising the following steps: The acquisition of the fused images includes at least the following:

[0041] Step S100: Obtain the pixel coordinates of the four vertices of the preset target image rectangle ROI region in the fused and corrected image, in a clockwise direction. , , , ;

[0042] Step S101: Let the observation point be point O in the three-dimensional coordinate system, the projection of point O onto the horizontal plane G be g, point g be the origin of the three-dimensional coordinate system, and point g be located on the vertical center line of the rear camera image on the target image plane. Establish the target image coordinate system with point g as the origin.

[0043] Step S102: Based on the corner point order in a clockwise direction, C1, C2, C3, and C4 respectively, solve the pixel coordinates nameFC(i)x and nameFC(i)y of the fusion image observed from observation point O in the ROI region after the target image is captured by the right redundant camera, left redundant camera, and rear redundant camera in the target image coordinate system, where name=right,rear,left; i=1,2,3,4;

[0044] Step S103, let the pixel distance between the installation positions of the left and right cameras and their corresponding target images be... The pixel coordinates of the fused image of the ROI region after imaging by the right redundant camera, left redundant camera, and rear redundant camera, observed from the observation point O, are transformed into the vertex coordinates of the ROI region: nameC(i)x, nameC(i)y, where name=right,rear,left; i=1,2,3,4.

[0045] A method for generating vehicle side and rear view images based on time synchronization, further comprising obtaining the corresponding original distorted image from the fused image, including at least:

[0046] Step S201: Based on the pixel coordinates of the observed fused image and the pixel coordinates of the four vertices of the target image's rectangular ROI region, nameC(i)x, nameC(i)y, where name=right,rear,left; i=1,2,3,4, solve using perspective transformation. Obtain the homography matrix of multi-camera fused images. The difference-corrected image from multiple cameras is denoted as nameFC(i)x and nameFC(i)y, where name = right, rear, left; i = 1, 2, 3, 4.

[0047] Step S202: Repeat step S201 to sequentially obtain the homography matrices corresponding to the right, rear, and left redundant cameras at different positions. Difference correction images from multiple cameras;

[0048] Step S203: The final planar image from the camera is fused. ), Using perspective transformation, the difference-corrected image coordinates corresponding to the pixel coordinates of the final fused image from each camera are obtained. );

[0049] Step S204, difference-corrected image coordinates ( ) and the homography matrix of the corresponding camera Using perspective transformation, obtain the distortion-corrected image coordinates corresponding to the difference-corrected image coordinates of each camera. );

[0050] Step S205: Based on the distortion correction principle of fisheye cameras, obtain the coordinates of the distortion-corrected image (…). The coordinates of the corresponding original distorted image () , );

[0051] Step S206: Repeat steps S203 to S205 to obtain the original distorted image coordinates corresponding to different cameras. , ), where k is the camera number.

[0052] A method for generating vehicle side and rear view images based on time synchronization, further including information weights. The calculation methods include:

[0053]

[0054] in, This indicates the number of the redundant camera; RESQH and RESQV represent the horizontal and vertical resolutions of the fused image, respectively. This represents the pixel coordinates of the fused image, and f represents the frame sequence number;

[0055] This represents the sharpness score of redundant camera k. This represents the weighting function for regional information in the merged image.

[0056] A method for generating vehicle side and rear view images based on time synchronization, further comprising a region information weighting function. Acquisition includes:

[0057]

[0058] Among them, the vertical information weight function Horizontal information weighting function ;

[0059]

[0060]

[0061] in, The pixel horizontal resolution of the final merged image;

[0062] Among them, ofst_h1, ofst_h2, and low_val are all manually set thresholds, low_val is the subjectively perceived lowest weight, and ofst_h1 is the subjectively determined lowest weight region. The threshold is set based on experience;

[0063] This represents the point from the top pixel to the right camera ROI (the area of ​​interest in the target image captured by the right camera). The vertical pixel distance.

[0064] A time-synchronized method for generating vehicle side and rear view images, preferably wherein the first distortion correction includes:

[0065] Step S1: Obtain the optical distortion parameters of the vehicle-mounted Ethernet camera. The optical distortion parameters include a mapping table of the discrete field of view angle θ and its corresponding image height γ of the experimental module for image acquisition in the actual experimental environment under the specified CMOS chip conditions of the camera lens.

[0066] Step S2: Based on the pinhole imaging principle, establish the spatial coordinate relationship between each pixel point of the actual shooting plane U and each point of the fisheye image on the imaging plane I, and establish the first mapping function between the discrete field of view angle θ and the object height λ.

[0067] λ = tan(θ)

[0068] Step S3: Based on the optical distortion parameter mapping table, construct the second mapping function θ=G(γ) between image height γ and discrete field of view angle θ:

[0069] Step S4: Based on the imaging geometry, solve for the correction scaling factor between the actual imaging plane U and the corresponding corrected imaging plane and distortion imaging plane. and distortion scaling factor ;

[0070] Step S5: Utilize the first mapping function, the second mapping function, and the corrected scaling factor. and distortion scaling factor Establish pixel-level coordinates of the corrected distortion-free image. pixel-level coordinates of the distorted image The corresponding third mapping function, and Assign the pixel value of the coordinate position to The coordinates of the image are used to obtain the image after distortion correction.

[0071] Beneficial effects:

[0072] 1. The technical solution of this invention utilizes a time-synchronized side and rear view image generation system. By employing Ethernet cameras and combining Ethernet synchronization technology with passive triggering acquisition by CMOS, it achieves simultaneous acquisition by all cameras, enabling true simultaneous photography of the three-dimensional world. This fundamentally avoids image changes caused by shifts in the actual object's position due to different camera acquisition times during driving, thus solving the root cause of uncontrollable errors in the fusion boundary image resulting from multi-camera image fusion. Furthermore, by using redundant cameras working synchronously and employing algorithms to select the camera with the most image information in real time, the system ensures that the cameras provide as much image information as possible for driving, improving driving safety. The multi-image fusion technology based on planar texture mapping also better aligns with the subjective observation experience of the human eye.

[0073] 2. The technical solution of the present invention, through an in-vehicle Ethernet camera, plus the EAVB protocol and corresponding time synchronization correction, can keep the images captured by redundant cameras at the same time, which helps to stitch the fused images and improve the imaging effect.

[0074] 3. In the technical solution provided by this invention, when determining whether a camera is contaminated, information weights are introduced, and a weight function is creatively defined that the information weight equals the sharpness score and the weight of the fused image area information. During the calculation of the information weights, not all areas of the camera are considered. A target image coordinate system is established through a preset target image, and the final effective area is determined after image fusion. The sharpness score and area information weight of the effective area are then calculated, making the calculation results more accurate and avoiding the inclusion of parts of the image formed by cameras that do not participate in the fusion. That is, parts that do not affect the formation of a safe image are not considered.

[0075] 4. The technical solution provided by this invention creatively proposes a method for calculating sharpness score, which divides the image region into blocks and calculates the pixel difference of each small block, then continuously calculates the difference of multiple frames of images, sets a threshold for comparison, and makes the calculation results more scientific and accurate. Attached Figure Description

[0076] The following figures are for illustrative purposes only and do not limit the scope of the invention.

[0077] Figure 1 This is a schematic diagram of the time link between the vehicle-mounted Ethernet camera and the main processing unit in one embodiment of the present invention;

[0078] Figure 2 This is a mapping table of discrete field-of-view angles and their corresponding image heights obtained from actual experimental environment image acquisition of the experimental module under the condition of LENS matching and specified CMOS in one embodiment of the present invention.

[0079] Figure 3 This is a schematic diagram of the imaging model principle of a fisheye camera in one embodiment of the present invention.

[0080] Figure 4 This is an undistorted checkerboard image captured by a fisheye camera in one embodiment of the present invention.

[0081] Figure 5 This is a checkerboard image after the first distortion correction in one embodiment of the present invention.

[0082] Figure 6 This is an image after a second distortion correction in one embodiment of the present invention.

[0083] Figure 7 This is a schematic diagram illustrating the horizontal and vertical sharpness scores within each small block of n consecutive frames of images in one embodiment of the present invention.

[0084] Figure 8 This is a schematic diagram illustrating a severely contaminated camera surface and the dyeing of the contaminated area in one embodiment of the present invention. Figure 8'a' indicates that the original image has a severely contaminated surface. Figure 8 b represents the original image that has been processed and colored.

[0085] Figure 9 This is a schematic diagram illustrating slight contamination on the camera surface and the dyeing of the contaminated area in one embodiment of the present invention. Figure 9 'a' represents the original image with slight surface contamination. Figure 9 b represents the original image that has been processed and colored.

[0086] Figure 10 This is a schematic diagram of the pixel coordinates of the four vertices of the rectangular ROI region of the target image T in a distortion-corrected image according to an embodiment of the present invention.

[0087] Figure 11 This is a schematic diagram of a target image formed by the left, rear, and right cameras in one embodiment of the present invention, arranged on the same plane and perpendicular to the horizontal plane.

[0088] Figure 12 This is a schematic diagram of observing a target image from the perspective of an observation point in one embodiment of the present invention.

[0089] Figure 13 This is a schematic diagram of the coverage area displayed after the images captured by the left, rear, and right cameras are fused in one embodiment of the present invention.

[0090] Figure 14 This is a schematic diagram of an image formed on a fusion plane after the image captured by the working camera is installed at a preset installation position and facing a specified field of view in one embodiment of the present invention.

[0091] Figure 15 This is a schematic diagram illustrating how, in one embodiment of the present invention, an information weight of 1 in an image is colored pure white, and an information weight of 0 is colored pure black.

[0092] Figure 16 Images from redundant cameras (uncontaminated and contaminated) in various regions are shown in one embodiment of the present invention.

[0093] Figure 17 This is the final image with textures added to the final fused image of the redundant cameras in each region in one embodiment of the present invention. Detailed Implementation

[0094] To provide a clearer understanding of the technical features, objectives, and effects of this invention, specific embodiments are now described with reference to the accompanying drawings, in which the same reference numerals denote the same parts. For the sake of simplicity, the parts related to this invention are shown schematically in each drawing and do not represent their actual structure as a product. Furthermore, for the sake of clarity and ease of understanding, in some drawings, components with the same structure or function are shown only schematically, or only one is labeled.

[0095] Regarding control systems, as is well known to those skilled in the art, functional modules and application programs (APPs) can take any suitable form, whether hardware or software, and can be multiple discrete functional modules or multiple functional units integrated onto a single hardware device. In its simplest form, the control system can be a controller, such as a combinational logic controller or a microprogrammed controller, as long as it can implement the operations described in this application. Of course, the control system can also be integrated as different modules onto a single physical device, without departing from the basic principles and scope of protection of this invention.

[0096] In this invention, "connection" can include direct connection, indirect connection, communication connection, and electrical connection, unless otherwise specified.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly specifies otherwise. It will also be understood that, when used in the specification, the terms “comprising” and / or “including” mean the presence of the stated features, values, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, values, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items.

[0098] It should be understood that the term “vehicle” or “of a vehicle” or other similar terms used herein generally include motor vehicles, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, boats including various vessels, aircraft, etc., and also include hybrid vehicles and electric vehicles.

[0099] Specifically, the present invention provides a time-synchronized method for generating vehicle side and rear view images, which is applied to a car equipped with multiple cameras. The multiple cameras are connected to a smart host through an in-vehicle Ethernet bus, and the smart host controls the fusion and stitching of images captured by the multiple cameras to form the final side and rear view.

[0100] Existing technological solutions, such as in-vehicle surround view systems, involve installing four cameras at the front, rear, left, and right of a car to achieve a 360-degree surround view. However, if only one camera is installed in the same area, the surround view system will fail if the camera in that area becomes contaminated.

[0101] Therefore, in this embodiment, multiple cameras are installed in the same area. If one camera is contaminated, the others can still be activated and operate. However, there is no primary or secondary distinction among the multiple cameras; any one of them can operate if it is uncontaminated, and if it is contaminated, the uncontaminated camera will be selected to operate.

[0102] Ideally, there should be two redundant cameras.

[0103] Prior to this, in order to keep the images captured by multiple cameras synchronized for subsequent stitching, this embodiment uses an in-vehicle Ethernet camera with a fisheye lens.

[0104] A method for generating vehicle side and rear view images based on time synchronization, specifically including:

[0105] Step S1: Synchronize the image acquisition time of multiple vehicle-mounted Ethernet cameras and perform distortion correction on the acquired images;

[0106] Step S2: Obtain images captured by redundant vehicle-mounted Ethernet cameras, and obtain a fused image after image fusion. Based on the original distorted image corresponding to the fused image, select the uncontaminated vehicle-mounted Ethernet camera as the working camera.

[0107] Step S3: Obtain images captured by working cameras at different installation positions and perform differential homography matrix correction in distortion correction. Correction and fusion homography matrix The image captured by the working camera is mapped to obtain the image of the fusion plane corresponding to the working camera at the installation position.

[0108] Step S4: The pixel values ​​of the overlapping areas of the images of working cameras at different installation positions on the fused image are fused according to the information weights.

[0109] Specifically, the time synchronization process for the in-vehicle Ethernet camera is as follows:

[0110] The redundant vehicle-mounted Ethernet camera synchronous acquisition module is responsible for synchronously acquiring image data from multiple cameras and transmitting the image data to the fisheye distortion correction module.

[0111] Specifically, the redundant vehicle-mounted Ethernet camera synchronous acquisition module includes at least two wide-angle high-definition Ethernet cameras, each of which is connected to other modules via an Ethernet cable.

[0112] The vehicle-mounted Ethernet camera uses the EAVB protocol to encapsulate the acquired image data into Ethernet packets and transmits them via Ethernet cable for time synchronization. Different vehicle-mounted Ethernet cameras correct their own RTC clocks according to the synchronization time and acquire a frame of image at a unified timestamp.

[0113] Specifically, the acquired image data can be packetized and transmitted using the IEEE 1722 protocol. The data before the AVTP payload is collectively referred to as the header. The header contains customizable control information, such as distinguishing cameras by stream ID. The AVTP payload within the Ethernet packet of the same camera contains image data segments sent sequentially by that vehicle-mounted Ethernet camera.

[0114] Time synchronization between redundant cameras can be achieved specifically through the IEEE 802.1as protocol. Figure 1 As shown.

[0115] For example, a single automotive Ethernet camera sends a Pdelay_Req message to the main processor via Ethernet to request a measurement of the Ethernet signal link propagation delay. When the Pdelay_Req message leaves the automotive Ethernet camera's MAC layer, the camera records the timestamp t1 using its local RTC clock. Simultaneously, when the main processor's MAC layer receives the Pdelay_Req message, it records the timestamp t2 using its local RTC clock. The main processor then replies to the received Pdelay_Req message by sending a Pdelay_Resp message to the Ethernet camera via the automotive Ethernet network. The local timestamp t2 of the _Req message is recorded, and the time t3 when the Pdelay_Resp message is sent from the main processor's MAC layer is recorded using the main processor's local RTC clock. Then, t3 is sent to the vehicle Ethernet camera via a Pdelay_Reso_Follow_Up message. The vehicle Ethernet camera records the timestamp t4 of its local RTC clock when its MAC layer receives the Pdelay_Resp message, and parses the timestamp t3 of the Pdelay_Resp message sent from the main processor's MAC layer by parsing the Pdelay_Resp_Follow_Up message. The transmission delay of the link below the Ethernet MAC layer basically satisfies a symmetrical relationship, that is, the transmission time from the Ethernet camera to the main processor and the transmission time from the main processor to the Ethernet camera are the same. Therefore, the link propagation delay is... It can be calculated using the following formula:

[0116]

[0117] Different Ethernet cameras obtain their respective link propagation delays to the main processor in this way. , ... When the main processor periodically sends local time messages to each Ethernet camera, each Ethernet camera parses the local timestamp tm recorded in the message, and adds its own link propagation delay. This gives us the local RTC timestamp tc of the main processor after receiving the message. At the same time, each Ethernet camera modifies its local RTC clock timestamp to tc, thus achieving time synchronization between all Ethernet cameras and the main processor.

[0118] Each camera acquires one frame of image at preset fixed intervals ts based on the local RTC clock after time synchronization. This achieves synchronous image acquisition by redundant Ethernet cameras and completely avoids spatial image errors caused by differences in acquisition time between different cameras.

[0119] In step S1, distortion correction includes at least a first distortion correction, wherein the first distortion correction includes at least: acquiring a distorted image of a preset target image perpendicular to the ground through a redundant vehicle-mounted Ethernet camera, and obtaining a first distortion-corrected image by performing a first distortion correction on the distorted image using an optical distortion parameter mapping table.

[0120] The first distortion correction specifically includes:

[0121] The automotive Ethernet camera typically uses a fisheye lens. Lens suppliers provide an optical distortion parameter table for this lens. This table maps the discrete field of view angles to their corresponding image heights obtained from image acquisition in actual experimental environments using a LENS-compatible CMOS sensor. Figure 2 .

[0122] Obtain the optical distortion parameters of the vehicle-mounted Ethernet camera. The optical distortion parameters include a mapping table of the discrete field of view angle θ and its corresponding image height γ of the experimental module under the specified CMOS chip conditions of the camera lens in the actual experimental environment.

[0123] Based on the pinhole imaging principle, the spatial coordinate relationship between each pixel point on the actual shooting plane U and each point on the fisheye image on the imaging plane I is established, and the first mapping function between the discrete field of view angle θ and the object height λ is established.

[0124] Specifically, see Figure 3 ,

[0125] The height of the object is λ. Figure 1 In the optical distortion parameter table, the angle θ is in degrees and the image height γ is in mm.

[0126] Let the distance from the optical center of the lens to the actual shooting plane be 1, satisfying the relationship, then:

[0127] λ = tan(θ),(γ -> θ) Formula (1)

[0128] θ= arctan(λ),(θ ->γ) Formula (2)

[0129] First mapping function: λ = tan(θ)

[0130] Based on the optical distortion parameter mapping table, a second mapping function θ=G(γ) is constructed between the image height γ and the discrete field of view angle θ.

[0131] Specifically, in this embodiment, it is assumed that:

[0132] For any angle θ, θ(n) satisfies θ θ(n+1); In the distortion parameter table, find the corresponding γ(n) and γ(n+1) for θ(n) and θ(n+1). Then, the γ corresponding to θ should satisfy γ(n). γ γ(n+1);

[0133] Where n is the pixel number, θ(n) and θ(n+1) represent adjacent discrete angles, γ(n) and γ(n+1) are adjacent image heights, and θ(n), θ(n+1), γ(n), and γ(n+1) are obtained from the optical distortion parameter table based on the known θ or γ.

[0134] when When it is known,

[0135] θ = G(γ) = Formula (3)

[0136] When θ is known

[0137] =F(θ)= Formula (4)

[0138] Based on imaging geometry, solve for the correction scaling factor between the actual imaging plane U and the corresponding corrected imaging plane and distortion imaging plane. and distortion scaling factor ;

[0139] Specifically, it includes:

[0140] Let ψ be the angle between the line connecting point P and the horizontal direction, with ψ ranging from 0 to 180 degrees. Then, the horizontal coordinates of point P on the actual shooting plane U are... and vertical coordinates And the horizontal coordinates of point p' on the imaging plane I and vertical coordinates Satisfying the relation:

[0141] Formula (5)

[0142] Formula (6)

[0143] Select the maximum horizontal width of the CMOS imaging plane of the actual Ethernet camera. and the maximum horizontal resolution of actual fisheye distortion images Maximum vertical resolution and the maximum horizontal resolution of the distortion-corrected image Maximum vertical resolution Substituting into Equations 1 and 2, we can obtain the distortion-free actual captured image. and distortion scaling factor and correction scaling factor Satisfying the relation:

[0144] Formula (7)

[0145] Formula (8)

[0146] Using the first mapping function, the second mapping function, and the corrected scaling factor and distortion scaling factor Establish pixel-level coordinates of the corrected distortion-free image. pixel-level coordinates of the distorted image The corresponding third mapping function, and Assign the pixel value of the coordinate position to The coordinates of the image are used to obtain the image after distortion correction.

[0147] Specifically, it includes:

[0148] Pixel-level coordinates of the fisheye distortion image of the imaging plane Pixel-level coordinates of the actual undistorted image. Satisfying relationships,

[0149] , formula (9)

[0150] Formula (10)

[0151] , formula (11)

[0152] , formula (12)

[0153] Divide both sides of the equations (9) and (11) respectively, and substitute the equations (1), (5), (6), (7), and (8) into the equations to obtain the result. and The corresponding third mapping function:

[0154] Formula (13)

[0155] Similarly: Divide both sides of the equations (10) and (12) respectively, and substitute formulas (1), (5), (6), (7), and (8) into the equations to obtain the results. and The corresponding third mapping function:

[0156] Formula (14)

[0157] Where γ is the image height corresponding to point P, γ max It is half the maximum horizontal width of the camera's CMOS imaging surface. Let be the angle between the line connecting point P and the center point O of the shooting plane U and the horizontal direction. , This represents the maximum horizontal and maximum vertical resolution of the actual fisheye distortion image. , This indicates the maximum horizontal resolution and maximum vertical resolution of the distortion-corrected image.

[0158] Taking a CMOS sensor with a horizontal length of 4mm as an example, when the imaging surface just covers the entire horizontal direction of the CMOS sensor, the horizontal resolution of the resulting fisheye-distorted image is... With 1280 pixels, op corresponds to the horizontal direction. It is 2;

[0159] Assume the minimum accuracy of the optical distortion parameter table satisfies γ(n). γ When γ(n+1) is γ(n), γ(n) is 1.95;

[0160] γ(n+1) is 2.03. It is 55.5. It is 56.

[0161] Substituting these equations into Equations 1 and 2, we can obtain the horizontal position of point P corresponding to OP on the actual shooting plane U. It is 1.4721.

[0162] Assume the horizontal resolution of the actual distortion-free image is... If the value is 1280 pixels, then according to Equation 4, we can obtain... It is 320. It is 434.75.

[0163] The distortion-corrected image is the actual captured U-shaped image, where the coordinates of any pixel point are ( Substituting into formulas (14) and (15), we can obtain the pixel coordinates corresponding to the fisheye distortion image, i.e., the image plane I. Since the image pixel coordinates are integers, the calculated corresponding pixel coordinates are floating-point numbers. Therefore, the actual pixel value of point P can be obtained from... The estimated value is obtained by measuring the values ​​of the surrounding pixels of the point;

[0164] The specific estimation method is as follows:

[0165] ( When ) is a floating point, the four nearest surrounding pixels The coordinates are respectively ( (), (), (), ),

[0166] Get , , The corresponding pixel values ​​are respectively ;

[0167] calculate , , Corresponding weighting coefficients:

[0168]

[0169]

[0170]

[0171]

[0172] Pixel values ​​are calculated based on weighting coefficients.

[0173]

[0174] Traverse the pixel coordinates of point P according to the image resolution of the distortion-corrected image. And calculate the pixel coordinates of point p in the corresponding fisheye distortion image. The pixel value of point P is assigned to the interpolated value calculated at point P. This completes the first distortion correction of the basic fisheye image. The fisheye distortion image is as follows: Figure 4 Distortion-corrected images such as Figure 5 .

[0175] The installation positions of redundant cameras will inevitably have spatial differences, so the images they capture will inevitably have differences. Furthermore, due to the manufacturing process precision, different Ethernet cameras will inevitably have certain optical center offset and focal plane angle errors between the imaging plane and the CMOS plane. All of these factors lead to errors in restoring the fisheye distortion image to the distortion-corrected image according to the established distortion correction mathematical model.

[0176] Redundant camera difference correction eliminates the differences in the ROI regions of the distorted images from redundant Ethernet cameras to the greatest extent possible.

[0177] Therefore, a second distortion correction is required based on the first distortion-corrected image;

[0178] Distortion correction includes at least a second distortion correction, wherein the second distortion correction includes at least: establishing a difference correction plane coordinate system, obtaining the feature points of the preset target image at their corresponding first coordinate points in the first distortion-corrected image and their corresponding second coordinate points in the difference correction plane, and solving for the homography matrix using perspective transformation. Using homography matrix The inverse transformation is then applied to the first distortion-corrected image for a second distortion correction.

[0179] Obtain the homography matrix Then, by using inverse perspective transformation, all the first coordinate points corresponding to the difference correction plane are established as the second coordinate points in the first distortion correction image. The pixel values ​​of the second coordinate points in the first distortion correction image are obtained and the pixel values ​​are assigned to the first coordinate points.

[0180] Specifically, it includes:

[0181] In the difference correction plane coordinate system, the feature points of the preset target image ( ) at the first coordinate corresponding to the difference correction plane ( The mapping function for ) is:

[0182]

[0183]

[0184] in, This is the plane coordinate scale transformation factor. and These are the horizontal and vertical pixel offsets, respectively;

[0185] Step S101: Obtain a preset target image perpendicular to the ground using redundant cameras, and obtain the actual physical coordinates of multiple feature points in the preset target image on the target image plane. ;

[0186] Step S102, based on the actual physical coordinates ( Solve for the coordinates of the actual image plane U after distortion correction and the actual physical coordinates. The corresponding pixel coordinates of the feature points ( );

[0187] Step S103: Establish a difference correction plane coordinate system, and solve for the coordinates of the corresponding feature points in the difference correction plane coordinate system by mapping the coordinates of multiple feature points in the target image to the coordinates of the feature points in the difference correction plane coordinate system. ;

[0188] Step S104, solve based on perspective transformation ( to( homography matrix :

[0189] Perspective transformation:

[0190]

[0191] :

[0192]

[0193] Step S105: Again, using perspective transformation, adjust the coordinates of all pixels in the difference-corrected image. Substitution of 7 ( ), using the result obtained in step S104 Calculate the corresponding coordinates of the image after distortion correction. .

[0194] Then, using the pixel value estimation method employed in the first distortion correction, the distorted pixel coordinates corresponding to all pixel coordinates in the difference-corrected image are calculated. The pixel values ​​in the distortion-corrected image are obtained and assigned to the corresponding coordinates in the difference-corrected image, forming the final difference-corrected image, as shown below. Figure 6 As shown.

[0195] Specifically, during driving or in weather conditions such as rain or snow, although the hydrophobic film on the camera surface can greatly reduce the probability of dirt adhesion, there is still a possibility of stains getting on it. This staining is sudden and its adhesion is variable, which reduces the image information acquired by the Ethernet camera and hinders the driver's perception of the actual environment. Sharpness judgment calculates the sharpness score of each area of ​​the camera to determine the degree of contamination in each area, thereby identifying working and non-working cameras.

[0196] The raw image captured by a single camera is divided into several small IMB blocks with a side length of b1. Within each block, a sharpness score is calculated using a traditional sharpness calculation algorithm. Sharpness is a measure of the information content of an image. Information in the high-frequency and mid-frequency regions of an image accounts for the vast majority of the image information. One method of sharpness calculation is to calculate the differences between adjacent pixels, which reflects the scale of information content. When the image is sharp, the differences between adjacent pixels are large; when the image is blurry, the differences between adjacent pixels are small.

[0197] Specifically, the current frame number Images with a horizontal resolution of RESH and a vertical resolution of RESV According to the side length It can be divided into horizontal HNUM and vertical VNUM small square blocks. ,( ), where HNUM = VNUM = The pixel value of each pixel within the small block is (i,j), (i∈[1,bl],j∈[1,bl]), then the sharpness scores of each small block in the vertical and horizontal directions. The calculation is as follows:

[0198]

[0199] in, This represents a small block with vertical index v and horizontal index h, f is the frame number of the current image, bl is the side length of the small block, and i and j represent the pixel coordinates within the small block image, i∈[1,bl], j∈[1,bl];

[0200] Continuously calculate the horizontal and vertical sharpness scores within each small block of n frames of images, such as... Figure 7 As shown, each small block within the frame is divided into... The sharpness score is obtained by summing the variances of the previous n consecutive frames (v∈[1,VNUM], h∈[1,HNUM]). The sharpness score corresponding to each redundant camera is recorded as follows: :

[0201]

[0202] Set the resolution score threshold , The settings are based on the bl size and the number of frames n. Specifically, it is assumed that the difference between adjacent pixel values ​​is within a certain range. If the image is considered to be starting to blur, then the sharpness score threshold is set. If the following formula is satisfied, based on the classic grayscale recognition scale, You can choose 8 to 12.

[0203]

[0204] The cumulative sharpness score of each small block is compared with the sharpness score threshold. If the score is greater than or equal to the sharpness score threshold, the image sharpness of that area is considered to meet the requirements.

[0205] For easy and intuitive observation, small patches with a sharpness score below the threshold are colored black, while those with a sharpness score at or above the threshold are colored white. Cameras with severely contaminated surfaces and their corresponding patches are statistically colored as follows: Figure 8 As shown, Figure 8 'a' represents the uncolored image. Figure 8 b represents the image after statistical coloring.

[0206] Comparison of cameras and their corresponding small-block statistical coloring Figure 9 As shown, Figure 9 This indicates that the camera surface is only slightly contaminated. Figure 9 'a' represents the uncolored image. Figure 9 b represents the image after statistical coloring.

[0207] Specifically, it's important to note that not all camera areas are of paramount safety significance for driving. For instance, the sky area provides less information for driving safety compared to the ground area, and obscuring this area of ​​the camera with dirt poses less of a safety hazard than obscuring the ground area. As external devices, cameras are always susceptible to contamination. Therefore, if the ground area is uncontaminated, the image captured by that camera is still acceptable for driving safety. Furthermore, the areas selected for the fused image from redundant cameras installed at different locations will vary. Data from some areas within a camera at a particular location will not be included in the final fused image. Therefore, this portion of the image is redundant and can be discarded, meaning that contamination in this area has no impact on the overall fused image. The effective content determination module combines the final display area weights of the cameras with their sharpness scores to calculate the fused information content score for each redundant camera. Based on this score, the camera with the highest score is selected as the working camera, and the remaining cameras are designated as standby cameras.

[0208] The block coordinates of the sharpness score patches obtained by the sharpness score calculation method are defined as follows: , Its horizontal and vertical pixel coordinates are respectively .

[0209] Suppose the target image has k horizontal feature points and j vertical feature points. Based on the steps of the redundant camera difference correction module, the pixel coordinates of the four vertices of the T-shaped ROI region in the target image can be determined in the fused and corrected image. , , , , like Figure 10 As shown.

[0210] Each camera installation area corresponds to a target image. The target images are placed on the same plane perpendicular to the horizontal plane. The relative positions of each target image are on the pixel plane Q where the target images are placed. The three target images are centered horizontally. Figure 11 .in It is the point from the leftmost pixel of the Q pixel screen to the rightmost point of the ROI captured by the camera in the target image. ) horizontal pixel distance, It is the point from the top pixel of Q to the right camera ROI of the target image captured by the camera on the right. Vertical pixel distance, It is the point from the top pixel of Q to the right camera ROI of the target image captured by the camera on the right. Vertical pixel distance;

[0211] It is the point from the leftmost pixel of the Q pixel screen to the rear camera ROI of the target image captured by the rear camera. ) horizontal pixel distance;

[0212] It is the point from the leftmost pixel of the Q pixel screen to the left camera ROI of the target image captured by the left camera. The horizontal pixel distance.

[0213] Let the observation point be point O in the three-dimensional coordinate system, and its projection onto the horizontal plane G be g, as follows: Figure 12 Where point O is the pixel distance from the y-axis of the ground plane. The pixel distance to the target image along the z-axis is The pixel distance from each point on the target image to point O along the x-axis is... Let point g be the origin of the three-dimensional coordinate system. Figure 12 The corner points, ordered from left to right and top to bottom, are C1, C2, C3, and C4. Therefore, the corner coordinates of the fused image Q observed in the ROI regions of each target image are as follows in this coordinate system:

[0214] The right-side camera ROI area,

[0215] =

[0216] =

[0217] =

[0218] =

[0219] =

[0220] =

[0221] =

[0222] =

[0223] Rear camera ROI area

[0224] =

[0225] =

[0226] =

[0227] =

[0228] =

[0229] =

[0230] =

[0231] =

[0232] The left-side camera ROI area,

[0233] =

[0234] =

[0235] =

[0236] =

[0237] =

[0238] =

[0239] =

[0240] =

[0241] Based on the principle of perspective, let the pixel distance between the installation positions of the left and right cameras and their corresponding target images be... The relationship between the four vertices C(i), i=1, 2, 3, 4 of each target image rectangular ROI region in the final fused image, and the pixel coordinates in the fused image seen from observation point O, is as follows:

[0242] The right-side camera ROI area,

[0243] = , = Formula 16

[0244] Rear camera ROI area, type 17

[0245] = , = Formula 17

[0246] Left camera ROI area, type 18

[0247] = , = Formula 18

[0248] Let the homography matrix of the difference-corrected images from each installation location and the fused images from multiple cameras be... , .

[0249] The difference-corrected image from a single camera on the right and the fused image from multiple cameras exhibit a homography matrix. (The remaining text appears to be incomplete and requires further context.) , ), i=1, 2, 3, 4 instead of ( ), ( ),i=1,2,3,4 instead of ( Substituting these values ​​into Equation 8, the homography matrix of the difference-corrected image from the right-side camera and the multi-camera fused image can be obtained. ;

[0250] The difference-corrected image from a single rear camera and the fused image from multiple cameras have a homography matrix, which will ( , ), i=1, 2, 3, 4 instead of ( ), Substituting the values ​​of i=1,2,3,4 into equation 8, we can obtain the homography matrix of the difference-corrected image and the multi-camera fused image from the right-side camera. ;

[0251] The single-camera difference-corrected image on the left and the fused image from multiple cameras have a homography matrix. (The remaining text appears to be incomplete and requires further context.) , ), i=1, 2, 3, 4 instead of ( ), ( Substituting the values ​​of i=1,2,3,4 into equation 8, we can obtain the homography matrix of the difference-corrected image from the left camera and the fused image from the multi-camera system. ;

[0252] Let the pixel resolution of the final fused image Q be RESQH horizontally and RESQV vertically, then the coordinate range of the point is:

[0253] ,

[0254] ,

[0255]

[0256] RESQV =

[0257] The above gives the range of values ​​for Qx and Qy in the fused image;

[0258] Let's take the right-side camera as an example:

[0259] The final image from the right camera will be merged into a planar image. ( ), Using perspective transformation, the difference-corrected image coordinates corresponding to the pixel coordinates of the final fused image from each camera on the right are obtained. The formula is as follows:

[0260]

[0261] Will( ) and the homography matrix of each camera on the right Using perspective transformation, the distortion-corrected image coordinates corresponding to the difference-corrected image coordinates of each camera are obtained. The formula is as follows:

[0262]

[0263] Based on the distortion correction principle of fisheye cameras, obtain the coordinates of the distortion-corrected image. The coordinates of the corresponding original distorted image () , );

[0264] Repeat the above steps to obtain the original distorted image coordinates corresponding to different cameras. , ), where k is the camera number.

[0265] Based on the original distorted image ( , The corresponding image region is obtained, and the image region is divided into blocks to obtain the pixels of the final fused image from each camera k on the right. k( The corresponding sharpness score ( ), denoted as k .

[0266] Similarly, the final fused planar images from the rear and left side cameras are obtained. ( )and , ( )and By following the steps described above, the pixel points of the final fused image from each of the rear cameras can be obtained. k( The corresponding sharpness score ( ), denoted as k The final image from each camera on the left side is fused together, showing individual pixels. k( The corresponding sharpness score ( ), denoted as k .

[0267] Taking the pollution camera on the left as an example, the sharpness score corresponding to its final fused planar image. k According to threshold Judgment as Figure 13 As shown;

[0268] The weight of the camera area information at each installation location is the area with the highest probability of subjective observation by the driver in the fused image captured by the camera at its actual installation location, namely the area below the ground and vehicle body extension line at the rear and side rear. This area is obtained by installing a clean camera at the designated installation location and facing the designated field of view, thus obtaining the image of the camera on the fusion plane. Preferably, the selected area is... The following areas, such as Figure 14 Gray area.

[0269] According to the principle of perspective, the closer the texture is to the center region of the fused image, the greater its information effectiveness. A vertical information weighting function is then defined. Horizontal information weighting function Then the information weight function

[0270] Vertical information weighting function

[0271]

[0272] Horizontal information weighting function

[0273]

[0274] ofst_h1, ofst_h2, and low_val are all manually set. low_val is the subjectively perceived lowest weight, ofst_h1 is the subjectively recognized lowest weight region, and the region between ofst1 and ofst2 is the weight gradient region.

[0275] Information weight function :

[0276]

[0277] If we color the information weight 1 as pure white and 0 as pure black, then... like Figure 15 As shown

[0278] The current frame number of the single camera k in the final fused image. Corresponding information weights Score based on sharpness and regional information weight Joint decision:

[0279]

[0280] in, This indicates the number of the redundant camera; RESQH and RESQV represent the horizontal and vertical resolutions of the fused image, respectively. This represents the pixel coordinates of the fused image, and f represents the frame sequence number;

[0281] This represents the sharpness score of redundant camera k. This represents the weighting function for regional information in the merged image.

[0282] The camera with the highest information weight score among all redundant cameras within the same installation area is selected as the working camera, and the remaining cameras are designated as standby cameras. The fused image is the final planar fusion image from the working cameras in each installation area.

[0283] Specifically, the images from the cameras in each installation area corresponding to the final fusion plane have coverage areas on the final fusion plane. The images in these coverage areas are then weighted and fused. The pixel value weighting method is calculated based on information weights. The information weights of the left-side working camera, Score the clarity of the left-side working camera; The information weight of the rear working camera, The sharpness score is given to the rear working camera; The information weight of the working camera on the right. The sharpness score is given to the working camera on the right. The images are fused to represent the coverage areas of the right and rear working cameras. To fuse images covering the areas covered by the left and rear working cameras, These are images from the right, rear, and left working cameras on the final fusion plane, respectively.

[0284]

[0285]

[0286]

[0287]

[0288]

[0289] Based on the actual display effect, the texture area is manually defined. Finally, the pixels in that area of ​​the merged image are replaced with texture pixels.

[0290] Redundant cameras in various areas (uncontaminated and contaminated) such as Figure 16 As shown;

[0291] The final fused image from redundant cameras (uncontaminated) in each area, plus the texture effect, is as follows: Figure 17 As shown.

[0292] The above descriptions are merely preferred embodiments of the present invention, and the present invention is not limited to the above embodiments. Those skilled in the art will understand that the forms in these embodiments are not limited thereto, nor are the adjustments possible. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the basic concept of the present invention should be considered to be included within the scope of protection of the present invention.

Claims

1. A method for generating vehicle side and rear view images based on time synchronization, characterized in that, At least including: Step S1: Synchronize the image acquisition time of multiple vehicle-mounted Ethernet cameras and perform distortion correction on the acquired images; Step S2: Obtain images captured by redundant vehicle-mounted Ethernet cameras, and obtain a fused image after image fusion. Based on the original distorted image corresponding to the fused image, select the uncontaminated vehicle-mounted Ethernet camera as the working camera. Obtaining the corresponding original distorted image from the fused image includes at least the following: Step S201: Based on the pixel coordinates of the observed fused image and the pixel coordinates of the four vertices of the target image's rectangular ROI region, nameC(i)x, nameC(i)y, where name = right, rear, left; i = 1, 2, 3, 4, solve using perspective transformation. Obtain the homography matrix of multi-camera fused images. The difference-corrected image from multiple cameras is denoted as nameFC(i)x and nameFC(i)y, where name = right, rear, left; i = 1, 2, 3, 4. Step S202: Repeat step S201 to sequentially obtain the homography matrices corresponding to the right, rear, and left redundant cameras at different positions. Difference correction images from multiple cameras; Step S203: The final planar image from the camera is fused. ), Using perspective transformation, the difference-corrected image coordinates corresponding to the pixel coordinates of the final fused image from each camera are obtained. ); Step S204, difference-corrected image coordinates ( ) and the homography matrix of the corresponding camera Using perspective transformation, obtain the distortion-corrected image coordinates corresponding to the difference-corrected image coordinates of each camera. ); Step S205: Based on the distortion correction principle of fisheye cameras, obtain the coordinates of the distortion-corrected image (…). The coordinates of the corresponding original distorted image () , ); Step S206: Repeat steps S203 to S205 to obtain the original distorted image coordinates corresponding to different cameras. , ), where k is the camera number; Step S3: Obtain images captured by working cameras at different installation positions and perform differential homography matrix correction in distortion correction. Correction and fusion homography matrix The image captured by the working camera is mapped to obtain the image of the fusion plane corresponding to the working camera at the installation position. The difference-corrected homography matrix The correction fusion homography matrix is ​​used to correct image geometric deviations caused by individual differences in cameras. Used to project images from various cameras onto a unified fusion display plane; Step S4: The pixel values ​​of the overlapping areas of the images of working cameras at different installation positions on the fused image are used to fuse the images of the overlapping areas according to the information weights. The information weights are calculated based on the sharpness score of the corresponding area of ​​the image captured by the camera and a preset area information weight function. The region information weighting function is a preset function based on the position of the image region in the fused image and its importance to driving safety.

2. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, The vehicle-mounted Ethernet camera transmits image content and control information via the Ethernet EAVB protocol. It has a built-in RTC clock, and the intelligent host corrects the RTC clock of each vehicle-mounted Ethernet camera at regular intervals via the vehicle-mounted Ethernet bus.

3. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, The time synchronization of the image acquisition time of the vehicle-mounted Ethernet camera specifically includes: The vehicle-mounted Ethernet cameras calculate their respective link propagation delays with the main processor. ; When the main processor periodically sends local time messages to the vehicle Ethernet camera, the vehicle Ethernet camera parses the local timestamp tm recorded in the message, and then adds the link propagation delay. Get the local RTC timestamp tc of the main processor after the message is received; Meanwhile, the vehicle-mounted Ethernet cameras modify their local RTC clock timestamps to tc.

4. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, Distortion correction includes at least a first distortion correction, wherein the first distortion correction includes at least: acquiring a distorted image of a preset target image perpendicular to the ground through a redundant vehicle-mounted Ethernet camera, performing a first distortion correction on the distorted image using an optical distortion parameter mapping table, and then obtaining a first distortion-corrected image.

5. The method for generating vehicle side and rear view images based on time synchronization as described in claim 4, characterized in that, Distortion correction includes at least a second distortion correction, wherein the second distortion correction includes at least: establishing a difference correction plane coordinate system, obtaining the first coordinate points of the feature points of the preset target image corresponding to the first coordinate points in the first distortion-corrected image and the second coordinate points corresponding to the feature points in the difference correction plane, and solving the homography matrix using perspective transformation. Using homography matrix Perform an inverse transformation to perform a second distortion correction on the first distortion-corrected image; Obtain the homography matrix Then, by using inverse perspective transformation, all the first coordinate points corresponding to the difference correction plane are established as the second coordinate points in the first distortion correction image. The pixel values ​​of the second coordinate points in the first distortion correction image are obtained and the pixel values ​​are assigned to the first coordinate points.

6. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, In step S2, for redundant vehicle-mounted Ethernet cameras installed in the same location, the distorted images of the original cameras after synchronous acquisition are divided into several regions. The sharpness score of each region is calculated independently, and a set of sharpness scores for each region is obtained for each camera. Each redundant camera Each corresponds to a set of sharpness scores .

7. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, In step S2, for redundant vehicle Ethernet cameras installed in the same location, the corrected images from the vehicle Ethernet cameras are mapped onto the fused images of the actual display application, resulting in a corrected fusion homography matrix for the fused images corresponding to the installation location differences of each vehicle Ethernet camera. And set the regional information weighting function for the merged image. And based on the installation location, each redundant vehicle-mounted Ethernet camera Sharpness score and its corresponding difference-corrected homography matrix Obtain information weights Among the redundant cameras, the camera with the highest information weight is the working camera, and the rest are standby cameras.

8. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, The coverage area includes: a fused image of the coverage areas of the right and rear working cameras. Images fused from the coverage areas of the left and rear working cameras ; The fused image of the camera coverage area includes the weighted average of the images and information weights of the working cameras in the corresponding area on the final fusion plane. The fusion calculation method includes: Among them, the information weights of the left, rear, and right working cameras. These are images from the right, rear, and left working cameras on the final fusion plane, respectively.

9. A method for generating vehicle side and rear view images based on time synchronization according to claim 8, characterized in that, Information weights of the left, rear, and right working cameras The calculation formula is as follows: in The sharpness score is given to the rear working camera; The sharpness score is given to the working camera on the left. Score the clarity of the working camera on the right. This is the information weighting function.

10. The method for generating vehicle side and rear view images based on time synchronization as described in claim 6, characterized in that, The sharpness score is obtained by dividing the original distorted image into VNUM×HNUM blocks with a side length of b1. Calculate the sharpness score within each small block. The sharpness score is obtained by summing the sharpness scores within each small block of n consecutive frames of images. : in, This represents the current image frame number, RESH and RESV represent the horizontal and vertical resolutions of the image, respectively, and bl is the side length of the small block. , where HNUM = VNUM = .

11. The method for generating vehicle side and rear view images based on time synchronization as described in claim 10, characterized in that, Clarity score The calculation methods include: in, This represents a small block with vertical index v and horizontal index h, f is the frame number of the current image, bl is the side length of the small block, and i and j represent the pixel coordinates within the small block image, i∈[1,bl], j∈[1,bl]; Set the resolution score threshold , The settings are based on the bl size and the number of frames counted, n. The range is 8 to 12; Sharpness score threshold The sharpness score of each small block is compared with the sharpness score threshold. If the score is greater than or equal to the sharpness score threshold, the image sharpness of that area is considered to meet the requirements.

12. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, The acquisition of the fused image includes at least the following: Step S100: Obtain the pixel coordinates of the four vertices of the preset target image rectangle ROI region in the fused and corrected image, in a clockwise direction. , , , ; Step S101: Let the observation point be point O in the three-dimensional coordinate system, the projection of point O onto the horizontal plane G be g, point g be the origin of the three-dimensional coordinate system, and point g be located on the vertical center line of the rear camera image on the target image plane. Establish the target image coordinate system with point g as the origin. Step S102: Based on the corner point order in a clockwise direction, C1, C2, C3, and C4 respectively, solve the pixel coordinates nameFC(i)x and nameFC(i)y of the ROI region after the target image is captured by the right redundant camera, left redundant camera, and rear redundant camera in the target image coordinate system, with observation point O as the observation point. Name = right, rear, left; i = 1, 2, 3, 4. Step S103, let the pixel distance between the installation positions of the left and right cameras and their corresponding target images be... The pixel coordinates of the fused image of the ROI region after imaging by the right redundant camera, left redundant camera, and rear redundant camera, observed from the observation point O, are transformed into the vertex coordinates of the ROI region: nameC(i)x, nameC(i)y, where name=right,rear,left; i=1,2,3,4.

13. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, Information weight The calculation methods include: in, This indicates the number of the redundant camera; RESQH and RESQV represent the horizontal and vertical resolutions of the fused image, respectively. This represents the pixel coordinates of the fused image, and f represents the frame sequence number; This represents the sharpness score of redundant camera k. This represents the weighting function for regional information in the merged image.

14. The method for generating vehicle side and rear view images based on time synchronization as described in claim 1, characterized in that, Regional information weighting function Acquisition includes: Among them, the vertical information weight function Horizontal information weighting function ; in, The pixel horizontal resolution of the final merged image; Among them, ofst_h1, ofst_h2, and low_val are all manually set thresholds, low_val is the subjectively perceived lowest weight, and ofst_h1 is the subjectively determined lowest weight region. The threshold is set based on experience. This represents the point from the top edge of the image to the ROI of the target image captured by the right-hand camera. The vertical pixel distance.

15. The method for generating vehicle side and rear view images based on time synchronization as described in claim 4, characterized in that, in, The first distortion correction includes: Step S1: Obtain the optical distortion parameters of the vehicle-mounted Ethernet camera. The optical distortion parameters include a mapping table of the discrete field of view angle θ and its corresponding image height γ of the experimental module for image acquisition in the actual experimental environment under the specified CMOS chip conditions of the camera lens. Step S2: Based on the pinhole imaging principle, establish the spatial coordinate relationship between each pixel point of the actual shooting plane U and each point of the fisheye image on the imaging plane I, and establish the first mapping function between the discrete field of view angle θ and the object height λ. λ = tan(θ) Step S3: Based on the optical distortion parameter mapping table, construct the second mapping function θ=G(γ) between image height γ and discrete field of view angle θ. Step S4: Based on the imaging geometry, solve for the correction scaling factor between the actual imaging plane U and the corresponding corrected imaging plane and distortion imaging plane. and distortion scaling factor ; Step S5: Utilize the first mapping function, the second mapping function, and the corrected scaling factor. and distortion scaling factor Establish pixel-level coordinates of the corrected distortion-free image. pixel-level coordinates of the distorted image The corresponding third mapping function, and Assign the pixel value of the coordinate position to The coordinates represent the position, thus obtaining the image after distortion correction.

Images