Method for harmonizing images acquired from non overlapping camera views

Abstract

To provide a method for harmonizing images acquired by two or more cameras connected to a vehicle and having fields of view not overlapping each other, and a multi-camera vision system for a vehicle.SOLUTION: An image processing method for harmonizing images acquired by a first camera and a second camera connected to a vehicle and arranged in such a way as their fields of view cover the same road space at different times as the vehicle travels along a travel direction is disclosed. The method includes: acquiring, by a selected camera, a first image at a first time; selecting a first region of interest potentially bounding a road portion from the first image; sampling the first region of interest; acquiring, by the other camera, a second image in such a way that the potential road portion is included in a second region of interest; sampling the second region of interest; and determining one or more correction parameters for harmonizing images acquired by the first camera and the second camera based on a comparison of the image content between the first regions of interest and the second regions of interest.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method for harmonizing images acquired by two or more cameras connected to a vehicle and having non-overlapping fields of view. [Background technology]

[0002] It is known to provide vehicles with multi-camera automotive vision systems that include a number of cameras positioned in the front, rear and left and right side mirrors of the vehicle to capture images of the environment around the vehicle.

[0003] Images from these cameras are typically relayed to an electronic control unit (ECU) that includes a processor that processes the images, among other tasks, before providing one or more processed images to a display located within the vehicle's cabin or on the windshield to provide assistance to the vehicle driver.

[0004] Different types of fields of view can be generated by combining input images received from multiple cameras by a vehicle ECU before being displayed to the driver or used for other applications, such as automated or semi-automated vehicle navigation. In particular, regions of interest from input camera images can be first remapped to a target viewport and then merged to generate a mosaic image representing a view from a specific selected 3D point of the environment around the vehicle. For example, a virtual camera may be mounted above the vehicle with a central top-down view, and input camera textures corresponding to regions of interest in the camera images may be projected onto a surface viewport corresponding to a flat 2D plane and merged to generate a mosaic view showing the top surface of the vehicle and the environment around the vehicle, commonly referred to as a top-down view (or bird's-eye view). Other merged views, such as multidimensional views (e.g., a 3D bowl view in which a bowl-shaped 2D projection is used in addition to a flat 2D plane), can also be generated using the rendering capabilities of the vehicle ECU.

[0005] Each vehicle camera has its own lens, image sensor, and in many implementations, an independent image signal processing chain (so the vehicle cameras may have different exposure or gain controls, different white balances, etc.) Furthermore, each vehicle camera faces in a different direction and sees a different area of ​​the vehicle's environment.

[0006] Therefore, the brightness and hue captured by each vehicle camera may vary slightly due to the vehicle camera's constraints (lenses, image sensors, etc.) and different orientations. These slight differences in color and brightness adversely affect the visual quality of the displayed synthetic view and compromise the optical illusion of a view corresponding to the virtual camera at 3D points around the vehicle.

[0007] To improve the visual quality of the merged field of view, brightness and color matching are typically applied. In particular, matching between two vehicle cameras with overlapping fields of view is achieved by using the common ground area captured by the cameras as a basis for matching. For example, matching between a forward-facing camera and a right-side mirror camera can be achieved by using the road area at the corner where the fields of view of these cameras overlap.

[0008] For example, WO2018 / 087348 (Ref:SIE0426) discloses a method for matching the brightness and color of a composite image of the environment around a vehicle using histograms showing the luminance Y and color values ​​U, V of the merged input images.

[0009] It is further known for vehicles to be equipped with a hitch that allows the vehicle to tow a trailer. Because the trailer (including the object being transported by the trailer) prevents the driver from seeing most of the area around the trailer, the trailer presents a large blind spot, making it difficult for an untrained driver, especially, to reverse a vehicle equipped with a trailer without assistance. Accordingly, it is also known for such trailers to incorporate rear cameras pointed toward the rear of the trailer (and possibly trailer side cameras facing outward from each side of the trailer). Images from these cameras can be acquired by a controller within the trailer and provided to the vehicle ECU to generate an enhanced display to assist the driver. In some cases, any trailer camera can be directly connected to the vehicle ECU.

[0010] For example, the vehicle ECU can generate an invisible trailer view by merging images acquired by a vehicle rear camera and a trailer rear camera. Specifically, the invisible trailer view is constructed by remapping a region of interest in an image acquired by the rear camera onto a target viewport (an invisible trailer viewport) and merging the remapped region of interest. In this manner, a rear view in which the trailer is substantially invisible is provided to the driver via the vehicle display or windscreen. For example, WO 2021 / 032434 (Ref: 2019PF00307) discloses generating a first camera image from a vehicle rear camera and a second camera image from a trailer rear camera connected to the vehicle's trailer. The invisible trailer view is generated by overlapping these camera images such that the second camera image covers a portion of the first camera image depending on the hitch angle between the vehicle and the trailer.

[0011] Other applications may require merging images acquired from rear cameras on a vehicle and connected trailer. For example, DE102019133948 (Ref: V25-2103-19DE) discloses using multiple cameras on a vehicle-trailer combination to build a 3D view of the environment around the vehicle-trailer combination, which is displayed to the driver.

[0012] In a merged field of view created to assist the driver of a vehicle coupled to a trailer, brightness and / or color discrepancies may be visible to the driver in the merged region of images acquired by the rear cameras of the vehicle and the connected trailer. However, these cameras are positioned without overlapping such that their fields of view cover a common ground area. Therefore, there is no reliable common standard for harmonizing the rear cameras of the vehicle and the connected trailer.

[0013] Similarly, there is no common ground available as a reliable reference for harmonizing images acquired by the front and rear cameras of a vehicle (with or without a trailer). Summary of the Invention

[0014] According to the present invention there is provided a method for harmonizing images acquired by a first camera and a second camera connected to a vehicle and having simultaneously non-overlapping fields of view.

[0015] Embodiments of the present invention recognize that while the first and second cameras may not capture the same portion of the road at a given time, the first and second cameras may capture the same portion of the road at different times as the vehicle is traveling along its direction of travel, which may be advantageously used as a reliable indicator for reconciling fields of view that include a composite image captured by the first and second cameras. For example, the reconciled fields of view may be an invisible trailer field of view in embodiments where the first and second cameras are a rear camera on the vehicle and a rear camera on a trailer towed by the vehicle. In other embodiments, the first and second cameras may be front and rear cameras on the vehicle.

[0016] More particularly, embodiments of the invention involve sampling at least one first region of interest (ROI) from a first image acquired at a first time by one of a first camera and a second camera selected based on the determined orientation of the vehicle. The first ROI is defined in the first image to include a portion of a reference road within the imaged scene. A second ROI is sampled from a second image acquired by the other camera at a second time such that the second ROI also includes a portion of the reference road according to a distance traveled by the monitored vehicle after the first time. Based on a comparison between the image data within the sampled ROIs, one or more correction parameters are determined to harmonize the images acquired by the first and second cameras.

[0017] In some embodiments, the image data in the first and second ROIs are compared after being converted to YUV format. In these embodiments, the difference in estimated brightness values ​​Y for the first and second ROIs is compared to a threshold to determine whether the ROIs actually contain portions of the same reference road, based on the recognition that the difference in estimated brightness values ​​Y is significant if the ROIs contain different objects in the imaged scene (e.g., because the object has moved onto, beyond, or left the reference road portion prior to the time of acquisition of the second image).

[0018] Further aspects of the invention include a multi-camera vision system for an automobile, a vehicle and trailer combination or vehicle, and a computer program product configured to carry out the methods of the invention. [Brief explanation of the drawings]

[0019] Embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a vehicle and trailer combination including a multi-camera vision system according to the present invention; [Figure 2]2 illustrates a method according to the present invention that may be performed by the multi-camera vision system of FIG. 1; [Figure 3A] FIG. 1 is a diagram showing a vehicle moving forward. [Figure 3B] FIG. 1 is a diagram showing a vehicle moving forward. [Figure 4A] 3A and 3B show images acquired by the vehicle camera RV according to the operation of the method of FIG. 2 as the vehicle proceeds as shown in FIGS. 3A and 3B. [Figure 4B] 3A and 3B show images acquired by the trailer camera TR according to the operation of the method of FIG. 2 as the vehicle travels as shown in FIGS. 3A and 3B. [Figure 5A] 10 shows a vehicle traveling in the opposite direction. [Figure 5B] 10 shows a vehicle traveling in the opposite direction. [Figure 6A] 5A and 5B show images acquired by the trailer camera TR according to the operation of the method of FIG. 2 as the vehicle travels as shown in FIGS. 5A and 5B. [Figure 6B] 5A and 5B show images acquired by the vehicle camera RV according to the operation of the method of FIG. 2 as the vehicle proceeds as shown in FIGS. 5A and 5B. [Figure 7] 3 illustrates a displayed merged invisible trailer view harmonized by operation of the method of FIG. 2; Detailed Description of the Embodiments

[0020] Referring to FIG. 1, there is shown a vehicle 11 and trailer 12 combination including a multi-camera vision system configured to perform an image matching method in accordance with an embodiment of the present invention.

[0021] Vehicle 11 is provided with a hitch 13 that allows vehicle 11 to tow an object, such as a trailer 12 as shown in Figure 1. In particular, trailer 12 is hooked to hitch 13 via a drawbar 14 such that a hitch angle exists between vehicle 10 and trailer 12 (when vehicle 10 is towing trailer 12 that is moving along a curved trajectory).

[0022] It should be noted that trailer 12 shown in FIG. 1 is merely a non-limiting example of several types of trailers that may be towed by vehicle 10—other examples include caravans and horse-drawn carriages. Thus, any object towed by vehicle 10 is referred to herein as a trailer. Thus, if a trailer is used as a platform for transporting an object that obscures the driver's rearward vision (e.g., a boat trailer for carrying a boat or a bike trailer for carrying a motorcycle), for purposes of this application, the object being transported is treated as part of the towed trailer. It should also be noted that vehicle 11 shown in FIG. 1 is merely a non-limiting example of various types of vehicles (e.g., trucks, tractors, etc.) that may tow a trailer. Accordingly, the coupling means for connecting the trailer to the vehicle may be different from hitch 13 and drawbar 14 shown in FIG. 1.

[0023] A multi-camera vision system includes multiple cameras located at the front (FV camera), rear (RV camera), and left and right side mirrors (ML, MR cameras) of a vehicle to capture images of the environment around the vehicle. The side cameras do not need to be located on the mirrors; they can be located in any suitable location that can capture images from the environment to the sides of the vehicle.

[0024] The system further includes a trailer rear camera (TR camera) pointed toward the rear of the trailer 12 (and may optionally include side cameras pointing outward from each side of the trailer 12). Therefore, as shown in Figure 1, the fields of view FOV1 and FOV2 of the vehicle camera RV and trailer camera TR do not overlap at any given time in the common portion of the road 18 along which the vehicle 11 is traveling. This does not exclude that FOV1, FOV2 may overlap in some areas of the scene, but these common areas are not uniform but vary and therefore cannot be relied upon for image matching of the cameras RV and TR.

[0025] The system includes a vehicle ECU 15 running an application configured to receive images captured by the vehicle cameras FV, ​​RV, MR, ML, and a controller 16 in the trailer 12 configured to collect images captured by the trailer camera TR (and by the trailer side camera, if present). Images collected by the trailer controller 16 are provided to the vehicle ECU 15 via streaming or wireless or wired connection. In some cases, any trailer camera may be directly connected to the vehicle ECU 15.

[0026] A processor in vehicle ECU 15 is configured to process images acquired from vehicle and trailer cameras FV, ​​RV, MR, ML, TR in order to provide the processed images to a display 22 located within the cabin of vehicle 11 or on the windshield. Such camera information may also be processed by ECU 15 for autonomous or semi-autonomous driving, parking, or braking of the vehicle, and for storing a stream of images captured by one or more cameras, for example, as dash cam or security footage for later retrieval.

[0027] ECU 15 (or other processing unit within vehicle 11) can also estimate the distance traveled by vehicle 11 over time by processing sensor data provided by odometry sensors (shown schematically and cumulatively as 17 in FIG. 1 ). For example, odometry sensors 17 may include sensors that measure the space traveled by some of the wheel and steering angles. In addition to or instead of the sensor data provided by sensors 17, the change in position of vehicle 11 over time may be estimated using GPS tracking.

[0028] Referring to FIG. 2, an image reconciliation method 100 operable within the system is disclosed.

[0029] In method step 101, the heading of the vehicle 11 is determined, for example using the odometry sensor 17 and / or GPS position tracking information.

[0030] 3A , the determination of step 101 first considers the case where the vehicle 11 is moving forward along a substantially straight direction indicated by the longitudinal axis 20. In FIG. 3A , the field of view FOV1 of the RV camera covers a first road portion 50 and a second road portion 51 in the space between the vehicle 11 and the trailer 12, respectively, beside a first side 500 and a second side 510 of the trailer drawbar 14. In this manner, these road portions 50, 51 are not occluded by the trailer 12 or the drawbar 14 and can therefore be imaged by the RV camera and sampled as portions of a reliable reference road for purposes of matching cameras FV, ​​RV (which will be described in more detail). Note that in this application, a reference road portion includes not only a portion of paved road, but also any portion of ground where the vehicle may have a uniform color and texture (e.g., a portion of a highway, street, country road, or pitch).

[0031] In response to determining that vehicle 11 is traveling along the forward direction shown in Figure 3A, the RV camera is selected to capture a first image at a first time t1 corresponding to the position of vehicle 11 shown in Figure 3A (step 102). Figure 4A shows image 200 captured by the RV camera at t1.

[0032] Next, two ROIs 201, 202 are selected to be sampled from image 200 (step 103). In particular, ROIs 201, 202 are positioned and sized within acquired image 200 to correspond to portions 50 and 51, respectively, of the road beside drawbar 14.

[0033] Here, an example of a method for selecting two ROIs 201, 202 is disclosed.

[0034] When the vehicle ECU 15 receives the image 200 acquired at t1, the ECU 15 is configured to assume that the trailer 12 is substantially aligned with the vehicle 11 along its longitudinal axis and check two ROIs 201, 202 that are expected to include portions 500, 510 of the road beside the drawbar 14.

[0035] For example, the ECU 15 may be configured to define these ROIs 201, 202 by knowing the image area occupied by the trailer 12 and drawbar 14 when the trailer 12 is substantially aligned with the vehicle 11. In one implementation, the ECU 15 may learn this area by detecting the trailer 12 and drawbar 14 in a set of images captured by an RV camera and including the trailer 12 aligned with the vehicle 11. While this may provide a highly accurate ROI (location and size), it should be understood that this approach adds complexity to the implementation. Alternatively, the ECU 15 may estimate this area by knowing dimensional parameters of the vehicle 11 and drawbar 14 (e.g., at least the width of the vehicle 11 and the length of the drawbar 14). This information may be provided to the ECU 15 in a variety of ways, such as receiving this information from a user input, receiving scans of the trailer 12 and drawbar 14, or obtaining CAD data of the vehicle over a network connection or the like. In either case, the default ROI location can be determined from the known position of the camera RV on the vehicle 11 from the vehicle CAD and the known width of the vehicle 11 (which can also be obtained from the vehicle CAD). This dictates the minimum length of a suitable drawbar, which should be at least half the width of the vehicle. This allows the default location of the ROI to be determined with minimal user input and processing power.

[0036] The ECU 15 then determines whether either of the checked ROIs 201, 202 includes a portion of the drawbar 14 or the trailer 12 (due to the steering of the vehicle at the image acquisition time t1). In one embodiment, the ECU 15 applies image detection to the ROIs 201, 202 to detect whether either of the ROIs 201, 202 includes a portion of the drawbar 14 or the trailer 12. In another embodiment, the ECU 15 measures the steering angle of the vehicle 11 at the image acquisition time t1 using odometry data provided by the sensor 17 and / or GPS location information and compares the measured angle with a threshold. In response to determining that the measured steering angle is less than the threshold (including a null value), the ECU 15 determines that neither of the ROIs 201, 202 includes a portion of the drawbar 14 or the trailer 12. Additionally or alternatively, the ECU 15 can make a similar determination by using a measured hitch angle between the longitudinal axes of the vehicle 11 and the trailer 12. This angle may be measured in any number of ways, such as using image information from the captured image 200 to detect rotation of the trailer 12 about a longitudinal axis passing through the hitch 14. Similarly, image information from the vehicle mirror cameras ML, MR may detect features from the surface of the trailer moving laterally within their respective fields of view to estimate the relative angle between the vehicle and trailer. Other techniques for determining the relative angle between the vehicle and trailer include using information from a rear-facing ultrasonic or radar sensor located behind the vehicle 11, where a change in the difference measured by the sensor indicates a change in the relative angle between the vehicle 11 and the trailer 12.

[0037] 4A, the ECU 15 determines that none of the checked ROIs 201, 202 includes any portion of the drawbar 14 or the trailer 12. In response to this determination, the ROIs 201, 202 are selected to be sampled from the image 200 (step 103).

[0038] Further, from FIG. 4A, it is noted that the selected ROIs 201, 202 are defined to correspond to portions 50, 51 of the road spaced from either side 500, 510 of the drawbar 14 so as to be minimally affected, if at all, by shadows cast by the drawbar 14 (and by the trailer 12) at any time of day and in any lighting condition.

[0039] 3A , it should be further understood that if the vehicle 11 had been steered to the right at time t1, instead of moving forward along a straight trajectory, the steering angle may have been such that only a portion 50 of the road beside the left side 500 of the drawbar 14 was visible in the image captured by the RV camera at time t1. Similarly, if the vehicle had been steered to the left at time t1, the steering angle may have been such that only a portion 51 of the road beside the right side 510 of the drawbar 14 was visible in the image captured by the RV camera at time t1.

[0040] In these cases, method step 103 involves selecting only one of the ROIs 201, 202 that corresponds to the part of the road 50, 51 that can be imaged by the RV camera in the steering direction.

[0041] In other embodiments, when ECU 15 receives the image captured by the RV camera at t1, ECU 15 may perform detection of trailer 12 and drawbar 14 to determine the image area occupied by trailer 12 and drawbar 14, and select one or more ROIs 201, 202 around the detected area that may include portions 50, 51 of the road beside sides 500, 510 of drawbar 14, respectively. In some other embodiments, the selection of the ROI may be based on detection of portions of the road within the imaged scene, for example, using texture-oriented methods or by evaluating pixel intensity.

[0042] Furthermore, while the above-described embodiment is based on sampling portions 50, 51 of the road beside the drawbar 14 from the image acquired by the RV camera at acquisition time t1, it should be understood that, in addition or alternatively, a portion of the road seen within the field of view FOV1 of the RV camera beside the trailer 12 may also be sampled as a basis for image matching. In this case, the selection of the ROI should also take into account the shadow cast by the trailer onto the road 18 (as can be seen in FIG. 4A , the shadow cast by the trailer may cover part of the surrounding road depending on the orientation of the sun).

[0043] The description of method 100 continues with reference back to the case where two ROIs 201, 202 are selected in method step 103 to be sampled from image 200 shown in Figure 4A. Nevertheless, the following disclosure may also apply when only one of ROIs 201, 202 is selected (for steering) in method step 103.

[0044] The selected ROIs 201, 202 are sampled from the image 200 (step 104) and the image data for each is stored in the system's memory or in other storage means accessible by the system (e.g., a database or server accessible by the system via a network connection).

[0045] Next, in method step 105, the distance traveled by the vehicle 11 after the acquisition time t1 of the image 200 is monitored to determine a second time t2 at which to acquire a second image by the TR camera of the trailer 12, such that the same road portions 50, 51 corresponding to the ROIs 201, 202 sampled from the image 200 are included in the corresponding ROIs defined in the second image. The distance traveled may be monitored using odometry data provided by the sensors 17 and / or GPS distance traveled.

[0046] For example, Figure 3B shows that the vehicle 11 has moved further along the forward direction from the position shown in Figure 3A, covering a distance dx such that each of the road portions 50, 51 is visible within the field of view FOV2 of the TR camera (when not occluded by an object moving into the scene while moving the travel distance dx).

[0047] A time t2 corresponding to the traveled distance dx is determined, and an image 300 is captured by the TR camera at t2 (step 106). The captured image 300 is shown in Figure 4B and includes two ROIs 301 and 302 that are expected to include portions of the road 50 and 51 due to the traveled distance dx (in the example of Figure 4B, the ROIs 301 and 302 actually include the portions of the road 50 and 51 because no objects occlude them).

[0048] 3A-3B, if cameras RV and TR are at the same height above the road surface, at the same relative angle to the road surface, and have the same projection model, then the travel distance dx is approximately equal to the distance D between the RV camera and the TR camera (which substantially corresponds to the sum of the lengths of trailer 12 and drawbar 14). In this case, referring to FIGS. 4B-4A, ROIs 301 and 302 may be defined as pixel areas at locations in image 300 that substantially correspond to within the pixel areas of image 200 and ROIs 201 and 202. This improves comparability between image data in sampled ROIs 201 and 202 and image data in sampled ROIs 301 and 302. Therefore, it should be understood that if the relative positions, heights, and / or projection models of cameras RV and TR are different, then distance dx will change accordingly, and / or ROIs 201, 202 and 301, 302 will need to be mapped differently from each other.

[0049] In either case, the acquisition time t2 determined for the TR camera may correspond to a travel distance greater than or less than dx, as long as the road portion 50, 51 is still visible within the field of view FOV2 of the TR camera.

[0050] After acquiring image 300 at t2, ROIs 301, 302 are sampled (step 107) and the respective image data is stored in the system's memory (or other storage means accessible by the system).

[0051] Returning to initial method step 101, operation of method 100 commences in response to a determination that the direction of vehicle 11 is reversed.

[0052] 5A, the determination of step 101 considers the case where vehicle 11 is reversing along a substantially straight direction corresponding to longitudinal axis 30. In FIG. 5A, the TR camera's field of view FOV2 covers a portion of first road 60 and a portion of second road 61 within the space between vehicle 11 and trailer 12 that are visible within the field of view FOV1 of the RV camera at sides 500, 510 of drawbar 14 as vehicle 11 travels in the reverse direction (as shown in FIG. 5B). In this manner, road portions 60, 61 are no longer occluded by trailer 12 or drawbar 14 and are therefore sampled at different times by both the TR camera and the RV camera and can be used as a reliable reference for image matching.

[0053] In response to determining that the vehicle 11 is traveling along the reverse direction shown in Figure 5A, the TR camera is selected to capture a first image at a first time t1 corresponding to the position of the vehicle 11 shown in Figure 5A (step 108). Figure 6A shows an image 400 captured by the TR camera at t1.

[0054] Next, two ROIs 401, 402 are selected (step 109) to be sampled from the image 400. In particular, the ROIs 401, 402 are positioned and sized within the acquired image 400 to include portions 60, 61 of the road.

[0055] An exemplary method for selecting two ROIs 401, 402 will now be described.

[0056] When the vehicle ECU 15 receives the image 400 captured at time t1, the ECU 15 is configured to assume that the trailer 12 is substantially aligned with the vehicle 11 along its longitudinal axis and check for two ROIs 401, 402 corresponding to portions of the road visible by the RV cameras beside the sides 500, 510 of the drawbar 14. For example, the ECU 15 is configured to define these ROIs 401, 402 by knowing the image areas occupied by the trailer 12 and the trailer drawbar 14 when the trailer 12 is substantially aligned with the vehicle 11.

[0057] The ECU 15 then determines whether the vehicle 11 is reversing along a substantially straight trajectory. For example, the ECU 15 measures the steering angle of the vehicle 11 or the hitch angle between the vehicle 11 and the trailer 12 at the image acquisition time t1 using odometry data provided by the sensor 17 and / or GPS position information, and compares the measured angle with a threshold. In response to determining that the measured steering angle or hitch angle is a value less than the threshold (including a null value), the ECU 15 determines that the vehicle 11 is reversing along a straight direction. In response to this determination, the ECU selects two ROIs 401, 402 to be sampled from the image 400.

[0058] 5A , it should be appreciated that if, at t1, instead of reversing along a straight trajectory, vehicle 11 were to steer right while reversing, the steering angle would be such that only a portion 60 of the roadway would be visible within the field of view FOV1 of the RV camera next to side 500 of drawbar 14 as vehicle 11 travels in the opposite direction. Similarly, if, at t1, instead of reversing along a straight trajectory, vehicle 11 were to steer left while reversing, the steering angle would be such that only a portion 61 of the roadway would be visible within the field of view FOV1 of the RV camera next to side 510 of drawbar 14 as vehicle 11 travels in the opposite direction.

[0059] In these cases, method step 109 involves selecting only one of the ROIs 401, 402 that corresponds to the part of the road 60, 61 that can also be imaged by the RV camera in the steering direction.

[0060] The description of method 100 will continue to refer to the case where two ROIs 401, 402 are selected to be sampled from image 400 shown in Figure 6A in step 109. Nevertheless, the following disclosure also applies when only one ROI (due to steering) is selected to be sampled in method step 109.

[0061] ROIs 401, 402 are sampled from image 400 (step 110) and the image data for each is stored in memory within the system (or other storage means accessible by the system).

[0062] Next, in method step 111, the distance traveled by the vehicle 11 after the acquisition time t1 of the image 400 is monitored to determine a second time t2 at which a second image is acquired by the RV camera of the vehicle 11 so that the portions of the road 60, 61 corresponding to the ROIs 401, 402 sampled from the image 400 (acquired by the TR camera) are included in the corresponding ROIs defined in the second image.

[0063] For example, Figure 5B shows vehicle 11 moving further along the reverse direction from the position shown in Figure 5A, traveling a distance dx such that portions 60, 61 of the road are each visible within the field of view FOV1 of the RV camera (if not occluded by an object moving into the scene during the travel distance dx). A time t2 corresponding to the travel distance dx is determined, and image 600 is captured by the RV camera at t2 (step 112).

[0064] The acquired image 600 is shown in FIG. 6B and includes two ROIs 601, 602 near the imaged sides 500, 510 of the drawbar 14, which are expected to include road portions 60, 61 according to the travel distance dx (in the example of FIG. 6B, there are no objects occluding road portions 60, 61, so ROIs 601, 602 actually include these portions 60, 61).

[0065] 5A-5B, road portions 60, 61 are imaged by the TR camera at time t1 at approximately the same acquisition distance as the acquisition distance between the same road portions 60, 61 and the RV camera at time t2. In this manner, the likelihood of there being an object at time t1 occluding one of portions 60, 61 is reduced. Further, with reference to Figures 6A-6B, ROIs 401, 402 are defined to be pixel areas at locations in image 400 that substantially correspond to pixel areas of image 600 and ROIs 601, 602.

[0066] The method 100 continues by sampling the ROIs 601, 602 from the image 600 (step 113), and the image data for each is stored in the system's memory (or other storage means accessible by the system).

[0067] For the sake of brevity, the harmonization process performed by the method 100 will be disclosed with reference only to the ROIs 201, 202, 301, 302 sampled in steps 102-107 (following the forward determination in initial step 101) of the method 100. It should be noted that the principles of the present disclosure equally apply to the harmonization process performed based on the ROIs 401, 402, 601, 602 sampled in steps 108-113 (following the backward determination in initial step 101) of the method 100.

[0068] Image data of the sampled ROIs 201, 202 (extracted from the image 200 acquired by the RV camera at t1) and image data of the sampled ROIs 301, 302 (extracted from the image 300 acquired by the TR camera at t2) are retrieved from the system's memory (or other storage means accessible by the system) and provided to a harmonization network (implemented by the vehicle ECU 15 or other processing unit of the system), which converts the retrieved image data to YUV format if it has not already been converted (step 114).

[0069] Next, luminance components Y1 and Y2 are estimated from the pixel data of ROIs 201 and 202, and luminance components Y3 and Y4 are estimated from the pixel data of ROIs 301 and 302 (step 115). Various methods can be used to estimate Y1 to Y4. For example, several techniques for estimating Y1 to Y4 based on histograms generated to describe the luminance of ROIs 201, 202, 301, and 302 are disclosed in the above-mentioned WO2018 / 087348 (including a non-segmentation-based method, a histogram segmentation-based method, and a bi-modal histogram segmentation-based method).

[0070] Based on the understanding that the brightness of different imaged objects may differ significantly, the difference between the estimates Y1 and Y3 of the ROIs 201, 301 is compared to a threshold value (step 116) to determine whether both of these ROIs 201, 301 contain the same reference road portion 50.

[0071] Upon determining that the absolute values ​​of Y1-Y3 are less than the threshold, it is assumed that this small difference is due to a lack of brightness matching between the RV camera and the TR camera. Thus, the image data within ROIs 201, 301 are confirmed to belong to the same reference road portion 50.

[0072] In response to determining that the absolute values ​​of Y1-Y3 exceed a threshold, the image data within ROIs 201, 301 are determined to belong to different imaged objects. For example, this may be the case if an object (such as another vehicle or a person) moves into road portion 50 between acquisition times t1 and t2 of images 200, 300 from which ROIs 201, 301 are extracted. In another case, an object may cover road portion 50 at t1 and leave portion 50 between acquisition times t1-t2.

[0073] A similar test is performed to verify whether the two ROIs 202, 302 both contain the same reference road portion 51 by comparing the difference between Y2 and Y4 with a threshold (step 116).

[0074] In response to determining that the absolute value of at least one of the differences Y1-Y3 and Y2-Y4 is less than a threshold, such difference is used to determine a correction parameter that harmonizes the brightness of the images acquired by the RV camera of the vehicle 11 and the TR camera of the trailer 12 (step 117). Various methods can be applied to determine the brightness correction parameter based on the brightness difference value, such as the method described in WO2018 / 087348. Once determined, the brightness correction parameter can be stored in the system's memory (or any other storage means accessible by the system).

[0075] Furthermore, chromaticity values ​​U1, V1 and U2, V2 are estimated from the pixel data of ROIs 201, 201, and chromaticity values ​​U3, V3 and U4, V4 are estimated from the pixel data of ROIs 301, 302. The difference values ​​of U1-U3 and V1-V3 are used to determine correction parameters that match the colors of the images acquired by the RV and TR cameras (step 117). Various methods, such as those described in WO2018 / 087348, can be applied to determine color correction parameters based on the luminance difference values. Once determined, the color correction parameters can be stored in the system's memory (or other storage means accessible by the system).

[0076] In some embodiments, each of the differences U1-U3, V1-V3 is used to calculate the color correction parameters only if its value is verified to be less than a threshold value.

[0077] Furthermore, while the embodiments described above are based on comparisons between Y, U, and V values ​​estimated to describe the entire data content of ROI 201, 202, 301, and 302, in other embodiments, ROI 201, 202, and 301 may be divided into sub-regions where respective Y, U, and V values ​​are estimated and compared to determine harmony parameters. A sub-region may correspond to a single pixel or a group of pixels within ROI 201, 202, 301, and 302.

[0078] After calculating the Y, U, V correction parameters, method 100 may be re-executed at a later stage, starting again from step 101, to determine the orientation of vehicle 11. For example, the system may be configured to initiate the method periodically (and / or triggered by measured driving activity / environmental conditions), thus updating the stored harmonic correction parameters over time.

[0079] Referring back to step 116, method 100 may also be re-executed following a determination that both Y1-Y3 and Y2-Y4 have absolute values ​​that exceed the threshold (which determination may cause method 100 to be re-executed).

[0080] The determined harmonic correction parameters may then be retrieved by the system when required to be applied (step 118) in the process of generating a composite view including the combined images acquired by the RV and TR cameras, such as an invisible trailer view displayed on the main display 22 of the vehicle 11 or on the windshield providing a digital rearview mirror. In some embodiments, the harmonic correction parameters are applied to at least one of the images acquired by the RV and TR cameras before the images are combined into the composite view. In other embodiments, the correction parameters are applied to the composite view, particularly to the combined region between the images acquired by the RV and TR cameras.

[0081] 7 shows an example of a displayed image 700 of an invisible trailer field of view blended using correction parameters determined by operation of the disclosed method 100, in which an image 701 acquired by a TR camera is composited (without blending due to the spatial separation between the acquired images) with an image 702 acquired by an RV camera along line 703. The row of boxes at the bottom of the field of view corresponds to various buttons and widgets in the user interface of an application that controls the display and forms part of the vision system. If the display is equipped with a touchscreen, these widgets may be interactive, allowing the driver to directly select the function represented by each widget.

[0082] Other composite views may benefit from applying harmonic correction parameters obtained by operation of method 100 to provide other composite views, such as a top view of the environment around trailer 12, which may be displayed on display 22 and used to perform automatic or semi-automatic operations, or which may be stored and retrieved at a later stage (e.g., for investigation after an accident or after the contents of the trailer have been stolen).

[0083] Returning to step 116, if it is determined that both the Y1-Y3 and Y2-Y4 differences have absolute values ​​that exceed the threshold, the system does not have updated harmonization parameters for harmonizing the composite view. Thus, the system may determine (step 119) whether correction parameters previously generated and stored by operation of method 100 are available. In response to a positive determination, the system may apply the past correction parameters to harmonize the composite view (step 120). In response to a negative determination (e.g., because the method has only been performed one time or the past parameters are not retrievable), no harmonization is applied (step 121—and in this case, a negative determination may cause a re-execution of method 100).

[0084] Although implementation of method 100 has been disclosed for coordinating the RV and TR cameras of vehicle 11 and trailer 12, the same principles can be similarly applied to coordinating the forward FV camera and rearward RV camera of vehicle 11 (or other vehicles with or without trailers) based on sampling the same portion of road by the FV and RV cameras as vehicle 11 moves along its direction of travel and using the sampled portion of road as a common reference for coordinating.

Claims

1. An image processing method (100) for harmonizing images acquired by a first camera (RV, FV) and a second camera (TR, RV) connected to a vehicle (11), wherein their fields of view (FOV １ , FOV ２ ) are configured to cover the same road space (18) at different times when the vehicle moves along the traveling direction, Determining the traveling direction of the vehicle (101), Selecting one of the first camera and the second camera to acquire a first image (200, 400) based on the traveling direction, Acquiring the first image (200, 400) at a first time by the selected camera (102, 108), Selecting at least one first region of interest (201, 202; 401, 402) of the first image, wherein the first region of interest potentially surrounds a part of the road (50, 51; 60, 61) (103, 109), Sampling the at least one first region of interest (201, 202; 401, 402) from the first image (200, 400) (104, 110), Monitoring the distance traveled by the vehicle (11) along the traveling direction so that a part of the potential road (50, 51; 60, 61) is included in a second region of interest (301, 302; 601, 602) of the second image (300, 600) in order to determine a second time for acquiring a second image (300, 600) by the other camera after the first time (105, 111), Acquiring the second image (300, 600) at the second time by the other camera (106, 112), Sampling the second region of interest (301, 302; 601, 602) from the second image (300, 600) (107, 113), Checking (116) whether both the first region of interest and the second region of interest (201 - 301, 202 - 302; 401 - 601, 402 - 602) include a part of a road (50, 51; 60, 61), and Determining (117) one or more correction parameters to reconcile the images acquired by the first camera and the second camera based on a comparison of the image contents of the first region of interest and the second region of interest (201 - 301, 202 - 302; 401 - 601, 402 - 602) in response to the confirmation that both the first region of interest and the second region of interest include a part of the road, An image processing method (100) including the above. **Claim 2** The method according to claim 1, wherein the first camera is a rear camera (RV) of the vehicle (11), and the second camera is a rear camera (TR) of a trailer (12) connected to the vehicle via a connecting means (14). **Claim 3** Selecting the first camera (RV) such that the first image (200) is acquired at the first time (102) in response to determining that the vehicle (11) is moving forward, and each of the first region of interest or at least one first region of interest (201, 202) of the first image (200) corresponding to a region known to include a road space between the trailer (12) and the vehicle (11). The method according to claim 2. **Claim 4** The method according to claim 3, wherein each of the first region of interest or at least one first region of interest (201, 202) of the first image (200) is adjacent to a region surrounding the connecting means. **Claim 5** Selecting (103) the at least one first region of interest comprises Defining at least two first regions of interest (201, 202) in the first image (200), wherein when the trailer (12) is aligned with the vehicle (11) along the longitudinal axis (20), one (201) is on the first side (500) of the region surrounding the coupling means (14) and the other (201) is on the second side (510) of the region surrounding the coupling means (14). Determining whether any of the two first regions of interest (201, 202) includes a part of at least one of the trailer (12) and the coupling means (14). In response to a determination that neither of the two first regions of interest (201, 202) includes a part of at least one of the trailer (12) and the coupling means (14), selecting both of the two first regions of interest (201, 202) to be sampled from the first image (200) (103), and In response to a determination that any of the two first regions of interest (201, 202) includes a part of at least one of the trailer (12) and the coupling means (14), selecting the other first region of interest to be sampled from the first image (200) (103). The method according to claim 4, comprising the above steps.

6. Determining whether any of the two first regions of interest (201, 202) includes a part of at least one of the trailer (12) and the coupling means (14) comprises detecting whether a part of at least one of the trailer (12) and the coupling means (14) is included in any of the two first regions of interest (201, 202). The method according to claim 5.

7. Including measuring at least one of the steering angle of the vehicle (11) and the hitch angle between the vehicle (11) and the trailer (12). Determining whether any of the two first regions of interest (201, 202) includes at least a part of at least one of the trailer (12) and the coupling means (14) is made based on at least one of the measured steering angle and hitch angle, the method according to claim 5. **Claim 8** Selecting the second camera (TR) so as to acquire the first image (400) at the first time (108) in response to determining that the vehicle (11) is moving in the reverse direction, the method according to claim 2. **Claim 9** A part (60, 61) of the potential road corresponding to the at least one first region of interest (401, 402) of the first image (400) is within the field of view (FOV １ ) of the first camera (RV) when the vehicle (11) is moving in the reverse direction, the method according to claim 8. **Claim 10** Selecting the at least one first region of interest (401, 402) of the first image (400) Defining at least two regions of interest (401, 402), when the trailer (12) is aligned with the vehicle (11) along the longitudinal axis (30), one (401) is on the first side (500) of the region surrounding the coupling means (14) and includes a corresponding part (60) of the first road that comes to be seen within the field of view (FOV １ ) of the first camera (RV), and the other (402) is positioned on the second side (510) of the region surrounding the coupling means (14), defining, Measuring at least one of the steering angle of the vehicle (11) and the hitch angle between the vehicle (11) and the trailer (12), Selecting both of the two first regions of interest (401, 402) so as to be sampled from the first image (109) in response to determining that at least one of the measured steering angle and hitch angle is less than a threshold value, and In response to determining that at least one of the measured steering angle and hitch angle exceeds a threshold value, selecting (109) one of the two regions of interest (401, 402) based on at least one of the measured steering angle and hitch angle, The method according to claim 9, comprising.

11. The method according to claim 1, wherein the first camera and the second camera are a front camera (FV) and a rear camera (TR) of the vehicle (11).

12. Including converting (114) the image data in the first region of interest and the second region of interest (201-301, 202-302; 401-601, 402-602) into YUV format, Determining whether both the first region of interest and the second region of interest (201-301, 202-302; 401-601, 402-602) include a part (50, 51; 60, 61) of the road, Estimating (115) at least one first luminance component from at least a part of the image data in the first region of interest (201, 202; 401, 402) and at least one second luminance component from at least a part of the image data in the second region of interest (301, 302; 601, 602), and Comparing (116) the difference between the estimated first luminance component and the second luminance component with a threshold value, and In response to determining that the difference is less than the threshold value, confirming that both the first region of interest and the second region of interest (201-301, 202-302; 401-601, 402-602) include a part (50, 51; 60, 61) of the road, The method according to claim 1, comprising.

13. In response to the confirmation that the first region of interest and the second region of interest include a part of the road, Based on the difference between the estimated first region of interest and the second luminance component, determining one or more brightness correction parameters for harmonizing the brightness of the images acquired by the first camera and the second camera (RV, TR) (117). Estimating a first chrominance component from at least a part of the image data within the first region of interest (201, 202; 401, 402), and estimating a second chrominance component from at least a part of the image data within the second region of interest (301, 302; 601, 602), and Based on the difference between the estimated first chrominance component and the second chrominance component, determining one or more chrominance correction parameters for harmonizing the color of the images acquired by the first camera and the second camera (RV, TR) (117). The method according to claim 12, comprising:

14. Generating a composite view (700) including a synthetic image (701, 702) acquired by the first camera and the second camera (RV, TR), wherein the generating includes applying the determined one or more harmonization correction parameters (118). The method according to claim 1.

15. A first camera (RV, FV) and a second camera (TR, RV) connected to a vehicle (11), wherein at different times when the vehicle moves along the traveling direction, their fields of view (FOV １ , FOV ２ ) are configured to include the same road space (18), the first camera (RV, FV) and the second camera (TR, RV), and One or more processing units (15) configured to perform the method (100) according to claim 1. A vehicle multi-camera vision system including:

16. A combination of a vehicle (11) and a trailer (12), including the multi-camera vision system of claim 15, wherein the first camera is the rear camera (RV) of the vehicle (11), and the second camera is the rear camera (TR) of the trailer (12). A combination of a vehicle and a trailer.

17. When executing a computer program by the vehicle multi-camera vision system according to claim 15, a computer program having an instruction to cause the system to perform the method (100) according to claim 1.

Images