Device and method for displaying information

Abstract

A display method for use in a vehicle (100), the method comprising: Receiving a first image showing an area outside the vehicle (100) from a first image acquisition device (110); Capturing a hidden area of ​​the first image, for which part of the field of view of the first image is at least partially obscured; Identifying image data in a second image that corresponds to the obscured area; Generating a composite image from the first image and the identified image data; and Displaying at least a part of the composite image; where capturing a hidden area further includes: comparing the first image with another image obtained from the same image acquisition device (110), wherein the first and subsequent images are acquired at different times; and the detection of corresponding sections of the first image and the subsequent image that are the same, while other corresponding sections of the first image and the subsequent image differ.

Description

TECHNICAL AREA

[0001] The present invention relates to a device and a method for displaying information. In particular, but not exclusively, the present invention relates to a display method for use in a vehicle and a display device for use in a vehicle. Aspects of the invention relate to a display method, a computer program product, a display device or assembly, and a vehicle. BACKGROUND

[0002] WO 2015 123791 A1 describes technologies for generating a composite image. The methods may include receiving initial image data containing object data corresponding to an object, and receiving secondary image data containing obscuring data. The methods may also include displaying the composite image on a screen. These methods are described in the context of vehicle-mounted cameras.

[0003] Cameras mounted on or inside a vehicle and positioned to view an area outside the vehicle often have their view obscured by dust and dirt covering the camera lens or the window behind which the camera is located. This phenomenon is understandable, especially for a user who has driven the vehicle off-road. However, if the driver can no longer use the camera view because dirt is obscuring the lens, they are forced to either operate the vehicle without the cameras' useful functions or get out of the car to clean the lens. If this happens frequently, it quickly becomes a source of irritation for the user.

[0004] An example of a vehicle-mounted camera is a reversing camera. Typically, a single camera mounted at the rear of the vehicle is used to improve the driver's view of the area behind the vehicle when reversing. Due to the location of reversing cameras, dirt and water are frequently splashed or blown onto the lens during normal driving, necessitating regular lens cleaning. Another example of vehicle camera systems are surround-view camera systems. These systems capture image data from multiple cameras mounted around the vehicle and combine the views from these cameras to provide an essentially 360-degree view around the vehicle.These surround-view camera systems are particularly useful for helping a driver park a large vehicle in a tight parking space or when driving a vehicle off-road on difficult terrain. In such situations, it would be better if the driver could perform the desired maneuver inside the vehicle using the cameras, rather than having to stop the vehicle and clean the camera lenses before the maneuver can be completed.

[0005] It is known that the problem of hidden cameras is solved by using cleaning systems for the camera lenses or the windows behind which the cameras are located. Such cleaning systems involve wiping the lens or window with a rubber sheet or spraying a cleaning fluid or air when visibility is obstructed by dirt. However, such systems can be particularly difficult to integrate into a vehicle and contribute to the overall vehicle maintenance costs. An alternative known approach to addressing this problem is the use of hydrophobic coatings on camera lenses or windows to reduce buildup of dirt and dust. However, these coatings can deteriorate over the life of a vehicle and increase maintenance costs when they need to be reapplied.

[0006] The object of the present invention is to overcome at least some of the above-mentioned problems and to increase the advantages that vehicle-mounted camera systems can offer the driver.

[0007] It is the subject of embodiments of the invention to mitigate at least one or more of the problems of the prior art, including the aforementioned problems. It is the subject of certain embodiments of the invention to provide a method for the temporary compensation of a partially obscured camera lens on a vehicle. According to certain embodiments of the invention, this is achieved by using software to detect dirt or other material that obscures part of the camera's field of view and then filling in the obscured portion with historical image data or image data from another camera. SUMMARY OF THE INVENTION

[0008] Aspects and embodiments of the invention represent a display method, a computer program product, a display device and a vehicle according to the attached claims.

[0009] According to one aspect of the invention, a display method for use in a vehicle is provided, the method comprising: obtaining a first image showing an area outside the vehicle from a first image capture device; capturing a hidden area of ​​the first image for which part of the field of view of the first image is at least partially obscured; identifying image data in a second image corresponding to the hidden area; generating a composite image from the first image and the identified image data; and displaying at least part of the composite image.

[0010] Detecting a hidden area may involve comparing the first image with another image obtained from the same image acquisition device, with the first and subsequent images being captured at different times.

[0011] The detection of a hidden area may further include the detection of corresponding sections of the first image and the subsequent image that are the same, while other corresponding sections of the first image and the subsequent image differ.

[0012] Capturing a hidden area may also include defining a boundary that encompasses the corresponding sections of the first image and the subsequent image that are identical.

[0013] The second image can be obtained from the first image acquisition device or from a second image acquisition device with a different field of view relative to the first image acquisition device, with the first and second images being captured at different times.

[0014] The second image can be obtained from a second image acquisition device at the time the first image is taken, wherein the fields of view of the first and second image acquisition devices overlap and the overlap area at least partially covers the obscured area.

[0015] The first or second image capture device can be mounted on or inside the vehicle to capture images of the surroundings outside the vehicle.

[0016] The display procedure may further include: determining the vehicle's position at the time the first and second images were taken; and storing a display of the vehicle's position.

[0017] Creating a composite image can involve matching sections of the first image and the second image.

[0018] Matching sections of the first image and the second image can include matching overlapping sections of the first image and the second image.

[0019] Matching sections of the first image and the second image may involve performing a pattern fitting to identify features that are present in both the first and second images, with these features being correlated in the composite image.

[0020] The display procedure may further include determining a pattern recognition area within the first image, including the obscured area, and determining a second image, including image data for the environment within the pattern recognition area.

[0021] Determining a pattern recognition area may involve determining coordinates for the pattern recognition area according to a current position of the vehicle.

[0022] Determining a pattern recognition area may also involve receiving a signal indicating the vehicle's orientation and adjusting the pattern recognition area coordinates according to the vehicle's orientation.

[0023] The display procedure may further include: obtaining at least one image property for each of the first and second images; calculating an image correction factor as a function of the at least one image property for each of the first and second images; and adjusting the appearance of the first image or the second image according to the calculated image correction factor.

[0024] The at least one image property can refer to a property of the image, a setting of an image capture device used to capture the image, or an environmental factor at the time the image was captured.

[0025] Generating a composite image may also include displaying the portion of the composite image that corresponds to the hidden area.

[0026] The creation of a composite image may further involve the use of identified image data from the second image and at least one third image within the obscured area.

[0027] The display method may further include the storage of at least a predetermined number of images obtained from the first image acquisition device at different times.

[0028] According to another aspect of the invention, a computer program product is provided that stores computer program code which, when executed, is arranged to implement the above-mentioned method.

[0029] According to a further aspect of the invention, a display device for use with a vehicle is provided, comprising: a first image acquisition device arranged to obtain a first image showing an area outside the vehicle; a display device arranged to display a composite image; and a processing device arranged to: capture a hidden area of ​​the first image, for which a portion of the field of view of the first image acquisition device, which is at least partially obscured, is captured; identify image data in a second image corresponding to the hidden area; generate a composite image from the first image and the identified image data; and cause the display device to display at least a portion of the composite image.

[0030] A display device as described above, wherein the image acquisition device comprises a camera or other type of device arranged for generating and outputting still or moving images. The display means may include a screen, for example an LCD screen suitable for installation in a vehicle. Alternatively, the display may also include a projector for generating a projected image. The processing means may include a controller or processor suitable for vehicle control units.

[0031] The processing equipment can be further arranged to apply the above procedure.

[0032] According to another aspect of the invention, a vehicle is provided which includes the above display device.

[0033] According to a further aspect of the invention, a display method, a display device or a vehicle is provided, as essentially referred to herein. Fig. 10, Fig. 6 and Fig. 1 of the associated drawings described.

[0034] Within the scope of this application, it is expressly provided that the various aspects, embodiments, examples, and alternatives set forth in the preceding paragraphs, in the claims, and / or in the following descriptions and drawings, and in particular their individual features, may be adopted independently or in any combination. That is to say, all embodiments and / or features of an embodiment may be combined in any way and / or combination, unless these features are incompatible. The applicant reserves the right to amend an originally filed claim or to file a new claim accordingly, including the right to amend an originally filed claim to be dependent on another claim and / or to include a feature of another claim, even if it was not originally claimed in this manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] One or more embodiments of the invention will now be described only by way of example, with reference to the accompanying persons in which: Fig. Figure 1 illustrates a section of a vehicle with a vehicle-mounted camera system; Fig. Figure 2 illustrates a composite 3D image derived from vehicle-mounted cameras; Fig. Figure 3 illustrates a composite 2D image, particularly from a bird's-eye view, taken from vehicle-mounted cameras; Fig. Figure 4 illustrates a method for providing a composite image; Fig. Figure 5 illustrates a composite 2D image, particularly from a bird's-eye view, showing the tracking of a moving vehicle; Fig. Figure 6 illustrates a device for implementing the method of Fig. 4; Fig. Figure 7 illustrates a system for creating a composite image, including the adjustment of image properties; Fig. Figure 8 illustrates a procedure for generating a composite image, including adjusting the image properties; Fig. Figure 9 illustrates a field of view from a first image acquisition device that includes obscured areas; and Fig. Figure 10 illustrates a method for forming a composite image to compensate for hidden image areas according to an embodiment of the invention. DETAILED DESCRIPTION

[0036] It is becoming increasingly common for vehicles to be equipped with one or more video cameras to provide live video images (or still images) of the vehicle's surroundings. The vehicle can be a land vehicle, such as a wheeled vehicle, with a display device for showing the captured images. The display device can be a head-up display for showing information in a head-up arrangement for at least the driver of the vehicle, or another form of display device such as a screen or a projection device. The projection device can be arranged to project an image onto an interior part of the vehicle, such as a dashboard, door interior, or other interior parts of the vehicle.In the following discussion, whenever reference is made to the driver's view or the driver's position, this should be understood as the view of a passenger, although in manually propelled vehicles the driver's view is clearly of paramount importance. These images can then be displayed for the driver's benefit, e.g., on a screen mounted on the dashboard. In particular, it is known that at least one camera is directed towards the rear of the vehicle, generally behind the vehicle and downwards, to provide live video images to assist a driver reversing (where the driver's natural view of the area immediately behind the vehicle is particularly limited). It is further known that several such camera systems are provided to deliver live images of the area surrounding the vehicle on multiple sides, e.g., on a screen mounted on the dashboard.For example, a driver can selectively display different camera images to determine the positions of objects near each side of the vehicle.

[0037] These cameras can be positioned externally and mounted on the outside of the vehicle, or viewed from inside through the windshield or another vehicle window to capture images, with their lenses facing outwards and downwards. Such cameras can be positioned at various heights, for example, generally at roof level, driver's eye level, or at a suitable lower position to avoid the vehicle body obstructing their view of the area immediately beside the vehicle. Fig. Figure 1 shows an example of a camera mounted on the front of the vehicle, looking forward – in the direction the vehicle is traveling. If the relevant images are located on a different side of the vehicle, then the camera position is clearly set accordingly.

[0038] As in Fig. Figure 1, which illustrates a side view of the front section of a vehicle 100, shows a camera 110 mounted at the front of the vehicle below the plane of a hood or bonnet 105, such as behind or above a grille of the vehicle 100. Alternatively or additionally, a camera 120 can be positioned above the plane of the hood 105, for example at roof level or in an upper area of ​​the vehicle's windshield. It should be noted that, alternatively, cameras can be provided at the rear of a vehicle to assist a driver when reversing or performing other maneuvers, or in other positions for situations generally focused on one side of the vehicle.The field of view of each camera can generally be directed outwards and downwards in relation to the vehicle's passenger compartment to provide image data for a portion of the vehicle's surroundings, including the ground adjacent to the vehicle. It is noted that the camera can be mounted in other locations and be movable, for example, to rotate around an axis, allowing the camera's viewing angle to be controlled vertically or horizontally.

[0039] Advantageously, image data from multiple vehicle-mounted cameras can be combined into a single image, thus expanding the driver's field of vision. This can be used to address the problem that it can be difficult for a driver to determine the vehicle's position relative to objects beneath it. A method is now described by which a driver can visualize the terrain beneath a vehicle using historical (i.e., time-delayed) video footage from a vehicle-mounted camera system, such as the one described in Fig. The vehicle camera system shown in Figure 1 can be viewed. A suitable vehicle camera system comprises one or more video cameras positioned on a vehicle to record video images of the vehicle's surroundings, which can be displayed to assist the driver.

[0040] It is known that video images (or still images) are captured by several vehicle-mounted cameras and form a composite image depicting the environment around the vehicle. With reference to Fig. Figure 2 schematically illustrates such a composite image around a vehicle 200. In particular, the multiple images can be combined to form a three-dimensional (3D) composite image, which, for example, can generally be hemispherical, as shown in Overview 202. This combination of images can be referred to as stitching. The images can be still images or live video images. The composite image is created by mapping the images received from each camera onto a suitable part of the hemisphere. With a sufficient number of cameras and their suitable placement on the vehicle 200 to ensure appropriate fields of view, it should be noted that the composite image can thus extend around the vehicle 200 and from the bottom edge of the vehicle 200 on all sides up to a predetermined horizon height, which is represented by the top edge 204 of hemisphere 202.It should be noted that it is not absolutely necessary for the composite image to cover the entire area of ​​vehicle 200. For example, under certain circumstances it may be desirable to stitch together only camera images that generally project in the direction of movement of vehicle 200 and to both sides—directions in which vehicle 200 can be driven. This hemispherical composite image can be referred to as a shell. Of course, the composite image cannot be mapped onto a precise hemisphere, as the images of the composite image can extend higher or lower, or even over the top of vehicle 200, essentially forming a composite image sphere. It should be noted that, alternatively, the images can be mapped onto any 3D shape surrounding vehicle 200, e.g.,a cube, a cylinder, or a more complex geometric shape, which can be determined by the number and position of the cameras. It will be appreciated that the size of the composite image is determined by the number of cameras and their camera angles. The composite image can be formed by appropriately scaling and / or stretching the images derived from each camera to fit together seamlessly (however, gaps may occur in some cases if the captured images do not provide a 360-degree view around the vehicle).

[0041] The composite image can be displayed to the user using any suitable display means, such as the head-up display described above, projection systems, or dashboard-mounted display systems. While it may be desirable to display at least a portion of the 3D composite image, viewed, for example, from an internal position in a selected viewing direction, a two-dimensional (2D) representation of a portion of the 3D composite image can optionally be displayed. Alternatively, a composite 3D image may never be generated—the video images derived from the cameras are simply mapped onto a 2D longitudinal view of the environment around the vehicle 200. This could be a side view extending from the vehicle 200 or a top view, as shown in Fig. 3 is shown.

[0042] Fig. Figure 3 shows a composite image 301 from a bird's-eye view of the vehicle's surroundings, generally displayed at 300, also referred to as a top view. Such a top view can easily be displayed on a conventional screen inside the vehicle 300 and provides the driver with useful information about the environment around the vehicle 300, extending almost to the sides of the vehicle 300. From the driving position, it is difficult or impossible to see the ground immediately beside the vehicle 300, so the top view from Fig. 3 is an important aid for the driver. The ground under the vehicle 300 remains obscured by a live camera view and can therefore typically be represented in the composite image 301 by a blank area 302 at the position of the vehicle 300 or by a representation of a vehicle to fill the blank. Without providing cameras under the vehicle 300, which, as explained above, is not possible for a composite image formed solely from stitched live camera images, the ground under the vehicle 300 is not visible.

[0043] In addition to the cameras used to provide a composite live image of the area around vehicle 300, historical images can be integrated into the composite image to provide images depicting the terrain beneath the vehicle—that is, the terrain within the boundaries of vehicle 300. Historical images are images previously captured by the vehicle's camera system, for example, images of the ground in front of or behind vehicle 300; the vehicle subsequently drove over this area. The historical images can be still images, video images, or frames from video images. Such historical images can be used to map the empty area 302 in Fig. 3. It is recognized that, particularly in off-road situations, the ability to see the terrain in the area under the vehicle (strictly speaking, a representation of the terrain derived from historical images taken before the terrain was obscured by the vehicle) enables the driver to fine-tune the vehicle's position and, in particular, the vehicle's wheels, and increases the driver's confidence in controlling the vehicle.

[0044] The composite image can be created by combining live and historical video images, and in particular by pattern matching to the live images, thereby filling the blind spot in the composite image, which encompasses the area beneath the vehicle. The surround camera system comprises at least one camera and a buffer positioned to store images as the vehicle travels along a route. The vehicle's path can be determined by any suitable means, including, but not limited to, a satellite positioning system such as GPS (Global Positioning System), IMU (Inertial Measurement Unit), wheel tracking (tracking the rotation of the wheels combined with knowledge of the wheel circumference), and image processing to determine movement based on the shift of images between frames.In areas where the blind spot from the live images overlaps with buffered images, the area of ​​the blind spot copied from delayed video images is pattern-matched by image processing to be combined with the live camera images that form the rest of the composite image.

[0045] With reference to Fig. Figure 4 illustrates a method for creating a composite image from live video images and historical video images. In step 400, live video images are acquired from the vehicle's camera system. The image data can be provided by one or more cameras whose field of view, as previously explained, is directed outwards from the vehicle. In particular, one or more cameras can be positioned in front of or behind the vehicle at a viewing point at a predetermined distance in front of the vehicle, generally facing downwards. It should be noted that these cameras can be positioned appropriately to capture images of ground areas that may later be obscured by the vehicle.

[0046] In step 402, the live images are combined into a composite 3D image (e.g., the one above in conjunction with Fig. 4 described image “shell”) or a composite 2D image. Suitable techniques for combining video images are known to those skilled in the art. It is understood that the composite image can be formed continuously according to the 3D images or processed on a frame-by-frame basis. Each frame, or perhaps only a subset of the frames, such as every nth frame for at least some of the cameras, is stored to insert historical images into an empty area of ​​the currently displayed composite image (this can be referred to as a live blind spot). For example, if the composite image 301 is a bird's-eye view from Fig. 3, which is displayed on a screen, is the area of ​​the living blind spot, section 302.

[0047] To limit image storage requirements, only video images from cameras generally facing forward (or forward and backward) may be stored, as it is only necessary to store images of the ground in front of the vehicle (or in front of and behind the vehicle) that the vehicle may later drive over, in order to provide historical images for insertion into the live blind spot. To further reduce storage requirements, not every single image section may be stored in its entirety.For a sufficiently fast stored frame rate (or slow drive speed), significant overlap can occur between successive frames (or intermittent frames designated for storage if only every nth frame is to be stored), so that only a portion of the image differing from one frame to the next can be stored, along with sufficient information to combine this portion with the preceding frame. Such an image portion can be referred to as a band or image segment. It should be noted that, apart from an initially stored frame, each stored frame can require the storage of only one band. It may be desirable to store an entire frame periodically to minimize the risk of processing errors that prevent images from being recovered from stored image strips.This identification of overlapping areas between images can be performed using suitable, well-known image processing techniques, which may also include pattern matching, i.e., matching image sections common to a pair of frames to be stored. For example, pattern matching can use known image processing algorithms for detecting edge features in images, which can therefore adequately identify the outline of objects in images. These outlines are then identified in an image pair to determine the degree of image displacement between the pair due to vehicle movement.

[0048] Each stored frame or subframe (or image strip) is stored in combination with vehicle position data. Therefore, in parallel with the acquisition of live images at step 400 and the live image processing at step 402, vehicle position data is received at step 404. The vehicle position data is used to determine the vehicle location at step 406. The vehicle position can be expressed as coordinates, for example, as Cartesian coordinates specifying the X, Y, and Z positions. The vehicle position can be absolute or relative to a predefined point. The vehicle position data can be obtained from any suitable known position sensor, e.g., GPS, IMU, knowledge of the vehicle's steering position and wheel speed, wheel ticks (i.e., information about wheel rotation speeds), image processing, or any other suitable technique.Image processing can include processing images derived from vehicle camera systems to determine the degree of overlap between captured images. These images are then appropriately processed to determine a distance, calculated by knowing the time between each image capture. This can be combined with image processing for storing the captured images as described above, e.g., pattern matching including edge detection. In some cases, it may be desirable to compute a vector indicating the vehicle's movement and position to determine the historical images to be inserted into the living blind spot area, as described below.

[0049] Each frame (or strip) to be stored from step 400 is saved in a frame buffer from step 408, along with the vehicle position obtained from step 406 at the time of image capture. That is, each frame is stored indexed by a vehicle position. The position can be absolute or relative to a reference point. Furthermore, the frame position can only be specified relative to a previously stored frame, so that the vehicle's position relative to any historical frame can be determined relative to the vehicle's current position by going backward through the frame buffer and observing the shift in the vehicle's position until the desired historical frame is reached. Each record in the frame buffer can include image data for that frame (or strip) and the vehicle position at the time the frame was captured.This means that metadata, including the vehicle position, can be stored along with the image data. The frame's viewing angle relative to the vehicle position is known from the camera position and the angle on the vehicle (which, as explained above, can be fixed or movable). Such information about viewing angle, camera position, etc., can also be stored in the image memory 408, which represents the image and coordinate information as (Frame <-> Coord). It should be noted that there can be significant differences in the format in which this information is stored, and that the techniques disclosed herein are not limited to specific image data or metadata storage techniques, or to the details of the stored position information.

[0050] In step 410, a pattern recognition area is determined. The pattern recognition area encompasses the area beneath the vehicle that is not visible in the composite image, which consists solely of stitched live images. With reference to Fig. 3. The pattern recognition area encompasses the blind spot 302. Coordinates for the pattern recognition area can be determined from the vehicle positioning information obtained in step 404 and processed to determine the current vehicle position in step 406. Assuming that highly accurate vehicle positioning information obtained in step 404 is estimated, the current position of the vehicle can be accurately determined. Historical image data from the image memory 408, i.e., previously acquired images, can be used to fill in the blind spot 302, based on knowledge of the vehicle position at the time the historical images were acquired. In particular, the current blind spot can be mapped onto a ground surface that is visible in historical images before the vehicle obscured that part of the ground.Historical image data can be used in any camera image because the position and angle of each camera on the vehicle are known. Therefore, if the current vehicle position is known, image data showing the ground in the blind spot can be extracted from images taken earlier, before the vehicle obscured that part of the ground. This image data can be processed to match the current blind spot and inserted into the stitched live images. Such processing can involve scaling and stretching the stored image data to account for changes in perspective from the outward-facing camera angle, such as how the ground would appear from a direct overhead view. Furthermore, such processing can also involve recombining multiple stored image fragments and / or images from multiple cameras.

[0051] The adaptation of previously stored image data into a live-generated composite image described above relies on precise knowledge of the vehicle's position, both at the time of data storage and during the image data capture process. It may be impossible to determine the vehicle's position with sufficient accuracy. For example, with reference to... Fig. 3. The actual current position of the vehicle (generally specified at 300) is represented by field 302, while due to inaccurate position information, the current vehicle position determined in step 406 may be represented by field 304. In the example of Fig. 3. The inaccuracy includes the determined vehicle position, which is rotated relative to the actual vehicle position. Translation errors can also occur. Errors in calculating the vehicle position can arise from wheel slippage, where wheel ticks, wheel speed, and / or steering input are used to determine relative changes in vehicle position. When using satellite positioning, the required accuracy may not be available.

[0052] It should be noted that if the degree of error in the vehicle's position differs between the time an image is saved and the time it is integrated into a live composite image, this can lead to an undesirable misalignment of the live and historical images. This can cause a driver to lose confidence in the accuracy of the representation of the ground beneath the vehicle. Even worse, if the misalignment is significant, there is a risk of vehicle damage, as the driver will be incorrectly informed about the position of objects under the vehicle.

[0053] Due to the risk of misalignment, a pattern matching is performed within the pattern recognition area in step 412 to match areas of live and stored images. As mentioned above in connection with the storage of single images, such pattern recognition may include suitable edge detection algorithms. The pattern recognition area determined in step 410 is used to access stored images from image memory 408. In particular, historical images with image data for the ground within the pattern recognition area are retrieved. The pattern recognition area may include the expected blind spot of the vehicle and a suitable overlap on at least one side to account for misalignments. Step 412 takes as input the live stitched composite image from step 402. The pattern recognition area may include sections of the live composite view adjacent to the blind spot 302.Pattern matching is performed to find overlapping sections of the live and historical images, allowing for a close alignment between the two. This alignment can then be used to select suitable sections of the historical image to fill the blind spot. It's important to note that the degree of overlap between the live and historical images can be chosen to allow for a predetermined degree of error between the determined vehicle position and its actual position. To account for potential changes in vehicle tilt and roll between a current position and a historical position when traversing undulating terrain, the pattern recognition area determination can also incorporate information from sensor data indicating vehicle tilt and roll.This can affect the degree of overlap between the pattern recognition area and the live images for one or more sides of the vehicle. It may not always be necessary to define a pattern recognition area; rather, pattern matching can involve a more comprehensive search through historical images (or historical images with an approximate time delay relative to the current images) relative to the total composite live image. However, by limiting the area within the live composite image where pattern matching to historical images is to be performed, and by limiting the volume of historical images to be matched, the computational complexity of the task and the time required can be reduced.

[0054] In step 414, selected sections of one or more historical images or shards are inserted into the blind spot of the composite live images to form a composite image that includes both live and historical images.

[0055] In addition to displaying a representation of the ground beneath the vehicle, a representation of the vehicle itself can be added to the output composite image. For example, a transparent vehicle image or an outline of the vehicle can be added. This can help a driver identify the vehicle's position and the portion of the image that represents the ground beneath it.

[0056] If the composite image is to be displayed over horizontal parts of the vehicle to create the impression that the vehicle is transparent or translucent (e.g., using a HUD or projection device as described above), generating the composite image may also require determining the driver's viewing direction. For example, a camera is positioned to provide image data of the driver, from which the driver's viewing direction is determined. The viewing direction can be determined from the driver's eye position, which is carried out concurrently with the other steps of the procedure. It should be noted that if the composite image, or a portion thereof, is to be displayed on a screen in the vehicle that is not intended to make the vehicle appear transparent, there is no need to determine the driver's viewing direction.

[0057] The combined composite image is output at step 416. As explained above, the composite image output can be on any suitable image display device, such as a head-up display (HUD), a dashboard-mounted screen, or a separate display device worn by the driver. Alternatively, portions of the composite image can be projected onto parts of the vehicle interior to create the impression that the vehicle is transparent or translucent. The techniques disclosed herein are not limited to any particular type of display technology.

[0058] With reference to Fig. Figure 5 illustrates the progression of live images 502 and historical images 503 during the movement of a vehicle 500. In the example of Fig. 5 The composite image 501 is shown as a bird's-eye view above the vehicle and includes an 11 m long shell surrounding the vehicle 500, with the blind spot under the vehicle 500 being filled with historical images 503. Fig. Figure 5 shows the composite image 501, which tracks the vehicle position as it turns first to the right and then to the left (in the view from Fig. 5 from bottom to top), with the outlines of each composite image displayed within a border. The vehicle's current location is shown shaded. As already mentioned, the techniques disclosed herein are not limited to presenting the driver with a composite top view of the vehicle, its surroundings, and the ground beneath the vehicle. A 3D representation can be provided, or any 2D representation derived from any part of a 3D model, such as the one shown in Fig. 2 shown, is derived and viewed from any angle inside or outside the vehicle.

[0059] Fig. Figure 6 illustrates a device used to carry out the procedure of Fig. 4 is suitable. The device can be completely housed within a vehicle. One or more vehicle-mounted camera(s) 610 (e.g., those from Fig. 1) Captures image frames used to create a live section of a composite image, a historical section of a composite image, or both. It should be noted that separate cameras can be used to provide live and historical images, or that their roles can be combined. One or more position or motion sensors 602 can be used to detect the vehicle's position or movement. Camera 610 and sensor 602 provide data to processor 604 and are controlled by processor 604. Processor 604 buffers images from camera 610 in buffer 606. Processor 604 also creates a composite image containing live images from camera 610 and historical images from buffer 606. Processor 604 controls a display 608 to display the composite image. It should be noted that the device of Fig. 6 into the vehicle of Fig. 1 can be integrated, with the camera 610 being provided by one or more of the cameras 110, 120. The display 608 is typically located in the vehicle passenger compartment or cabin and can, as described above, be in the form of a dashboard-mounted display or another suitable type. Some parts of the image processing can be performed by systems outside the vehicle.

[0060] As described above, creating a composite image involves combining live and historical images derived from a vehicle camera system. This might be necessary, for example, if a composite image provides a wider field of view than can be achieved with live images alone, such as images of parts or areas of the environment below the vehicle or otherwise not visible in live images because that area is obscured by the vehicle itself. As described above, combining live and historical images may require precise tracking of the vehicle's position at the time each image (including live images) was captured. Alternatively, matching portions (e.g., pattern matching) of live and historical images can be used to determine which portions of historical images should be combined into a composite image.Optionally, both techniques can be used as in . Fig. The methods described in section 4 can be used in conjunction with determining a pattern recognition area. The first approach requires that vehicle position data be stored and linked to stored images.

[0061] Integrating historical images into a composite image with live images, as described above, may require some degree of image processing to scale and stretch historical and / or live images so they fit together smoothly. This may also apply to live images. However, the composite image may still appear disjointed. For an improved user experience, it may be desirable for the composite image to have a uniform appearance, giving the impression of having been captured by a single camera, or at least to minimize jarring differences between live and historical images. Such discontinuities may include color and exposure inconsistencies.If left uncorrected, composite image errors can cause users to lose confidence in the accuracy of the information presented in a composite image or to assume a malfunction. It may be desirable for composite images to be captured by a single camera to make the appearance of the vehicle or a part of the vehicle transparent or translucent.

[0062] Such discrepancies can arise from changes in ambient light conditions between the time the live and historical images are captured (especially when a vehicle is moving slowly), and from changes in the captured image properties resulting from different camera positions (when multiple cameras are used and live or historical images are captured from different camera positions and stitched together), or from seemingly different camera positions after stretching and scaling historical images. Furthermore, the problem of differences in image properties between live and historical images can be exacerbated by the large field of view of cameras used in vehicle camera systems. Multiple light sources around the vehicle, or changes in light sources between live and historical images, can cause further discrepancies.For example, at night, parts of a live image might be illuminated by headlights. Historical images also include areas that were illuminated by headlights at the time the image was captured. However, for the vehicle's current position, this part of the environment (e.g., under the vehicle) would no longer be illuminated by the vehicle's headlights. This can advantageously prevent the impression that areas under the vehicle are directly illuminated by headlights in a composite image by ensuring that the image properties for historical image areas match the image properties for adjacent live images.

[0063] Discontinuities between live and historical images can be mitigated by adjusting the image properties of the historical images, the live images, or both, to ensure a higher degree of consistency. This can result in a seemingly seamless composite image, or at least a composite image where such seams are less noticeable. In some cases, it may be preferable to adjust the image properties of only historical images, or to adjust the image properties of historical images more extensively than those of live images. This may be because live images, by their very nature, may include or resemble regions that can be directly observed by the driver, and it may be desirable to avoid the composite image appearing dissimilar to the directly observed environment.

[0064] If images (or image fragments) are stored in a single-frame buffer or single-frame memory, as described above in conjunction with step 408 of Fig. As described in section 4, the settings applied to the video image and / or the captured image properties can also be saved. This can be done alongside the saved coordinate information. For example, settings and captured image properties can be stored in an embedded image dataset, as image metadata, or separately and correlated within the saved images. When combining live and historical images into a composite image, the image properties or setting information between the live and historical images can be compared. The image properties of historical images, live images, or both can be adjusted to reduce the occurrence of discontinuities. It can be assumed that this allows historical video data to be aligned with a live scene.Such an adjustment of image properties can apply to the entire historical or live image. Alternatively, the image adjustment can be focused on the seam area or blended across image regions. As a further option, all or a substantial portion of a historical image can be adjusted to be consistent with the image properties of adjacent live image regions within the composite image. Advantageously, this last option can mitigate the effect of spotlight illumination in historical images by aligning their properties with neighboring sections of a live image that are not directly illuminated by spotlights. Each part of the historical or live image that forms part of a composite image can be processed separately, or historical images and live sections can be treated as individual regions for the purpose of adjusting their image properties.

[0065] When storing image properties, an image processor can buffer a small number of individual frames, e.g., four frames, and calculate average statistics to be stored with each frame. This can be done for a rolling cart buffer. Additionally or alternatively, this technique for averaging image properties over a small group of frames can be applied to live images to determine their properties for comparison with those of historical images. The advantage of such averaging techniques is that they mitigate the potential negative impact of a single historical or live image with radically different image properties compared to preceding or subsequent images.When the live images are averaged in this way, each historical image or group of historical images can be processed to correspond to the current or moving average image properties of live images. In particular, using the image property under consideration, which is white balance (or more generally, color balance), the white balance WB can be calculated for each of a group of four live images and averaged according to equation (1) to obtain the average white balance AVGWB. Suitable statistical techniques for calculating an image white balance or other color balance property are known to or available to a person skilled in the art. (WB1+WB2+WB3+WB4) / 4=AVGWB

[0066] Based on knowledge of the average white balance for a group of live images, the white balance for a historical image (HWB) can be compared to determine a difference in white balance (XWB) according to equation (2). It should be noted that XWB can be positive or negative. HWB−AVGWB=XWB

[0067] After determining the white balance difference, the white balance of a historical image can be adjusted according to the average white balance of the live image according to equation (3) to obtain an adjusted historical image white balance HWB'. zu The specialist has obtained suitable techniques for adjusting the color balance of an image, or these techniques are known to him or are available to him. HWB−XWB=HWB'

[0068] In equations (1) to (3), the respective WB or HWB property can be for an entire image or a portion of an image. In equations (2) and (3) above, it should be noted that historical white balance can also include the average white balance for a group of historical images. In both cases, the group size can differ from four. The technique described above, in conjunction with equations (1) to (3), can be applied equally to any other image property, for example, image exposure. Image property information for historical images can be stored per image (or image strip) or per group of images. If average image properties are stored, this can be done separately for images from each camera or averaged across two or more cameras.The stored image properties can be averaged across fixed image groups or taken from a rolling image memory and saved individually for each historical image.

[0069] As mentioned earlier, an alternative to equation (3) is to calculate the difference in image properties between historical images and live images and then apply this difference to the live images so that they match the historical images. This can be achieved by adding the white balance difference XWB to the white balance WB for at least one live image (for example, white balance). In some situations, adjusting the image properties of a live image can be disadvantageous. For instance, if a light source is in the field of view of a live image, adjusting its properties to match a historical image can lead to overexposure, resulting in a washed-out appearance. Furthermore, as explained above, in some situations it is desirable for live images to appear as close as possible to the vehicle's surroundings when viewed directly by the driver.

[0070] Adjusting image properties for historical or live images can be done as part of step 412, which is described in Fig. 4 is shown and described above, where historical images are fitted into a blind spot of the live image during the generation of a composite image. With reference to Fig. Section 7 now describes a system for forming a composite image that focuses on adjusting the image properties to avoid image discontinuities. Fig. 7 can be considered an extension of sections Camera 610 and Processor 604 of Fig. 6 are considered. It should be noted that the descriptions of the Fig. 4 and Fig. The seven sections are complementary, although they are intended to explain different aspects of generating a composite image. In particular, the detailed explanation above regarding the determination of a pattern recognition area and pattern recognition for superimposed live and historical images also applies to the image property adjustment described below.

[0071] As previously described, a vehicle camera system can include one or more cameras 710. Camera 710 provides a live image as specified in 702 and also offers an output 704 for one or more image properties. The in Fig. The specific image properties for live images identified in item 704 include dynamic range, gamma, and white balance (R, G, B). Other image properties (referred to herein alternatively as image characteristics) are chroma and saturation. It is noted that other image properties may be used, or in any combination. Any image property or group of image properties that can be measured and adjusted to reduce image discontinuities in a composite image may be recorded, including image properties not expressly listed. White balance has already been described. Color balance may be performed for a three-component image, e.g., red, green, blue. Any known color balance measurement may be taken and output by the Camera 710 for a live image.Dynamic range refers to the Camera 710's HDR (High Dynamic Range) option, which captures multiple images with different exposure values. The dynamic range information displays the range and / or absolute exposure values ​​for each image. These multiple exposures can be combined into a single image with a wider dynamic range of brightness. Gamma is a measure of image brightness. Note that alternative measurements of image brightness may be used.

[0072] The live image data 702 and the corresponding image property data are supplied to the electronic control unit (ECU) 706, although it should be noted that a separate image processing system can alternatively be used. The techniques disclosed herein can be used in any combination of hardware or software. In particular, live images can be forwarded directly to an output composite image 708 (for display on a vehicle display, for example in the instrument cluster, and not in Fig. (7 shown). Alternatively, the live images at point 711 can be processed, for example, by appropriate scaling, stretching, or buffering before being fed to the output image 708. Likewise, the image property data for live images are received at point 712. The live image data 712 is used by the ECU 706 as part of the historical image correction at point 714, e.g., as described above in conjunction with equations (1) to (3) and as above in conjunction with Fig. 8 is described in more detail. The historical image correction continues to accept stored historical images 716 and image property data 718 for historical images as inputs. The historical images 716 and the image property data 718 for historical images can be buffered within the ECU 706 or buffered separately, such as in the separate buffer 606 of Fig. Figure 6 is shown. It should be noted that the historical images 716 and the image property data 718 are ultimately derived from the camera system 710, although in Fig. 7. For the sake of simplicity, no direct relationship is shown.

[0073] The output image 708 includes at least one live image 721 and at least one historical image 722, which is adjusted according to its correspondence with the live image. The method by which the images are combined to form an output composite image 708 was described above in conjunction with Fig. 4 described.

[0074] With reference to Fig. Section 8 now describes in more detail a method for historical image correction. In particular, the method implements the following: Fig. 8 the historical image correction of part 714 of Fig. 7. The procedure takes a series of image properties for live and historical images as inputs. The procedure of Fig. Section 8, for the sake of simplicity, considers a situation where there is a single live image and a single historical image that are to be displayed side-by-side in a composite image, and where it is desirable to adjust the image properties of either the historical or the live image. Three specific image properties are considered to perform this adjustment. Furthermore, the following are considered: Fig. 8 a situation in which each image property is calculated (and adjusted) in relation to the entire image. From the preceding discussion, the person skilled in the art will easily be able to see how the procedure of Fig. 8 can be extended to other situations in which multiple images are processed and / or only parts of images are processed, as well as to alternative image properties that are processed.

[0075] A first part of the image correction of Fig. Section 8 concerns the dynamic range of live and historical images. The procedure receives as inputs a measure of the dynamic range for a historical image (step 800) and a measure of the dynamic range for a live image (step 802). The dynamic range of an image can be measured using an automatic exposure control algorithm of the camera to adjust the exposure times based on the scene's lighting. This relationship between exposure time and dynamic range can be unique for each sensor type, as it depends primarily on the sensor's sensitivity and the automatic exposure control algorithm of the image signal processor (ISP).

[0076] In step 804, a dynamic range correction factor is calculated by dividing the dynamic range of the live image by the dynamic range of the historical image to provide a dynamic range correction factor of 806. Dynamic range (DR) is defined in ISO 15730 as DR = Lsat / Lmin, the method for calculating camera dynamic range. This terminology is used to describe the dynamic range of the scene by sensor and ISP suppliers, who take Lmin as the sensor's noise floor. Knowing that Lmin is constant for our sensor, and that the exposure values ​​and weights are adjusted so that Lsat appears digitally maximized in the image, dynamic range correction can then be applied as a gain to the formula.

[0077] The process also receives as inputs a measure of gamma for a historical image (step 808) and a measure of gamma for a live image (step 810). Gamma (also called tone mapping curve) is provided by ISPs or sensor suppliers as an adjustable curve relative to the dynamic range (DR). Since the image is adapted to a new scene with a new DR, it is also necessary to compensate for gamma. Gamma is adapted to the scene but is captured as part of an adaptive camera setting. It is not necessary to measure this from the camera itself, but it can depend on the gamma calculation method of the ISP or sensor supplier. Gamma correction is well known to those skilled in the art, for example, as described at https: / / en.wikipedia.org / wiki / Gamma_correction.

[0078] In step 812, a gamma correction factor is calculated by subtracting the gamma of the historical image from the gamma of the live image to obtain a gamma correction factor of 814.

[0079] The process also receives inputs including a white balance measurement for a historical image (step 816) and a white balance measurement for a live image (step 818). The color temperature of the light in the scene is detected to activate the white balance. The goal of the white balance is to apply a correction factor to the image so that the color temperature of the light in the scene appears white in the image.

[0080] In steps 820 and 822, an ambient light color temperature is calculated separately for each historical image and the live image. An automatic white balance algorithm requires knowledge of the scene's color temperature to apply the corrected white balance. This can be specific to different sensor and ISP vendors. The calculated ambient light color temperature is then used to provide inputs 824 and 826 with respect to the color temperature of each image. In step 828, the color temperature of each image is used to determine a YCbCr conversion matrix, which is well understood by those with technical expertise.

[0081] The dynamic range correction factor 806, the gamma correction factor 814, and the YCbCr conversion matrix 828 can then be applied to the historical image (or a portion thereof) to adjust the historical image to match the live image. In this process, Fig. 8. This image adjustment is performed by a look-up table (LUT) 830, which advantageously reduces the computational requirements of the control unit or a separate image processing system. The LUT 830 takes the YCbCr information for a historical image as input 832. Specifically, the historical image data includes YCbCr data for each pixel. If necessary, this YCbCr data can be adjusted in step 834 from a range of 0 to 255 for each pixel to a range of -128 to 128 for each pixel. In some embedded systems, Cb and Cr data are stored from 0 to 255, with their range defined in the YCbCr color space from -128 to +128. The Cb and Cr data format in an embedded system is implementation-specific, so step 834 may not be required or may differ for different implementations. The LUT then performs the image correction for each pixel according to equation (4): Yout=DR_correction factor*Yingamma_correction factorCbout=DR_correction factor*Cbingamma_correction factorCrout=DR_correction factor*Cringamma_correction factor +YCbCrCT conversion matrix

[0082] Step 830 involves applying the correction factors to a historical image after they have been calculated. The CT conversion matrix can include a 3x3 correction matrix that is applied to the corrected Y, Cb, and Cr values ​​for each pixel of the historical image. A lookup table can be generated by first calculating all values ​​from 0 to 255 for all Y, Cb, and Cr values. This can simplify processing the historical image, as you only need to look up values ​​in the table instead of performing the calculation for each pixel.

[0083] In step 836, the updated YCbCr data can be adjusted, if necessary, from a range of -128 to 128 for each pixel up to a range of 0 to 255 for each pixel. In some embedded systems, Cb and Cr data are stored from 0 to 255, with their range defined in the YCbCr color space from -128 to +128. The Cb and Cr data format in the embedded system is implementation-specific, so step 836 can be omitted if it is modified for different implantations. In step 838, the updated historical image is output for further processing to be combined into a composite image, including by the pattern matching procedure of Fig. 4 or by suitable scaling and stretching (or this can be the method for harmonizing the image properties of Fig. 8 preceding).

[0084] It should be noted that the procedure of Fig. Figure 8 is only an example and can be modified depending on the type and number of image properties being calculated. In particular, the way in which the correction factors for the image properties are calculated and applied to the image data can vary depending on the type of image properties and the desired degree of change for historical images. As one example, calculated correction factors for image properties can be scaled to increase or decrease their overall effect or in relation to other correction factors. As another example, the correction factors can be scaled so that their effect varies in different areas of an image. Each image property can specify a property of the image that can be derived from the image itself or calculated.Alternatively, an image property can specify a setting of an image capture device for taking the image or an environmental factor prevalent during the taking of the image, none of which is directly recognizable in the image itself, but which can influence the appearance of the image.

[0085] The problem of parts of a camera's field of view being obscured is described above. In summary, if a camera lens or a window through which a camera passes to maintain its field of view is dirty or otherwise obstructed, then parts of the images captured by that camera will also be obscured. This can make it more difficult for the driver to discern details in the image of the vehicle's surroundings and render the camera system less useful. To be clear, when part of an image or part of a camera's field of view is described as obscured, it is not necessary that all light has been prevented from reaching the camera in that area. Part of the image may be missing, or part of the image may be distorted or damaged by at least partially blocking some of the light and thus preventing it from reaching the camera.Furthermore, it is possible that within a covered section some smaller areas of the camera's field of view are not covered, but generally a significant portion of this section of the camera's field of view will be covered.

[0086] With reference to the Fig. 9 and Fig. Section 10 now describes a method for compensating for a hidden camera field of view. This method builds on the approach described above for generating composite images that include areas not directly visible in the live images captured by a camera. In short, the present invention compensates for hidden image areas by inserting image data from another image into the hidden area to form a composite image.

[0087] Fig. Figure 9 illustrates an image 900 taken by a first camera, such as one of cameras 110, 120 in Fig. 1 or Fig. 610 in Fig. 6. Within image 900 there are three hidden areas 902. Obviously, the number, size and shape of the hidden areas can vary, as can the proportion of the image that can be hidden.

[0088] The compensation process begins with step 1000 of Fig. 10, in which obscured areas of the image are detected. This can alternatively be referred to as contamination detection. This can be easily done for a moving vehicle by comparing a live image with a historical image taken by the same camera at an earlier time. One approach to detecting obscured areas is through flow analysis of camera images, where, as the vehicle moves, areas can be identified where the pixels do not change, or change substantially, from one image to the next. At the same time, other areas are undergoing change—if no parts of the image change, either the vehicle has stopped or the camera is completely obstructed.Generally, a current image can be compared to a previous one, and if pixels or groups of pixels are the same (while others change), it can be assumed that the lack of change is due to light being prevented from reaching that part of the camera lens (or the camera is capturing an image of dirt or other material on the lens or camera window). The unchanged areas may include a significant portion of the dark pixels where light is completely blocked. It should be noted that some parts of a captured area may not be completely obscured, but the image may still be affected by the ingress of dirt or dust.The specialist is familiar with techniques for comparing images and identifying areas when the image has not changed, such as those used for motion analysis or video compression.

[0089] In step 1010, a boundary can be set for at least one obscured region. An obscured region can be a single obscured area of ​​a screen. Alternatively, an obscured region can be a collection of obscured areas interspersed with unobscured areas. This can be particularly relevant if dirt has splashed onto a camera lens or window, creating multiple splashed obscured areas. It may be computationally simpler to combine a series of small, closely spaced obscured areas into a single obscured region. Fig. Figure 9 shows three such obscured areas 902, in which the boundaries of each area are represented by dashed lines. One such obscured area 902 is labelled as rectangular. It should be noted that it is unlikely that dirt would obscure a rectangular portion of a camera's field of view. However, it may be computationally similar to stitching a composite image, inserting image data from another image into a square obscured area. The other two obscured areas 902 are represented as more complex shapes, assuming that, without any other constraints, it is desirable to retain as much as possible of the original image captured by the camera and to minimize the amount of image data that needs to be inserted into the composite image. The person skilled in the art is well familiar with suitable techniques for defining an area boundary.

[0090] Once the boundary of an obscured area is defined, another image with corresponding image data can be identified in step 1012. For a single camera system, the only option is to identify a historical, buffered image in which the relevant image data is captured in a different, unobstructed portion of the image. For a multi-camera system, suitable image data can alternatively be identified in a live image from another camera that has a field of view overlapping the field of view of the first camera, and the overlapping portion at least partially encompasses the obscured area. Alternatively, a historical image captured by another camera, including the relevant image data, can be identified.When using historical images, whether from the first camera or another camera, it is assumed that the vehicle's position changed between the time the historical image was taken and the time the current image was taken. This change in position resulted in the obscured portion of the area around the vehicle being captured in the historical image (provided, of course, that the historical image is not similarly obscured in the corresponding part of the field of view). It should be noted that if there are multiple obscured areas in an image taken by the first camera, different images from different sources may be identified to provide the obscured image data.Furthermore, especially when a single obscured image is large, it can happen that different parts of the same obscured area are filled with different images.

[0091] If historical image data, either from the same camera or a different camera, is used, optical flow analysis can be employed to detect a motion vector that replaces a historical image area showing the view through the obscured area (or at least a simulation of the view that would have been possible at the time the historical image was captured). Using historical image data requires that the camera images were previously buffered at the point in time when the camera obstruction is detected.

[0092] In another variation, a composite image can be created by overlaying image data from different sources, for example, image data from another camera and a historical image. This can be desirable in a situation where the historical image is of higher quality, but the live image data from another camera has the advantage of revealing objects that have moved into the field of view since the historical image data was captured.

[0093] Once another image has been identified in step 1012, a composite image is generated in step 1014 by combining the image from the first camera and the image data from the additional image. In step 1016, at least part of the composite image is displayed to the driver.

[0094] It should be noted that the procedure for determining another image that can provide image data to fill in a hidden area, using the above in conjunction with the Fig. The approach described in sections 1 to 6 for combining live and historical images into a composite image, including parts of the environment surrounding the vehicle that are not directly visible in a camera's field of view, can be used. In particular, using a historical image from either the same camera or a different camera in a multi-camera system to fill an obscured image area can involve the same pattern recognition process using a pattern recognition area to insert historical image data into the composite image, as described in conjunction with Fig. 4 described. If image data for a hidden image area is supplied by another live image from a different camera with an overlapping field of view, it should be noted that the process can be simplified if the overlapping sections of the camera's field of view are known in advance. Accordingly, the system diagram of Fig. 6 may be suitable for certain embodiments of the invention, in particular in the case where historical images are to be used, since in this case it will probably be necessary to accurately capture the vehicle position in order to determine a pattern recognition area.

[0095] It is further noted that when a hidden image area is filled with image data from another image to form a composite image, it is likely necessary to perform various image processing operations, including scaling, stretching, rotating, or twisting the image data from the other image to fit correctly into the hidden area. Furthermore, in certain embodiments of the invention, it may be desirable to perform the same process of adjusting the image properties to avoid image discontinuities, as described above in conjunction with the Fig. 7 and Fig. 8 described.

[0096] To illustrate the present invention, embodiments are now described in connection, firstly for a single-camera system and secondly for a multi-camera system. Single-camera system

[0097] A single camera system can be embodied in a typical reversing camera system, in which a single vehicle-mounted camera is located at or towards the rear of the vehicle, for example in a tailgate or rear bumper. Splashing water and dirt cannot be avoided when driving on wet roads or off-road tracks. While the present invention does not guarantee a completely unobstructed view of the area behind the vehicle, according to the above in conjunction with the Fig. 9 and Fig. The methods described in section 10 involve capturing and temporarily storing a series of consecutive images. Each image (or at least some of the images) can be compared with a previously captured image to determine whether the lens is obscured, and if so, a composite image can be generated to at least partially compensate for this obstruction.

[0098] It should be noted that in a basic embodiment, the image correction system can only be started once the system determines that the vehicle is moving, since the system requires two or more images known to have slightly different views. In particular, the vehicle's movement can be detected separately, for example, using the position sensors 404 in Fig. 4. Successive images are buffered so that an image processor can compare them. The system compares these successive images and looks for parts of the image that do not change and indicate dirt or water on the lens. If an area of ​​the image is classified by the system as being affected by lens contamination, the system uses portions of buffered images that match the current view and inserts them into it to effectively replace the obscured part of the view with an image film that has better clarity than would otherwise be possible. These image fragments can be adjusted for white balance and color correction (along with other necessary edits such as cropping, scaling, stretching, and distorting, along with other image property corrections) so that they do not appear out of place to the user.While the vehicle is moving, the corrected area of ​​the image can be sufficiently compensated to allow the driver a longer interval between cleaning the lens or camera window.

[0099] When the system begins correcting the image using buffered image data, the user, such as the vehicle's driver, can be notified, for example, by a visible warning on the display, that the image is being corrected using historical data and that the driver should proceed with greater caution. This also serves to inform the driver when the lens needs cleaning. In one example, the displayed image could be artificially tinted to highlight the ongoing image correction. Any other visual indicator or other types of warnings, such as an audible alert, could be used.

[0100] In an extension of the simple embodiment described above, images can be buffered and then stored for a longer period when the vehicle is stationary or even switched off, to provide historical images that restart the process of detecting obscured areas and generating a composite image when the vehicle is switched on or resumes movement. If invisible areas are detected when the vehicle is first switched on, and these invisible areas are significant, this could be a suitable time to inform the driver that they need to clean the camera lens before driving.

[0101] As previously mentioned, it's possible that the driver may only see a portion of a composite image. It's not uncommon for a camera to have a wider field of view than strictly necessary. The required portion of the field of view can be cropped (also known as masking the remaining sections). It's worth noting that if the camera used has a wider field of view, for example, than the reversing view, the image data from historical camera images in areas not typically displayed can also provide the necessary image data to compensate for obscured areas. This additional image data can be used to supplement the buffered image data. In this way, the total available image data collected by the camera can cover a portion of the vehicle's path that has not yet been shown to the driver.This reduces the time delay associated with replacing obscured areas of a first image with historical image data, since the second image from which this image data is obtained can be more recent.

[0102] While the present invention cannot make the reversing cameras dirt- and spray-proof, since they will ultimately be at a point where they are significantly darkened, embodiments of the invention may allow for less frequent lens cleaning. Multi-camera system

[0103] As previously described, a vehicle can be equipped with a variety of cameras, each providing a view of an area outside the vehicle. The cameras can be positioned to provide a view of the front and rear, as well as the left and right sides of the vehicle. Image processing is performed on these camera images to stitch them together and display them as an essentially 360-degree view of the vehicle's surroundings, as shown in Fig. 2 shown and in connection with Fig. 4 described (optionally including historical images used to fill the hidden space under the vehicle).

[0104] In this configuration, the camera images typically overlap, at least slightly, with the neighboring camera around the vehicle. The images captured by each camera are stretched, cropped, and / or resolved to provide a display that is easily understandable for the driver. As already described in connection with the single-camera system, the image processor can also process areas of the 360-degree view (as described in the section on...). Fig. 2 (referred to as the "image cup"), which would otherwise be obscured by dirt on one or more camera lenses. This can be achieved in one or both ways. First, by storing and caching portions of the image data captured by the cameras, then using historical image data from another camera (e.g., a neighboring camera) to fill in an obscured area in an image captured by a first camera. Alternatively, image data from neighboring cameras in overlapping fields of view, which would otherwise be cropped when generating the composite image, can be used to fill in an obscured area. In particular, image data from an overlapping field of view can be matched using the clear view in the overlapping camera fields of view and used as a substitute for the area of ​​view obscured by dirt.This use of image data from an overlapping field of view can involve replacing image data in a narrowly defined obscured area, or the clipping and stitching line in each image can be moved so that if an area is obscured in the view from one camera, but clear in the view from another camera, the line is moved to utilize the unobstructed view.

[0105] The at least partial use of image data from an overlapping field of view to compensate for obscured areas when generating a composite image relies somewhat less on the presence of historical image data in the buffer, where the historical image data is spatially separated by vehicle movement. Rather, the overlapping parts of the available image data are detected by a self-calibration function of the image processor, and so-called redundant parts of the image data (which are normally clipped) can be accessed with less overhead for the image processor to extend and improve an otherwise obscured view.This means that if the vehicle cameras are positioned to capture a wider field of view than is required for a given display mode, the image processor can use image data that would not otherwise be displayed to the driver to correct a blocked lens with less latency than would otherwise be the case with corrections based on historical image data.

[0106] Furthermore, a multi-camera system can use buffered image data from cameras at the front of the vehicle to assist the driver when reversing. Generally, the use of historical image data for compensating for obscured areas is not limited to adjacent or nearby cameras. As long as at least one of the cameras has captured the portion of the external environment necessary to compensate for the obscured area (and the memory is sufficient to buffer enough image frames), a composite reversing camera view can be generated even if the reversing camera lens is heavily soiled. In an example, the driver brings the vehicle to a stop from driving forward and then selects reverse. The driver wants to reverse the vehicle, but the reversing camera lens is dirty.With this configuration, suitable image data from the buffered forward-facing camera can be used to compensate for the obscuration of the reversing camera. Furthermore, if the reversing camera is completely obscured, the system can provide the driver with a reverse playback of front camera data, time-adjusted and mapped data using positional data and data derived from the views recorded by the side-view cameras, to create a view of the area behind the vehicle based on historical front camera images. In this case, it is advisable to inform the driver about the application of this compensation, as it does not indicate any new event or obstacle that enters the area behind the vehicle while reversing.As an example, generating a fully simulated rear view can be useful on very narrow, single-lane roads that rely on passing places to allow two-way traffic. In a situation where the vehicle recognizes that it is traveling on such a road, for example, confirmed by GPS data and optionally by side-view cameras, the buffering of the camera image data can be extended, for example, for side-view cameras from the side where the passing places are located, and especially if the system has determined that one or more camera lenses are partially obscured.

[0107] It should be noted that embodiments of the present invention can be implemented in the form of hardware, software, or a combination of hardware and software. In particular, the method of Fig. 4, Fig. 7, Fig. 8 and Fig.10. Implemented in hardware and / or software. Such software may be stored in the form of volatile or non-volatile memory, such as a storage device like a ROM, whether erasable or rewritable or not, or in the form of memory such as RAM, memory chips, devices, or integrated circuits, or on an optically or magnetically readable medium, such as a CD, DVD, magnetic disk, or magnetic tape. It should be noted that the storage devices and storage media are embodiments of machine-readable memory suitable for storing a program or programs that, when implemented, carry out embodiments of the present invention.Accordingly, embodiments provide a program containing code for implementing a system or method as required in a previous claim, and a machine-readable memory that stores such a program. Furthermore, embodiments of the present invention can be transmitted electronically over any medium, such as a communication signal transmitted via a wired or wireless connection, and embodiments that include these accordingly.

[0108] All features disclosed in this specification (including all related claims, abstractions and drawings) and / or all steps of a method or process disclosed so may be combined in any combination, unless the combinations are mutually exclusive and contain at least some of these features and / or steps.

[0109] Each feature disclosed in this specification (including all related claims, abstractions, and drawings) may be replaced by alternative features that serve the same, equivalent, or similar purpose, unless expressly stated otherwise. Unless expressly stated otherwise, each disclosed feature is only an example of a generic set of equivalent or similar features.

Claims

[1] A display method for use in a vehicle (100), the method comprising: Receiving a first image showing an area outside the vehicle (100) from a first image acquisition device (110); Capturing a hidden area of ​​the first image, for which part of the field of view of the first image is at least partially obscured; Identifying image data in a second image that corresponds to the obscured area; Generating a composite image from the first image and the identified image data; and Displaying at least a part of the composite image; where capturing a hidden area further includes: comparing the first image with another image obtained from the same image acquisition device (110), wherein the first and subsequent images are acquired at different times; and the detection of corresponding sections of the first image and the subsequent image that are the same, while other corresponding sections of the first image and the subsequent image differ. [2] Display method according to claim 1, wherein the detection of a hidden area further comprises defining a boundary that includes the corresponding sections of the first image and the further image, which are identical. [3] Display method according to one of the preceding claims, wherein the second image is obtained from the first image acquisition device (110) or a second image acquisition device (120) with a different field of view in relation to the first image acquisition device (110), wherein the first and second images are acquired at different times. [4] Display method according to one of claims 1 to 3, wherein the second image is obtained from a second image capture device (120) at the time of the first image capture, wherein the fields of view of the first (110) and second (120) image capture devices overlap and the overlap area at least partially includes the hidden area. [5] Display method according to any of the preceding claims, wherein the first or second image capture device (120) is mounted on or inside the vehicle (100) to capture images of the environment outside the vehicle (100). [6] Notification method according to any of the preceding claims, further comprising: Determining the positions of the vehicle (100) at the time the first and second images were taken; and Saving a display of the vehicle's positions (100). [7] Display method according to one of the preceding claims, wherein the generation of a composite image comprises corresponding sections of the first image and the second image. [8] Display method according to claim 7, wherein the adaptation of sections of the first image and the second image comprises the adaptation of overlapping sections of the first image and the second image. [9] Display method according to claim 7 or claim 8, wherein the fitting of sections of the first image and the second image comprises performing a pattern fitting to identify features present in both the first image and the second image, wherein these features are correlated in the composite image. [10] Display method according to claim 9, further comprising determining a pattern recognition area within the first image including the obscured area; and determining a second image with image data for the environment within the pattern recognition area. [11] Display method according to claim 10, wherein determining a pattern recognition area comprises determining coordinates for the pattern recognition area according to a current position of the vehicle (100). [12] Display method according to claim 11, wherein determining a pattern recognition area further includes receiving a signal indicating an orientation of the vehicle (100) and setting the coordinates of the pattern recognition area according to the vehicle orientation. [13] Notification method according to any of the preceding claims, further comprising: Retain at least one image property for each of the first and second images; Calculating an image correction factor as a function of at least one image property for each of the first and second images; and Adjusting the appearance of the first image or the second image according to the calculated image correction factor. [14] Display method according to claim 13, wherein the at least one image property is indicative of a property of the image, a setting of an image capture device (110) used to capture the image, or an environmental factor at the time of image capture. [15] Display method according to one of the preceding claims, wherein the generation of a composite image further comprises displaying the section of the composite image corresponding to the hidden area. [16] Display method according to one of the preceding claims, wherein the generation of a composite image further comprises the use of identified image data from the second image and at least one third image within the obscured area. [17] Display method according to one of the preceding claims, further comprising storing at least a predetermined number of images obtained from the first image acquisition device (110) at different times. [18] A computer program product that stores computer program code which, when executed, is arranged to perform the method according to any one of claims 1 to 17. [19] A display device for use with a vehicle (100), comprising: a first image acquisition device (110) arranged to obtain a first image showing an area outside the vehicle (100); a display means arranged to display a composite image; and a processing device arranged to: Capturing a hidden area of ​​the first image, for which a part of the field of view of the first image capture device (110) is at least partially obscured; Identifying image data in a second image that corresponds to the obscured area; Generating a composite image from the first image and the identified image data; and cause the display device to display at least part of the composite image; wherein the detection of a hidden area further includes: comparing the first image with another image obtained from the same image acquisition device (110), wherein the first and subsequent images are acquired at different times; and the detection of corresponding sections of the first image and the subsequent image that are the same, while other corresponding sections of the first image and the subsequent image differ. [20] Display device according to claim 19, wherein the processing means are further arranged to carry out the method according to any one of claims 1 to 18. [21] Vehicle comprising the display device according to claim 19 or claim 20.

Images