System and method for making reliable stitched images

CN115516511BActive Publication Date: 2026-10-09CONNAUGHT ELECTRONICS
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202180034150.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-04-09
Filing Date
2021-04-07
Publication Date
2026-10-09
Estimated Expiration
2041-04-07

AI Technical Summary

Technical Problem

[0003]如果分量图像图中的相同物体的图像映射到拼接图像中的不同位置,则拼接过程会导致拼接图像中的赝像,例如重影效应

Benefits of technology

[0011] In summary, a method for making stitched images more reliable is described. This method effectively and accurately detects double-effect artifacts, where objects in the component images map to separate discrete locations in the stitched image, and is able to issue an alert if the content in the stitched image is unreliable. In some embodiments, detected artifacts are efficiently replaced in real time by dynamic artifacts to produce a reliable stitched image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115516511B_ABST
    Figure CN115516511B_ABST
Patent Text Reader

Abstract

A method for detecting artifacts in a stitched image, comprising: acquiring component images of an environment from respective vehicle-mounted cameras having overlapping fields of view; forming (410) a stitched image from the component images; processing (420) at least a portion of the stitched image corresponding to the overlapping fields of view with a classifier to provide a list of detected objects from the environment at respective locations in the stitched image; determining (430) whether any detected object in the list of detected objects is a duplicate of another object in the list of detected objects; and reporting any objects determined to be duplicates.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to systems and methods for producing reliable stitched images. Background Technology

[0002] A stitched image is an image created by combining at least two component images from one or more cameras in different poses with overlapping fields of view to create an image with a larger or different field of view than the component images.

[0003] If images of the same object in the component images are mapped to different locations in the stitched image, the stitching process can cause artifacts in the stitched image, such as ghosting.

[0004] US9509909 describes a method for correcting photometric misalignment, comprising extracting block samples from at least one of a synthetic view geometry lookup table, an input fisheye image, and a view overlap region; selecting sample indices from the extracted block samples; estimating the optimal color gain of the selected block samples; performing fine adjustments based on the estimated color gain and applying a color transformation; and generating a synthetic surround view image.

[0005] US2018 / 0253875 describes a method for stitching images, which includes selecting a stitching scheme from a set of stitching schemes based on one or more content metrics of the component images, and applying the selected stitching scheme.

[0006] DE102016124978A1 describes a method that uses an additional projection surface in a virtual three-dimensional space to improve the recognizability of vertical objects on a display device of a driver assistance system for a motor vehicle, so as to better represent one or more vertical objects on the display device.

[0007] US2012 / 0262580 describes a system that can provide a surround view of a vehicle using cameras located at different positions on the vehicle. The cameras can generate image data corresponding to the surround view, and a processing device can process the image data to generate the surround view.

[0008] US2009 / 0110327 describes a method for facilitating the identification of a plane in a 3D coordinate system, wherein a 3D model is generated based on a 2D image. An extrusion direction is set for the plane and a region of interest in one of the 2D images, and the plane is extruded until the region of interest in the plane matches the corresponding region in the 2D image.

[0009] The purpose of this invention is to make stitched images more reliable without being limited by previous work. Summary of the Invention

[0010] This invention is defined by the independent claims. The dependent claims provide further optional features.

[0011] In summary, a method for making stitched images more reliable is described. This method effectively and accurately detects double-effect artifacts, where objects in the component images map to separate discrete locations in the stitched image, and is able to issue an alert if the content in the stitched image is unreliable. In some embodiments, detected artifacts are efficiently replaced in real time by dynamic artifacts to produce a reliable stitched image. Attached Figure Description

[0012] Embodiments of the invention will now be described by way of example with reference to the accompanying drawings, wherein:

[0013] Figure 1 A vehicle with multiple onboard cameras is shown;

[0014] Figure 2 Showing more details Figure 1 The overlapping area between the vehicle and the two cameras;

[0015] Figure 3 Two stitched images of the same scene are shown, with different blended areas highlighted;

[0016] Figure 4 A method for detecting and processing dual-effect artifacts according to an embodiment of the present invention is shown;

[0017] Figure 5 It shows in Figure 4 The embodiment uses a convolutional neural network (CNN) for labeling images;

[0018] Figure 6 An image is shown with labels superimposed on the detected known objects;

[0019] Figure 7 It shows Figure 3 Further stitched images of the scene, with corresponding blending regions marked. In the left image, the double-effect artifact of the car is detected and marked. In the right image, the detected double artifact has been processed;

[0020] Figure 8 It shows Figure 3 More stitched images of the scene, with the corresponding blending regions marked. In the left image, the double-effect artifact of the streetlight structure is detected and marked. In the right image, the detected double artifact has been processed; and

[0021] Figure 9 This shows the result after the detected dual-effect artifact was processed. Figure 8 A stitched image. Detailed Implementation

[0022] For many tasks involving driving vehicles, obtaining information about the local environment is crucial. One way to achieve this is by analyzing images from camera modules mounted on the vehicle. The images can then be stitched together to provide a more convenient view.

[0023] When attempting to image the environment surrounding a vehicle, a single camera often lacks sufficient field of view to capture all the necessary data. One solution to this problem is to use multiple cameras. Figure 1 The diagram illustrates a vehicle 100 with four cameras 101, 102, 103, and 104 located around the perimeter of the vehicle. One edge of the field of view of each camera is marked with dashed lines 101a, 102a, 103a, and 104a. This camera configuration results in overlapping fields of view in regions 101b, 102b, 103b, and 104b. As an example, region 102b represents the overlap between the field of view 101a of the front camera 101 and the field of view 102a of the left camera 102. The illustrated configuration is merely exemplary. The disclosed teachings are equally applicable to other camera configurations.

[0024] The illustrated field of view faces approximately 180 degrees. A wide field of view is typically achieved by cameras with wide-field-of-view lenses, such as fisheye lenses. Fisheye lenses are preferred because they are generally cylindrically symmetrical. In other applications of the invention, the field of view may be less than or greater than 180 degrees. While fisheye lenses are preferred, any other lens providing a wide field of view can be used. In this document, a wide field of view is a lens with a field of view exceeding 100 degrees, preferably exceeding 150 degrees, and more preferably exceeding 170 degrees. Typically, cameras with such wide fields of view result in imaging artifacts and distortion in the acquired images.

[0025] The sensitivity of the camera used in this invention need not be limited to any particular wavelength range, but most commonly, it will be used in conjunction with a camera sensitive to visible light. The camera is typically in the form of a camera module, including a housing for a lens and a sensor, the lens being used to focus light onto the sensor. The camera module may also have electronics for powering the sensor and being able to communicate with it. The camera module may also include electronics for processing images. This processing may be low-level image signal processing, such as gain control, exposure control, white balance, noise reduction, etc., and / or it may include more powerful processing, for example, for computer vision.

[0026] If the camera is configured to provide images from all directions around the vehicle, such as Figure 1 As shown, their component images can provide a view of sufficient information to synthesize a stitched image from one of the various virtual camera poses, particularly from above the vehicle.

[0027] In such Figure 1In the case of a vehicle with multiple cameras, it may be desirable to display a single image of the surrounding environment, including a viewpoint directly above the vehicle, rather than multiple images from multiple cameras. Therefore, images from all four cameras must be stitched together. Before or as part of the stitching process, the images can be mapped to different surfaces. For example, the component images can be mapped to a spherical or cylindrical surface before stitching. In one embodiment, the component images are mapped to a flat-bottomed bowl surface before stitching. This mapping can also take into account lens distortion and / or facilitate the stitching process.

[0028] At other times, such as in the examples described below, for instance, if a vehicle is about to turn left, it might be desirable to generate a stitched image from component images acquired from the vehicle's front camera 101 and left-side camera 102, with the virtual cameras positioned above the vehicle to more clearly show the intersection to the driver. When the vehicle is about to turn right or reversing at a corner, a similar stitched image can be generated from the paired images.

[0029] There are several known methods for stitching component images together. These include direct stitching, linear blending, selective blending, and multi-band blending. To describe these processes, we now turn to... Figure 2 It shows Figure 1 More details about the vehicle shown. Figure 2 In the image, the overlapping region 102b has a blending segment 102c. The blending segment 102c is at least a portion of the overlapping region and is defined by two angles: an angular offset 102d and an angular width 102e. Data from any component image located within the blending segment 102c is combined with data from another component image, and the combined data is used when forming the stitched image. The projection of the blending segment onto the component images defines the blending region within the component images.

[0030] Direct stitching defines a transition line in the overlapping area between two component images. In practice, the angular width 102e is set to zero. The stitched image uses an image from one component image on one side of the line and an image from the other component image on the other side. This stitching process results in an abrupt transition between component images. Therefore, artifacts may appear as visible seams or discontinuities in the stitched image. In some cases, the likelihood or severity of artifacts can be reduced using known camera module coordination techniques and devices, such as those described in German patent application DE102019126814.1 (reference number: 2019PF00721), filed October 7, 2019, entitled "Electronic Control Unit".

[0031] A known variation of direct stitching is direct stitching using dynamic seams. In this case, the direct stitching line does not need to be straight, but rather has a path that adjusts according to the content of the stitched component images. This approach can resolve some ghosting effects, but is unlikely to resolve double-effect artifacts, in which a given object from the component images appears at discrete, separated locations in the stitched image.

[0032] Linear blending adjusts the pixel values ​​in the blending region 102c by linearly adjusting the pixel weights of one component image as the distance across the blending region increases. The pixel values ​​within the blending region 102c are calculated as a weighted average of the pixel values ​​from the component images. Because the weights gradually decrease to zero, a smooth transition from one view to another is observed, rather than abrupt changes. The problem with linear blending is that objects within the blending region may become blurred due to imperfect object alignment between the two different views. Therefore, ghosting effects may be observed within the blending region due to the blending of misaligned objects.

[0033] Selective blending uses linear blending and direct stitching to find the composite pixel value I from the linear blending for each pixel. 线性 And the composite pixel value I from direct stitching 拼接 Then, these synthesized pixel values ​​are combined with weights related to the difference between the two component image values ​​at the considered location. The lower the difference, the higher the weight of linear blending, and vice versa. Selective blending avoids blending pixels corresponding to mismatched objects, thus reducing blurring and ghosting effects. However, it fails when mismatched objects have similar colors or when the residual differences after photometric calibration are still too large. Since the latter is one reason for double-effect artifacts, selective stitching is not ideal for solving double-effect artifacts. In other words, selective stitching is effective at eliminating ghosting effects from objects with uneven colors, but is unlikely to address extreme differences that lead to double-effect artifacts.

[0034] Multi-band blending improves the appearance of blended regions in a stitched image by dividing the component image into sub-bands and adaptively blending these sub-bands. In one example, frequency sub-band decomposition is applied to blending region 102c. For the high-frequency band, a first small blending region is applied. For the low-frequency band, a second larger blending region is used. As a result, this operation averages low-frequency components over a longer spatial range and high-frequency components over a shorter spatial range. Because the smaller blending range better preserves high-frequency components, the result is a sharper rendering of details in the blended region. However, multi-band blending cannot solve the ghosting problem of non-planar objects.

[0035] In all cases, there is a significant risk of double-effect artifacts in stitched images. The likelihood or occurrence of such artifacts increases when objects with high contrast to the background are present. Double-effect artifacts also exist in other situations, but usually without a noticeable visual effect. For example, double-effect artifacts in roads and skies are rarely problematic—this is usually not an issue if the two blank sections of the road have replicated textures.

[0036] The appearance of a double-effect artifact depends on the stitching technique and parameters used. For example, variations in the size of the blending region within the overlapping area between component images can alter the appearance of the double-effect artifact.

[0037] To demonstrate this change, consider Figure 3 It shows two image portions of a stitched image produced using two different stitching parameters. Figure 3 On the left-hand side, the stitching parameters forming the first image 300 define a relatively wide blending region 302c. This blending region 302c is defined by an angular width 302e and an angular offset 302e. The angular offset is measured from directly in front of the vehicle. Figure 3 On the right side, the stitching parameters that form the second image 300' define different, relatively narrow blending regions 302c'.

[0038] When obtaining the generation Figure 3 When stitching the component images of the images, there is only one car on the road in front of the vehicle. However, a double-effect artifact in the first image 300 causes two partially transparent cars to appear in the blending region 302c. Another double-effect artifact appears in the second image 300', causing two cars to appear in the same position as in the first image 300. However, in the second image 300', the two clearly visible cars are outside the blending region 302c' and are opaque. Stitching directly along the middle of the narrow blending region 302' with a line will produce an image similar to the second image 300'.

[0039] Traditionally, the blending region of a component image can be considered as a cylindrical sector. As an example, consider... Figure 3 The first image 300 in the image has cylindrical sectors extending the height of the image and is defined by an angle width 302e and an angle offset 302d. Since the occurrence of double-effect artifacts depends on the stitching technique and parameters, the likelihood and / or severity of double-effect artifacts can be reduced by adjusting these techniques and parameters. Layering or stacking cylindrical sectors provides significant freedom in mitigating double-effect artifacts because it offers more flexibility and additional stitching parameters for adjustment. Since each layer is a cylindrical sector, it has two additional parameters besides the angle width and angle offset: layer height and layer thickness.

[0040] Now for reference Figure 4 The diagram illustrates a flowchart describing an embodiment of the invention. Step 410 is to acquire a stitched image, which can be accomplished by combining component images from a pair of cameras with overlapping fields of view using any of the techniques described above. However, embodiments of the invention can also be applied to any stitched image, wherein a given object appearing in each of a pair of component images can be mapped to a discrete location within the stitched image.

[0041] In any case, the stitched image is then processed to detect object 420. This detection can be achieved using a known object detection classifier, such as a machine learning algorithm. In this embodiment, the classifier can directly process the stitched image, meaning that the process only needs to occur once for each stitched image. Alternatively or additionally, a classifier can be used for each component image, and the results can be combined by mapping the results of each component image to the stitched image.

[0042] In a preferred embodiment, the CNN is used to label known objects in an image. An example of such a CNN is... Figure 5 As shown. Various methods for training a CNN to detect known objects are known to technicians. Once trained, a CNN can process an image and label the detected objects with appropriate labels. The labels applied to the detected objects typically define bounding boxes that enclose at least a portion of the detected object and the name of the known object that has been detected. The bounding boxes do not have to be rectangular and can be conveniently of different shapes and sizes. If the classifier is operated on the component images, the mapping between the component images and the stitched image may result in different shapes of bounding boxes in the stitched image.

[0043] CNNs can detect known objects by processing an input image (511) with one or more convolutional or pooling layers. In convolutional layers (512), one or more convolutional kernels pass through the image, and in pooling layers (513), the spatial resolution of the processed data is reduced. Figure 5 In the example shown, two convolutional layers 511 and 513 and two pooling layers 513 and 515 are used. In embodiments of the invention, any number of convolutional or pooling layers can form the hidden layer 510 of the CNN400. Data from the hidden layer 510 is then processed by the classification layer 520 to form the result. Figure 5 In the example shown, data from the hidden layer is flattened 521 to provide a feature vector, which is then passed through multiple fully connected layers 522. In this case, a softmax operation 523 is performed to identify known objects in the image, such as trucks, street light structures, or cars.

[0044] Known alternative classifiers can also be used to detect known objects. In some embodiments, the classifier can utilize information determined from other sensors, such as a LiDAR sensor on a vehicle. Additional optional inputs, such as edge enhancements or edge images, can also be input when training the classifier to assist when limited input images are available. For some classifiers, such additional inputs help reduce the complexity of the network; that is, using additional inputs can reduce the number of hidden layers in a CNN.

[0045] Typical output of a classifier is as follows Figure 6 The image is shown in the figure. The input image has multiple labels overlaid on the processed image, including a traffic light 601a, two trucks 601b and 601c, and a car 601d. Each label has an associated location and is marked with a bounding box that marks a region of the input image that the classifier considers to be associated with the labeled object.

[0046] Since the goal is to identify double-effect artifacts in one or more overlapping regions, the regions of interest (ROIs) of the stitched images being processed can be limited to the overlapping regions of the stitched images, i.e., the regions where the fields of view of the component images overlap. This reduction in ROI significantly speeds up processing, reduces unnecessary object detection, and greatly lowers the false alarm rate.

[0047] ROI can be defined by stitching techniques and parameters. For example, the stitching angle (e.g.) Figure 3 Parameters such as the angular width (302e) and angular offset (302d) can be used to define the ROI.

[0048] Once an object is detected, it is tested 430 times to see if it is similar to another object. In other words, the detected objects are processed to detect the similarity between contents within the bounding box. Typical duplicate content detection algorithms are computationally intensive. Due to the large distance between the object and the camera, double-effect artifacts primarily exhibit translational shifts. Therefore, duplicate content evaluation does not need to consider all possible distortions. In some embodiments, a classifier such as a CNN is trained to detect double-effect artifacts by focusing on translational shifts in the content. Limiting translational shifts helps reduce the number of false positives. In this case, false positives are similar objects not caused by double-effect artifacts. The probability of false positives is significantly minimized due to the narrow constraint imposed only on the translational shift test. The classifier training can be improved by considering the viewpoint variation between one camera and another. This training improvement is because duplicate objects from double-effect artifacts may appear slightly different due to viewpoint variations.

[0049] When detecting double-effect artifacts, the timing signal can also help avoid false positives. For example, when an object approaches a vehicle, duplicate objects created by double-effect artifacts tend to move together and may even blend together. This is typically not the case for other types of duplicate objects that might be detected.

[0050] Similarity testing can be incorporated into the classifier that detects objects, or it can be applied separately to a list of detected objects. In a preferred embodiment, after objects in the stitched image are labeled, the CNN evaluates the similarity of the labeled objects to detect duplicates. The result of the similarity test is the detection of similar labeled objects, i.e., the detection of possible double-effect artifacts.

[0051] Another optional step is to test whether the duplicated object detected by 440 is real. In some embodiments, this step includes processing at least one component image to see if a similar duplicated object is detected in either component image. If a duplicated object is present in either component image, it is likely not a stitched-together artifact. This step helps ensure that two real, similar objects are not mistakenly labeled as a double artifact. This is important because objects duplicated due to artifacts can be ignored or subsequently removed from the image; ignoring or removing real objects (such as cars) can be a serious error. Therefore, testing whether obviously duplicated detected objects are real improves the reliability of the stitched image.

[0052] In response to detecting and optionally confirming that the stitched image contains a double-effect artifact, several options are available. These can range from small actions such as marking that the stitched image may contain an artifact, to proactive responses in some cases by attempting to correct the stitched image or prevent such artifacts from appearing in subsequently generated stitched images.

[0053] Therefore, for example, the detection of an artifact can be reported to the driver to ensure they are aware of potentially misleading information in the stitched image. For instance, when parked, an alert can be issued to signal to the driver that two nearby prominent light poles appear to be caused by a double-effect artifact. The driver can then visually confirm in the mirror which prominent light pole is most relevant and maneuver the car accordingly. The stitched image and the detected double-effect artifact can also be recorded and entered by vehicle subsystems, such as hard drives or other storage areas.

[0054] The presence of double-effect artifacts can also be reported to a machine learning algorithm configured to adjust stitching parameters to mitigate them. In a preferred embodiment, the machine learning algorithm adjusting the stitching parameters is a convolutional neural network (CNN).

[0055] In some cases, the content of the stitched images can be simply adjusted to mark detected double-effect artifacts.

[0056] However, in this embodiment, the content of the stitched image is adjusted before it is displayed.

[0057] Artifacts are graphic pseudo-images added to an image to better represent missing or distorted objects. For example, consider... Figure 3 Note that all the vehicle cameras are located around the perimeter of the vehicle and face outwards. Therefore, no image from any of the vehicle cameras will allow the vehicle to be properly displayed in the image. Figure 3 The vehicle shown in the lower right corner of the image is simply an avatar of the vehicle superimposed in the correct position within the stitched image. This avatar significantly improves the appearance of the stitched image and makes it easier for the driver to see how the vehicle is oriented within the stitched image. Vehicle avatars are examples of static artifacts, which are artifacts generated at least partially before the method is executed and stored in memory for use when necessary. A drawback of static artifacts is that they require the replacement image to be known or computationally available beforehand. When static artifacts are not feasible, dynamic artifacts can be used. Dynamic artifacts are artifacts generated at runtime based at least partially on data determined from one or more cameras.

[0058] In some embodiments, the present invention uses artifacts to address detected double-effect artifacts. Specifically, in step 450, one of the replicated objects is replaced with a dynamic artifact. The dynamic artifact comprises image data from component images. For each double-effect artifact, one component image provides data for one object in the double-effect artifact, and another component image provides data for the other object. Therefore, replacing the region of the stitched image detected as a double-effect artifact with data from the other component image will eliminate the double-effect artifact.

[0059] Two examples of this process are as follows: Figure 7 and Figure 8 As shown. In Figure 7 In the image 700 on the left side, a double-image artifact in the form of a replicated car is present. The bounding box of the detected double-image artifact 701 is represented by a black and white dashed box. In the stitched image, the data in the lower left region of the blending region 702 comes from a component image originating from a left-facing vehicle camera. The data in the upper right region of the blending region 702 comes from a component image originating from a front-facing vehicle camera. Therefore, the data forming the leftmost car 703 object comes from the left-facing camera, and the car in the bounding box of the detected double-image artifact 701 comes from the front-facing camera. In the image 700' on the right side, the data in the bounding box of the detected double-image artifact 701 has been replaced by data from the left-facing camera, thus eliminating the double-image artifact. Figure 8 The process is illustrated as the streetlight structure was also identified as a double-effect artifact 801. Similarly, switching the data within the bounding box of the double-effect artifact to the data of another component image resolved the double-effect artifact.

[0060] Figure 9 The image shows a stitched result without noticeable double-effect artifacts. Further smoothing effects can be used to mask adjusted edge areas in the stitched image. For example, artifacts can be blended with the original stitched image to avoid abrupt changes in pixel values. In some embodiments, the smoothing effect blurs the edges of the artifacts. This smoothing effect helps to hide artifacts in the stitched image.

[0061] The stitched images can be further processed to refine them. Furthermore, separate processing can be performed to remove different image artifacts. For example, see German Patent Application No. 102019131971.4 (reference number: 2018PF02667), filed November 26, 2019, entitled "Image Processing Module". Pre-removal of double-effect artifacts prevents any subsequent processing that enhances the appearance of double-effect artifacts.

[0062] The described method aims to reliably resolve distracting high-contrast double artifacts in stitched images. Unlike other duplicate content detection methods, the described method does not waste resources detecting and / or eliminating double artifacts in unwanted areas, such as correcting the appearance of a uniform road surface. Instead, the described method focuses on reliably mitigating the most visually striking double artifacts (e.g., a second image of a car on the road ahead).

[0063] The described method can help vehicle drivers trust stitched images. For example, consider a driver viewing a stitched image on a display installed inside the vehicle, or a driver manipulating the car to stop by viewing a stitched image on a display screen installed in the passenger compartment. The described method can warn the driver of the presence of a double-effect artifact or remove the double-effect artifact from the displayed image. In both cases, the displayed stitched image will be more reliable.

[0064] The vehicle under consideration can also be an autonomous vehicle, i.e., a self-driving vehicle or a vehicle with driver assistance features. In this case, the accuracy of the considered image is particularly important. For example, vehicle control mechanisms can base vehicle control or driving recommendations on the stitched image. Therefore, by reporting or eliminating double-effect artifacts, the vehicle control mechanism can take appropriate action. Thus, by using the described method, undesirable driving decisions made or recommended due to double-effect artifacts in the stitched image can be reduced.

[0065] The vehicle system can also record alarms for detected dual-effect artifacts or corrected stitched images. Recording can be done in the form of a media storage device such as a hard drive.

[0066] While the above examples have been provided based on stitched images acquired from the vehicle's front camera 101 and left camera 102, it should be understood that at other times, stitched views from other combinations of cameras with adjacent fields of view may be of interest, and similarly, the invention can also be extended to creating surround view images stitched from component images acquired from all cameras 101...104 around the vehicle.

Claims

1. A computer-implemented method for detecting artifacts in stitched images, comprising: Component images of the environment are acquired from the corresponding vehicle-mounted cameras with overlapping fields of view; A (410) stitched image is formed from the component images; The classifier (420) processes at least a portion of the stitched image corresponding to the overlapping field of view to provide a list of objects detected from the environment at the corresponding locations in the stitched image; Determine (430) whether any detected object in the list of detected objects is a copy of another object in the list of detected objects; as well as The report was determined to be a copy of any object.

2. The computer-implemented method according to claim 1 further includes: For any reported duplicate object, the region of the stitched image containing the reported duplicate object is replaced (450) with an artifact from one of the component images.

3. The computer-implemented method according to claim 2, wherein, Each detected object (601a) in the list of detected objects is defined by a bounding box, and The process of replacing a region of the stitched image containing the reported copy of the object with an artifact from one of the component images includes replacing the reported copy of the object (701) with an artifact (701') having the same shape as the bounding box of the reported copy of the object.

4. The computer-implemented method according to claim 2 or 3, wherein, At least a portion of the artifact is mixed with the original stitched image (460).

5. The computer-implemented method according to any one of claims 1 to 3, further comprising: Select any report to copy the object; as well as In response to (440) the selected object appearing multiple times in the component image, the selected object is marked as non-replicated.

6. The computer-implemented method according to any one of claims 1 to 3, wherein, Forming a stitched image from the component images includes: Select multiple overlapping regions of the two component images to define multiple blending regions; Adjusting the shape or position of at least two of the multiple blending regions; and A stitched image is formed from two component images, which involves combining data from the two component images in multiple blended regions.

7. The computer-implemented method according to claim 6, wherein, Adjusting the shape or position of at least two of the plurality of blending regions includes using a convolutional neural network to select blending parameters (102e, 102d) that define the shape or position of at least two blending regions. The convolutional neural network has been trained by changing the blending parameters and evaluating whether an artifact is detected in the stitched image generated by each set of blending parameters.

8. The computer-implemented method according to any one of claims 1 to 3, wherein, Any reported reproduction of an object is a spliced ​​artifact, which is due to: The image of the object from the first vehicle-mounted camera is mapped to the first position in the stitched image; as well as The imaged object from the second vehicle-mounted camera is mapped to a second position in the stitched image, and the second position differs from the first position by more than a threshold.

9. The computer-implemented method according to claim 8, wherein, The threshold is set such that the reported duplicate object does not overlap with the original object.

10. The computer-implemented method according to any one of claims 1 to 3, wherein, The component images are simultaneously acquired from two corresponding cameras (101, 102).

11. The computer-implemented method according to any one of claims 1 to 3, wherein, Determining whether any detected object in the list of detected objects is a copy of another object in the list involves using a convolutional neural network to classify whether a detected object is a copy, and The convolutional neural network has been trained to identify the copied object as a translational copy of another object in the detected object list.

12. The computer-implemented method according to claim 11, wherein, A translation copy of another object is a translation copy of another object that has been adjusted to account for changes in perspective between the onboard cameras.

13. The computer-implemented method according to any one of claims 1 to 3, wherein, At least a portion of the stitched image includes a region of the stitched image formed by data from more than one component image.

14. A vehicle (100) including a camera module that operates according to any one of claims 1 to 13 of the computer-implemented method.

Citation Information

Patent Citations

  • virtual representation of an environment of a motor vehicle in a driver assistance system with multiple projection surfaces

    DE102016124978A1

  • Semi-automatic plane extrusion for 3D modeling

    US20090110327A1

  • Vehicle Surround View System

    US20120262580A1

  • Systems and methods for content-adaptive image stitching

    US20180253875A1

  • Method and apparatus for a surround view camera system photometric alignment

    US9509909B2