Around view monitor image processing apparatus and method for detecting object by considering correlation between image distortion area and camera

Abstract

The present invention relates to an around view monitor (AVM) image processing apparatus and method for detecting an object by considering the correlation between an image distortion area and a camera, and may comprise: a plurality of cameras distributively provided at the front, rear, left and right of a vehicle so as to simultaneously acquire and output a plurality of camera images; an AVM image generation unit for generating an AVM image by matching the plurality of camera images; a plurality of object detection units each having an artificial intelligence model in which the correlation between an input image and an object detection result is pre-learned, and acquiring and outputting an object detection result corresponding to a camera image through the artificial intelligence model; a correlation analysis unit which pre-defines and registers the correlation between an image distortion area and a camera, and which checks, according to the correlation, whether an object in the image distortion area is detected twice through two camera images so as to acquire and provide a final object detection result; and a data output unit for configuring and displaying a screen on which the AVM image and the final object detection result are simultaneously displayed.

Description

Around view monitor image processing device and method for performing object detection operation considering image distortion area and correlation between cameras

[0001] The present invention relates to an around view monitor image processing device and method that performs an object detection operation that takes into account the correlation between an image distortion area and a camera, which enables detection of an object located around a vehicle, and in particular, performs an object detection operation that takes into account the correlation between an image distortion area and a camera, so as to more accurately detect an object existing in an image distortion area.

[0002] Due to recent developments in the automobile industry, automobiles have become so widespread that it can be said that we are now in an era where every household owns one car. Various cutting-edge electronic technologies are being applied to automobiles to improve safety and enhance driver convenience.

[0003] Among these cutting-edge electronic technologies, there is an Around View Monitoring (AVM) device that captures and displays the surroundings of the car, allowing the driver to conveniently check the surroundings of the car with the naked eye.

[0004] The AVM image providing device captures the surrounding environment through cameras installed on the front, rear, left, and right sides of the vehicle, as shown in Fig. 1, synthesizes multiple captured images to create an AVM image, and then provides the image to a display device installed in the vehicle.

[0005] Accordingly, the driver can accurately recognize the surroundings of the vehicle through the displayed surrounding environment and park conveniently without having to look at the side mirrors or rearview mirrors.

[0006] Meanwhile, a technology has been proposed to prevent safety accidents caused by collisions with objects in advance by detecting and notifying objects around a vehicle using AVM images.

[0007] However, the camera provided in the AVM image providing device is implemented as a wide-angle camera such as a fish-eye camera, so objects existing in the areas on both sides of the camera, such as areas A, B, C, and D in Fig. 1, are captured with a very large distortion rate, and as a result, there is a problem that it is difficult to accurately detect objects located in the corresponding image distortion area.

[0008] The present invention provides an around view monitor image processing device and method that performs an object detection operation that takes into account the correlation between the image distortion area and the camera, so as to more accurately detect an object existing in the image distortion area by identifying a camera that photographs each image distortion area due to the characteristics of the camera provided in the AVM image providing device, and confirming and reflecting the correlation between them.

[0009] In order to achieve the above object, according to one embodiment of the present invention, an around view monitor image processing device that performs an object detection operation considering the correlation between an image distortion area and cameras may include: a plurality of cameras installed in the front, rear, left, and right of a vehicle to simultaneously acquire and output multiple camera images; an AVM image generation unit that generates a single AVM (Around View Monitor) image by matching the multiple camera images; a plurality of object detection units that have an artificial intelligence model in which the correlation between an input image and an object detection result is pre-learned, and acquire and output object detection results corresponding to the camera images respectively through the artificial intelligence model; a correlation analysis unit that pre-defines and registers the correlation between the image distortion area and the cameras, and determines whether an object existing in the image distortion area is detected redundantly through two camera images according to the correlation, and acquires and provides a final object detection result; and a data output unit that configures and displays a screen that simultaneously displays the AVM image and the final object detection result.

[0010] The above artificial intelligence model is characterized by repeatedly learning a large amount of learning data that has a camera image as an input condition and an object detection result as an output condition, and predicting and outputting the detection result of an object corresponding to the input image in an artificial intelligence manner.

[0011] The above correlation analysis unit is characterized by further including a function of identifying and defining correlation by segmenting the first camera image for recognizing the presence or absence of an object by image distortion area and vehicle movement direction and the second camera image for detecting object details, taking additional consideration of the vehicle movement direction.

[0012] The above correlation analysis unit is characterized by further including a function of obtaining and providing a final object detection result by multi-stage analysis of the object detection result of the first camera image and the object detection result of the second camera image according to the correlation.

[0013]

[0014] In order to achieve the above object, according to another embodiment of the present invention, a method for processing an around view monitor (AVM) image for performing an object detection operation considering the correlation between the image distortion area and the cameras of an AVM (Around View Monitor) image device may include the steps of: simultaneously acquiring multiple camera images through multiple cameras distributed in a vehicle; generating a single AVM (Around View Monitor) image by aligning the multiple camera images; obtaining object detection results corresponding to the camera images through an artificial intelligence model in which the correlation between the input image and the object detection result is pre-learned, and then obtaining and providing a final object detection result by checking whether an object existing in the image distortion area is detected in duplicate through two camera images based on the correlation between the image distortion area and the cameras; and configuring and displaying a screen for simultaneously displaying the AVM image and the final object detection result.

[0015] The present invention enables more reliable and accurate detection of objects existing in an image distortion area that is necessarily caused by the characteristics of a camera provided in an AVM image providing device.

[0016] Additionally, the camera correlation can be set in various ways depending on either the image distortion area or the vehicle movement direction, allowing for flexible response to changes in the vehicle operating environment.

[0017] Figure 1 is a drawing showing the camera arrangement structure of a typical AVM image providing device and the image capturing range of each camera.

[0018] FIG. 2 is a drawing for explaining an around view monitor image processing device that performs an object detection operation considering the correlation between an image distortion area and a camera according to one embodiment of the present invention.

[0019] FIG. 3 is a drawing for explaining the correlation between an image distortion area and a camera according to one embodiment of the present invention.

[0020] FIG. 4 is a drawing for explaining the correlation between an image distortion area and a camera according to another embodiment of the present invention.

[0021] FIG. 5 is a drawing for explaining a data provision screen according to one embodiment of the present invention.

[0022] FIG. 6 is a drawing for explaining an around view monitor image processing method for performing an object detection operation that takes into account the correlation between an image distortion area and a camera according to one embodiment of the present invention.

[0023] Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure complete disclosure of the present invention and to fully inform those skilled in the art of the scope of the invention. Like reference numerals throughout the drawings denote like elements.

[0024]

[0025] FIG. 2 is a drawing for explaining an around view monitor image processing device that performs an object detection operation considering the correlation between an image distortion area and a camera according to one embodiment of the present invention.

[0026] Referring to FIG. 2, the AVM (Around View Monitor) image processing device (100) of the present invention includes a plurality of cameras (110), an AVM image generation unit (120), a plurality of object detection units (130), a correlation analysis unit (140), and a data output unit (150).

[0027]

[0028] Multiple cameras (110) are distributed around the front, back, left, and right of the vehicle to simultaneously acquire and output multiple images with different shooting points. At this time, the cameras are implemented as wide-angle cameras such as fish-eye cameras, but are not limited thereto.

[0029]

[0030] The AVM image generation unit (120) acquires and prepares calibration information for each camera installed in a vehicle in advance, and based on this, aligns multiple camera images to generate a single AVM (Around View Monitor) image.

[0031] At this time, the calibration information may be information distributed by the vehicle manufacturer, but it is also possible to obtain it by directly performing camera calibration on a vehicle-by-vehicle basis if necessary.

[0032]

[0033] The object detection unit (130) is equipped with an artificial intelligence model in which the correlation between the camera image and the object detection result is pre-learned, and acquires and outputs the object detection result corresponding to each camera image through the artificial intelligence model.

[0034] At this time, the object detection result may be at least one of the existence, location, type, and shape information of the object. And the artificial intelligence model can be implemented as an artificial intelligence model implemented as NN (Artifical Neural Network), DNN (Deep Neural Network), CNN (Convolution Neural Network), RNN (Recurrent Neural Network), etc., and it repeatedly learns a large amount of learning data that has a camera image as an input condition and an object detection result as an output condition, so that the detection result of an object corresponding to the input image can be predicted and output in an artificial intelligence manner.

[0035]

[0036] The correlation analysis unit (140) identifies areas where image distortion occurs severely, such as areas A, B, C, and D in Fig. 1, and identifies and registers cameras (130) that photograph the corresponding areas for each image distortion area, thereby generating and storing correlation information.

[0037] And, if it is confirmed that two cameras with correlation with each other simultaneously detect an object existing in the same image distortion area by cross-analyzing the object detection result of the object detection unit (130) based on the above correlation information, the object detection result is finally acquired and output as valid information.

[0038]

[0039] The data output unit (150) configures a screen that simultaneously displays AVM images and object detection results, and displays them audiovisually through an in-vehicle multimedia device.

[0040] The object detection result of the present invention may be guided using a bounding box that is overlaid on an AVM image to indicate the location of the object, text that is positioned adjacent to the bounding box to indicate the type of object, etc., but need not be limited thereto.

[0041]

[0042] FIG. 3 is a drawing for explaining the correlation between an image distortion area and a camera according to one embodiment of the present invention.

[0043] Referring to Figure 3, when cameras are installed in a distributed manner on the front, right, rear, and left sides of a vehicle, adjacent cameras, such as areas A, B, C, and D, take overlapping shots, but an image overlapping area occurs in which object distortion is severe.

[0044]

[0045] Accordingly, in the present invention, it is determined that the front camera (111) and the right camera (112) that capture area A have a correlation with area A, and the camera correlations for the remaining areas B, C, and D can all be determined using the same principle.

[0046] And, based on the correlation between the image distortion area and the camera, it is determined that the object actually exists only when two cameras correlated to a specific image distortion area simultaneously detect the object.

[0047] That is, only when the front camera (111) and the right camera (112) simultaneously detect an object existing in area A, when the rear camera (113) and the right camera (112) simultaneously detect an object existing in area B, when the rear camera (113) and the left camera (114) simultaneously detect an object existing in area C, and when the front camera (111) and the left camera (114) simultaneously detect an object existing in area D, it is determined that the object actually exists in the area.

[0048]

[0049] In addition, the present invention considers that the object distortion rate for each camera varies depending on the direction of vehicle movement, and thus allows for further refinement of the correlation between the image distortion area and the camera by additionally reflecting the direction of vehicle movement.

[0050] FIG. 4 is a drawing for explaining the correlation between camera images according to another embodiment of the present invention.

[0051] Depending on the direction of vehicle movement, the location where a safety accident due to an object collision is more likely to occur may vary. That is, (1) Area A during a forward right turn, (2) Area B during a backward left turn, (3) Area C during a backward right turn, and (4) Area D during a forward left turn are areas where the possibility of a safety accident due to an object collision due to the creation of a blind spot increases.

[0052] Meanwhile, (1) when turning right forward, objects located in area A are captured with less distortion through the right camera, (2) when turning left backward, objects located in area B are captured with less distortion through the right camera, (3) when turning right backward, objects located in area C are captured with less distortion through the left camera, and (4) when turning left forward, objects located in area D are captured with less distortion through the left camera.

[0053]

[0054] Accordingly, in the present invention, (1) when driving a forward right turn, only the presence or absence of an object is recognized through a front camera image, and object details (e.g., object location and type) are detected through a right camera image, (2) when driving a backward left turn, only the presence or absence of an object is recognized through a rear camera image, and object details are detected through a right camera image, (3) when driving a backward right turn, only the presence or absence of an object is recognized through a rear camera image, and object details such as object type are detected through a left camera image, and (4) when driving a forward left turn, only the presence or absence of an object is recognized through a front camera image, and object details are detected through a left camera image.

[0055] In other words, the present invention further considers the direction of vehicle movement according to the image distortion area and the direction of vehicle movement, segments the first camera image for recognizing the presence or absence of an object and the second camera image for detecting object details, and then performs a multi-stage analysis of the object detection result of the first camera image and the object detection result of the second camera image to identify and define the correlation, thereby further improving the object detection performance.

[0056]

[0057] FIG. 5 is a drawing for explaining a data provision screen according to one embodiment of the present invention.

[0058] As shown in FIG. 5, the present invention can provide a screen that simultaneously displays an AVM image and an object detection result.

[0059] At this time, the object detection result can be visually and audibly guided using a bounding box that overlays the AVM image to indicate the object location and text that is adjacent to the bounding box to indicate the object type, or the object location and object type can be auditorily guided through voice.

[0060] Additionally, if necessary, information defining the level of risk by object location and type can be acquired and stored in advance, and based on this, the level of risk corresponding to the current object detection results can be calculated and a warning can be issued.

[0061] Additionally, the intensity of user guidance can be differentiated based on the level of risk. For example, the intensity of user guidance can be differentiated by (1) varying the content included in the guidance message, (2) increasing the number of guidance messages provided or the volume, or (3) varying the display method of images and text, but is not limited to these.

[0062]

[0063] FIG. 6 is a drawing for explaining an around view monitor image processing method for performing an object detection operation that takes into account the correlation between an image distortion area and a camera according to one embodiment of the present invention.

[0064] Referring to FIG. 6, the method of the present invention includes a step (S1) of simultaneously acquiring multiple camera images through multiple cameras distributed across a vehicle, a step (S2) of generating a single AVM (Around View Monitor) image by aligning the multiple camera images, a step (S3) of acquiring object detection results corresponding to the camera images through an artificial intelligence model in which the correlation between the input image and the object detection result is pre-learned, and then a step (S3) of obtaining and providing a final object detection result by checking whether an object existing in the image distortion area is detected redundantly through two camera images based on the correlation between the image distortion area and the cameras, and a step (S4) of configuring and displaying a screen that simultaneously displays the AVM image and the final object detection result.

[0065]

[0066] Additionally, the method according to the present invention can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system.

[0067] Examples of storage media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, hard disks, flash drives, and others, as well as those implemented in the form of carrier waves (e.g., transmitted over the Internet). Furthermore, computer-readable storage media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

[0068]

[0069] The above description is merely an example of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present invention.

[0070] Therefore, the embodiments disclosed in the present invention are intended to illustrate, rather than limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical concepts within the scope equivalent thereto should be construed as being included within the scope of the present invention.

Claims

1. Multiple cameras installed on the front, back, left, and right of the vehicle to simultaneously acquire and output multiple camera images; An AVM image generation unit that creates a single AVM (Around View Monitor) image by aligning multiple camera images; A plurality of object detection units each having an artificial intelligence model in which a correlation between an input image and an object detection result is pre-learned, and obtaining and outputting an object detection result corresponding to a camera image through the artificial intelligence model; A correlation analysis unit that predefines and registers the correlation between the image distortion area and the camera, and obtains and provides the final object detection result by checking whether an object existing in the image distortion area is detected in duplicate through two camera images according to the correlation; and An around view monitor image processing device that performs an object detection operation considering the correlation between the image distortion area and the camera, including a data output unit that configures and displays a screen that simultaneously displays the above AVM image and the above final object detection result.

2. In paragraph 1, the artificial intelligence model An around view monitor image processing device that performs an object detection operation that considers the correlation between the image distortion area and the camera, characterized by repeatedly learning a large amount of learning data that has a camera image as an input condition and an object detection result as an output condition, and predicting and outputting the detection result of an object corresponding to the input image using an artificial intelligence method.

3. In paragraph 1, the correlation analysis unit An around view monitor image processing device that performs an object detection operation that takes into account the image distortion area and the correlation between cameras, characterized in that it further includes a function of identifying and defining a correlation by segmenting a first camera image for recognizing the presence or absence of an object by the image distortion area and the vehicle movement direction and a second camera image for detecting object details, by additionally considering the vehicle movement direction.

4. In the third paragraph, the correlation analysis unit An around view monitor image processing device that performs an object detection operation that takes into account the correlation between the image distortion area and the cameras, characterized in that it further includes a function of obtaining and providing a final object detection result by multi-stage analyzing the object detection result of the first camera image and the object detection result of the second camera image according to the correlation.

5. In an around view monitor image processing method for performing an object detection operation considering the image distortion area of ​​an AVM (Around View Monitor) image device and the correlation between cameras, A step of simultaneously acquiring multiple camera images through multiple cameras distributed across a vehicle; A step of aligning the above multiple camera images to create a single AVM (Around View Monitor) image; A step of obtaining object detection results corresponding to camera images through an artificial intelligence model in which the correlation between the input image and the object detection result is pre-learned, and then obtaining and providing the final object detection result by checking whether an object existing in the image distortion area is detected repeatedly through two camera images based on the correlation between the image distortion area and the camera; and An around view monitor image processing method for performing an object detection operation considering the correlation between the image distortion area and the camera, including the step of configuring and displaying a screen that simultaneously displays the above AVM image and the above final object detection result.

Images