Image processing apparatus

Abstract

Provided is an image processing device for obstacle detection and recognition, which can appropriately cluster (group) difference data of a subject moving obstacle even if a relative angle of the subject moving obstacle with respect to a host vehicle remains constant, and which can improve reliability of detection and recognition of the obstacle even if the difference data of the subject obstacle decreases as in a situation of a collision path. There are a difference data calculation section (141) that calculates difference data of a plurality of bird's-eye images different in time, a difference data clustering section (151) that clusters the difference data, a cluster combination section (161) that combines clusters with each other based on features of the bird's-eye images using a result of the difference data clustering section (151), and a cluster combination determination section (171) that determines a combination result of the cluster combination section (161) based on features of a subject object.

Description

Technical Field

[0001] The present invention relates to an in-vehicle image processing apparatus, for example, for image-based obstacle detection and recognition in the environment near a vehicle. Background Technology

[0002] In recent years, image-based object detection devices have been used to detect nearby moving and static objects (obstacles).

[0003] The aforementioned image-based object detection device can be used to detect intrusions or anomalies in surveillance systems, or in vehicle systems that assist in safe driving.

[0004] In vehicle applications, this device is designed to display the surrounding environment to the driver and / or detect moving or static objects (obstacles) around the vehicle, inform the driver of the potential risk of the vehicle colliding with the obstacle, and based on the decision, the system automatically stops the vehicle to avoid a collision with the obstacle.

[0005] As such object detection devices, for example, there are known devices that perform bird’s-eye view transformation on images of the area around a vehicle and use the difference between two time-different bird’s-eye view transformed images (hereinafter also referred to as bird’s-eye view images) to detect obstacles (see Patent Documents 1 and 2).

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent No. 6003986

[0009] Patent Document 2: Japanese Patent Application Publication No. 2016-134764 Summary of the Invention

[0010] The problem the invention aims to solve

[0011] However, the devices described in the aforementioned patent documents 1 and 2 use the difference between two bird's-eye view converted images that are different in time when detecting obstacles.

[0012] Therefore, when this system is used in situations where the vehicle and the obstacle are on the collision path, the vehicle and the obstacle (pedestrian, etc.) approach the collision point at the same speed, and thus the relative angle between the obstacle and the vehicle remains the same before they reach the collision point, the system produces the effect of minimizing the motion of the obstacle in the image acquired by the sensor mounted on the vehicle. This reduces the amount of difference between the images acquired by the sensor, which may lead to false detections or incorrect object detection results, thus reducing the reliability of the system.

[0013] The present invention was made in view of the above circumstances. The object of the present invention is to provide an image processing apparatus for obstacle detection and recognition, which can appropriately cluster (group) the differential data of the moving obstacle even when the relative angle of the moving obstacle to the vehicle remains constant, and can improve the reliability of obstacle detection and recognition even when the differential data of the obstacle is reduced, such as in the case of a collision path.

[0014] Technical means to solve the problem

[0015] To achieve the above objectives, the image processing apparatus of the present invention is an image processing apparatus for detecting surrounding objects reflected in an image, comprising: an image difference calculation unit that calculates difference data of multiple bird's-eye view images at different times; a difference data clustering unit that clusters the difference data; a cluster combining unit that uses the result of the difference data clustering unit to combine the clusters with each other based on the features of the bird's-eye view images; and a cluster combining determination unit that determines the combining result of the cluster combining unit based on the features of the object.

[0016] The effects of the invention

[0017] By employing this configuration, the image processing apparatus of the present invention combines differential data clusters using a cluster combining unit, and then determines that appropriate combining has been performed using a cluster combining determination unit. As a result, even if the differential data of the object obstacle is reduced, obstacle detection can still be performed by appropriately clustering (grouping) the differential data of the object moving obstacle. Therefore, the reliability and accuracy of obstacle detection and recognition can be improved. Thus, even in the case of a collision path, erroneous obstacle detection can be avoided.

[0018] According to the present invention, differential data of moving obstacles are clustered (grouped) and combined with differential data clusters to determine that appropriate clustering has been performed. Thus, even when the differential data of the object obstacle is reduced, the reliability and even accuracy of obstacle detection and recognition can be improved.

[0019] Other issues, structures, and effects not mentioned above will be clarified through the following description of the implementation methods. Attached Figure Description

[0020] Figure 1 This is a schematic configuration diagram of an image processing apparatus according to an embodiment of the present invention.

[0021] Figure 2 This is a diagram illustrating the image space acquired by the sensor and converted by the image conversion unit in an exemplary scenario, as well as the difference calculated by the image difference calculation unit.

[0022] Figure 3These are two different periods representing the movement of the vehicle and the pedestrian toward the point of collision: (a) represents the previous period, and (b) represents the current period.

[0023] Figure 4 It is a graph calculated from the difference image (difference data) illustrating the movement of the vehicle and the pedestrian toward the point of collision.

[0024] Figure 5 It is a differentially grouped graph that illustrates the movement of the vehicle and the pedestrian toward the point of collision. Detailed Implementation

[0025] Hereinafter, preferred embodiments of the image processing apparatus of the present invention will be described with reference to the accompanying drawings.

[0026] Reference Figures 1-5 The configuration and operation of the image processing apparatus 110 of this embodiment will be described. Although the illustrations are omitted, the image processing apparatus 110 is configured such that a CPU, RAM, ROM, etc. are connected via a bus, and the CPU controls the overall operation of the system by executing various control programs stored in the ROM.

[0027] In the configuration described below, two camera sensors (hereinafter sometimes simply referred to as cameras or sensors) are paired as a single vehicle-mounted stereo camera, corresponding to the sensing unit 111. However, this is not a limitation to using a single monocular camera as a device in other configurations that are used as the sensing unit 111.

[0028] Figure 1 This is a block diagram illustrating the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 110 of this embodiment is, for example, mounted on a vehicle (its own vehicle) and performs bird's-eye view conversion on surrounding images captured by a camera sensor (sensing unit 111), using the difference between multiple bird's-eye view converted images (bird's-eye view images) that are different in time (moment) to detect and identify obstacles (surrounding objects reflected in the image).

[0029] exist Figure 1 The image processing apparatus 110 includes: a sensing unit 111 comprising two camera sensors located at the same height, an image acquisition unit 121, an image conversion unit 131, an image difference calculation unit 141, a difference data clustering unit 151, a cluster combination unit 161, a cluster combination determination unit 171, an obstacle detection unit 181, and a control application processing unit 191.

[0030] (Image Acquisition Department)

[0031] The image acquisition unit 121 processes images acquired by one or both of the two camera sensors corresponding to the sensing unit 111 to adjust image characteristics for further processing. This processing may include, but is not limited to, image resolution adjustment that can reduce or enlarge the input image to change the resulting image size, and image region of interest selection that crops (trims) a specific region of the input image for further processing. The parameters used for image resolution adjustment and image region of interest selection can be controlled based on the current driving environment and conditions (speed, rotation speed, etc.).

[0032] (Image Conversion Department)

[0033] The image conversion unit 131 has the function of performing geometric image conversion on the image acquired and processed by the image acquisition unit 121 according to a specific geometric formula or conversion table that has been pre-calculated or adjusted. Such image conversion may include, but is not limited to, affine transformations such as rotation, scaling, cropping, and bird's-eye view conversion with reference to a flat ground.

[0034] For example, such as Figure 2 As shown, the acquired images CT21 and CT22 are converted by the image conversion unit 131, resulting in bird's-eye view converted image CT31 and bird's-eye view converted image CT32.

[0035] (Image Difference Calculation Unit)

[0036] The image difference calculation unit 141 has the function of calculating a difference image representing the difference between at least two images converted by the image conversion unit 131 at different times. Known methods, including but not limited to simple inter-pixel difference calculation and filter-based image difference calculation, can be applied to the difference calculation.

[0037] For example, such as Figure 2 As shown, the differential image CT41, which represents differential data P0D generated from pedestrian P0 and differential data OB0D generated from a specified object OB0, is calculated by the image differential calculation unit 141 based on the bird's-eye view converted image CT31 corresponding to the previous period and the bird's-eye view converted image CT32 corresponding to the current period, and uses the motion data of the vehicle itself to adjust / align the image before performing image differential calculation processing.

[0038] (Differential Data Clustering Department)

[0039] The differential data clustering unit 151 has the function of clustering (grouping) the pixels of the differential image calculated by the image differential calculation unit 141. Known clustering methods that consider the distance between points (pixels) can be used for this task (e.g., the K-means algorithm). Therefore, the result is a cluster of differential pixels (differential data) that are close to each other and can represent objects or obstacles on a road (see [link to documentation]). Figure 4 and Figure 5 as well as Figure 2 ).

[0040] (Cluster junction)

[0041] The cluster combining unit 161 has the function of combining (combining) clusters of differential data generated by the differential data clustering unit 151 with each other, based on the characteristics of cluster groups and differential images. For example, clusters that are in the same radial position relative to their own vehicles (on the differential image) are combined into a single group if they tend to belong to the same object obstacle and meet a set of specified conditions (e.g., specified size or number of pixels). Other methods for combining clusters with each other may also be included.

[0042] (Cluster-based decision-making unit)

[0043] The cluster combination determination unit 171 has the function of determining the reliability of multiple cluster combinations performed by the cluster combination unit 161. In principle, this determination is based on a set of conditions defined by the observable characteristics of the object obstacle (e.g., the size / shape that can be performed in the case of a pedestrian). As a result, this determination becomes the validity or cancellation of the combination operation performed by the cluster combination unit 161. If the combination operation is valid, the clusters (combinations) combined by the cluster combination unit 161 remain as they are. When the combination operation is cancelled, the clusters combined by the cluster combination unit 161 are separated, returning to their original state before the operation performed by the cluster combination unit 161.

[0044] As an example, a test of predefined clustering can be performed on a recognizer (e.g., a pedestrian recognizer) that the system focuses on detecting, corresponding to the category of object obstacles. The clustering can be validated or devalidated based on the resulting recognition score. Other methods can be used, such as comparing pixel brightness analysis with the clustering, or comparing edge analysis (pixels with abrupt changes in brightness in an image) with the clustering. Other methods for determining the clustering of multiple clusters may also be included.

[0045] (Obstacle Detection Department)

[0046] The obstacle detection unit 181 has the following functions: using the image acquired by the image acquisition unit 121, the difference image calculated by the image difference calculation unit 141, the result of the difference data clustering unit 151, and the result of the cluster combination unit 161 which is effective based on the result of the cluster combination determination unit 171, it detects the three-dimensional object reflected in the image and calculates its position.

[0047] Note that in this specification, "obstacle detection" refers at least to the processing that performs the following tasks: object detection (position within image space), object recognition (cars / vehicles, bicycles, pedestrians, poles, etc.), distance determination from the vehicle itself to the object in 3D space, and calculation of the object's speed.

[0048] (Control Application Processing Department)

[0049] The control application processing unit 191 has the function of determining the control application performed by the vehicle equipped with the image processing device 110 based on the obstacles identified by the obstacle detection unit 181.

[0050] Here, refer to Figure 3 (a) and (b) illustrate the application of the image processing device 110 as a system for monitoring the surroundings of the vehicle V. Figure 3 (a) and (b) represent Figure 3 (b) in Figure 3 The scenes that occur after (a) are divided into different time frames using known periods. The scenes seen above are represented by CT11 and CT12, the scenes seen in the images acquired by the sensor corresponding to the sensing unit 111 are represented by CT21 and CT22, and the images of the scenes converted and acquired by the image conversion unit 131 are represented by CT31 and CT32.

[0051] Under the following conditions, vehicle V and pedestrian P1 move towards the collision point CP1 (intersection, etc.) at the same speed. Therefore, even if the distances shown as X1 (P1~CP1), Z1 (V~CP1), and X2 (P1~CP1), Z2 (V~CP1) decrease as vehicle V and pedestrian P1 approach the collision point CP1, as... Figure 3 In (a) angle (θ1) and Figure 3 As shown in angle (θ2) in (b), there is also a certain relative angle between vehicle V and pedestrian P1. The positional change of pedestrian P1 relative to vehicle V is clearly visible in the coordinates of CT11 and CT12, but the influence of the certain relative angle between pedestrian P1 and vehicle V is reflected in the images acquired by the sensors (sensing unit 111) shown by CT21 and CT22 and the minimum motion shown by the corresponding transformations of both sides shown by CT31 and CT32, and subsequently affects the differential image calculated by image differential calculation unit 141.

[0052] (Processing example of the image difference calculation unit)

[0053] Figure 4 An example of the result from the image difference calculation unit 141 is shown. Figure 4In this process, data from the older period (CT31) is first aligned with data from the newer period (CT32), and image difference calculation is performed based on known methods, such as filter-based image difference. For example, a simple method based on neighboring pixel difference filter scores can be implemented to calculate the difference between two bird's-eye view images. Neighboring pixel pairs of a reference pixel in a filter (e.g., obliquely separated) are compared, their brightness differences are calculated, and then the filter results for two different images on the same reference pixel are compared to calculate the difference count of the final value. In this simple method, the higher the difference count of the value, the greater the difference between the two images, and thus such a difference can be correlated with motion within the image. Therefore, setting a threshold for the difference count of the value allows it to be considered a reliable difference, and noise removal can be performed. After noise removal, the resulting difference image is then ready for further processing.

[0054] Additionally, the configuration used in pixel comparison is called a filter. The shape, number, and comparison direction (e.g., tilted, horizontal, vertical, etc.) of the analyzed pixels that define the filter can be adjusted based on the application.

[0055] An exemplary result of the difference image before noise removal is represented by CT41, and it can be seen that, in the scenario described above, only a specific part of pedestrian P1 appears to move between the data acquired in two different periods (in this case, the upper and lower body, and some pixel parts in the middle with very low difference counts, for example, low difference counts). Additionally, in CT41, the size of the quadrilateral corresponding to a pixel represents the magnitude (strength) of the difference between the images.

[0056] (Processing examples of differential data clustering, cluster combination, and cluster combination determination)

[0057] Next, based on Figure 5 The tasks performed by the differential data clustering unit 151, the cluster combining unit 161, and the cluster combining determination unit 171 are explained. The result of the task performed by the differential data clustering unit 151 (here, after noise removal) is represented by CT411. Based on a clustering method that considers the distance (in image space) between cluster center candidates between each differential pixel, the differential data pixels are grouped into two different clusters in the described scenario, represented by cluster (gr1) (e.g., corresponding to the upper body of pedestrian P1) and cluster (gr2) (e.g., corresponding to the lower body of pedestrian P1).

[0058] The result of the task performed by the clustering unit 161 is represented by CT412. Clusters (gr1) and (gr2) are combined into a combined cluster (hereinafter also referred to as cluster group) (gr11) based on the characteristics of the cluster group and the difference images. This characteristic may include, but is not limited to, the amount of low difference count pixels (i.e., pixels with a brightness difference of less than a specified value between multiple bird's-eye view images) existing between the combined clusters (s12) before noise removal (see...). Figure 4 The similarity of at least one of the following: CT41), the space (spacing) between candidate clusters, and the angle of each cluster relative to the vehicle position (more specifically, the camera position) (represented by the bottom center of the coordinates of CT412). A cluster (gr11) can be considered as part of the same object obstacle standing on the ground at a predetermined distance from the vehicle V in the space represented by the image transformed by the image transformation unit 131 (a bird's-eye view image transformation based on a flat ground).

[0059] Then, the cluster combination decision unit 171 takes the cluster group (gr11) as input for the decision processing. For example, it calculates the bounding box corresponding to the size and position of the cluster group (gr11) in the image captured by the sensor (e.g., Figure 3 The bounding box in CT22 (b) is processed using one or more previously trained recognizers to identify object obstacles (adult pedestrians, child pedestrians, bicycles, etc.), and the results of one or more recognizers are used to determine whether the cluster group (gr11) is a valid cluster combination or whether the cluster group (gr11) is canceled and separated into the original clusters (gr1) and cluster (gr2). Another exemplary metric for determining the result of cluster combination is, for example, the evaluation of the brightness difference between the combined groups and the space (interval) between them on the image converted by image converter 131. Other methods for evaluating the combination result are not only combinations of these methods, but also methods used in the same way as described above. The result of the task performed by the cluster combination determination unit 171 is represented by CT413 (when valid) and CT414 (when canceled), in which the cluster group (gr11) is maintained as is, and in CT414, the cluster group (gr11) is separated into the original clusters (gr1) and cluster (gr2).

[0060] By using the results of the cluster combined with the determination unit 171, it is possible to detect and calculate the position of the three-dimensional object reflected in the image near the collision point CP1 (intersection, etc.), and thus determine the control application to be performed by the vehicle.

[0061] As mentioned above, Figure 1 The image processing apparatus 110 for obstacle detection and recognition shown in this embodiment includes:

[0062] The sensing unit 111 is capable of capturing an image of the scene in front of the device on which the device is installed;

[0063] The image acquisition unit 121 processes the image acquired by the sensing unit 111 and adjusts its characteristics (including but not limited to image size, image resolution, and image region of interest).

[0064] The image conversion unit 131 performs a desired geometric image conversion on the image acquired and processed by the image acquisition unit 121;

[0065] The image difference calculation unit 141 calculates a difference image, which represents the difference between at least two images after being acquired and processed by the image acquisition unit 121 and converted by the image conversion unit 131.

[0066] The differential data clustering unit 151 performs data clustering (grouping) at the pixel level on the differential image calculated by the image differential calculation unit 141 using a prescribed clustering method, and generates a list of clusters (groups) that may represent obstacles in the vicinity of the vehicle.

[0067] Cluster combining unit 161 performs data combining on the data calculated by differential data clustering unit 151 so as to combine two or more clusters into a single cluster group according to their characteristics.

[0068] The cluster combination determination unit 171 determines the cluster combination result calculated by the cluster combination unit 161 based on a prescribed set of conditions, and determines whether the combination result is valid.

[0069] The obstacle detection unit 181 uses images acquired by the image acquisition unit 121, and the results of the differential data clustering unit 151, the cluster combination unit 161, and the cluster combination determination unit 171 to perform object detection and object recognition; and

[0070] The control application processing unit 191 determines the control application to be executed by the device equipped with the image processing device 110 based on the current state, which may include at least the output from the obstacle detection unit 181.

[0071] That is, the image processing apparatus 110 of this embodiment includes: an image difference calculation unit 141, which calculates difference data of multiple bird's-eye view images at different times; a difference data clustering unit 151, which clusters the difference data; a cluster combining unit 161, which uses the result of the difference data clustering unit 151 to combine the clusters with each other based on the features of the bird's-eye view images; and a cluster combining determination unit 171, which determines the combining result of the cluster combining unit 161 based on the features of the object.

[0072] By employing the above processing, even in the case of idle space between candidate groups due to the lack of observed motion in the image plane, all differential data corresponding to pedestrian P1 can be grouped. Therefore, even when vehicle V and pedestrian P1 move towards the collision point CP1 at the same speed, pedestrian P1 can be correctly detected and identified, improving the stability and reliability of the system.

[0073] The above describes the configuration and operation of the image processing apparatus 110 for obstacle detection and recognition according to this embodiment. In areas such as intersections, the image processing apparatus 110 can improve the reliability of obstacle recognition by appropriately clustering (grouping) the differential data of moving obstacles, even if the relative angle of the moving obstacle to the vehicle remains constant. Simultaneously, it can reduce the false object detection rate, improve the accuracy of object detection, and enhance driving safety.

[0074] While the presently intended preferred embodiments of the invention have been described, various modifications may be made to these embodiments, and all modifications are intended to fall within the scope of the true spirit and scope of the invention.

[0075] Furthermore, the present invention is not limited to the embodiments described above, but also includes various modifications. For example, the embodiments described above are for the purpose of explaining the present invention in a detailed manner for ease of understanding, and are not necessarily limited to having all the described configurations.

[0076] Furthermore, the aforementioned components, functions, processing units, and processing methods can also be implemented in hardware, for example, by designing some or all of them using integrated circuits. Alternatively, the aforementioned components and functions can be implemented in software by a processor interpreting and executing programs that perform their respective functions. The programs, tables, files, and other information implementing these functions can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

[0077] Furthermore, while the designation of control lines and information lines is considered necessary for the description, not all control lines and information lines may necessarily be shown on the product. In reality, it can be assumed that almost all components are interconnected.

[0078] Symbol Explanation

[0079] 110 Image Processing Device

[0080] 111 Sensing Unit

[0081] 121 Image Acquisition Department

[0082] 131 Image Conversion Unit

[0083] 141 Image Difference Calculation Unit

[0084] 151 Differential Data Clustering Department

[0085] 161 Cluster junction

[0086] 171 Cluster Combination Determination Unit

[0087] 181 Obstacle Detection Department

[0088] 191 Control Application Processing Unit.

Claims

1. An image processing apparatus for detecting surrounding objects reflected in an image, characterized in that it comprises: The image difference calculation unit calculates the difference data of multiple bird's-eye view images at different times; The differential data clustering unit clusters the differential data. The cluster combining section uses the results of the differential data clustering section to combine clusters together based on the features of the bird's-eye view; as well as The cluster combination determination unit determines the combination result of the cluster combination unit based on the characteristics of the object. The features of the bird's-eye view image include the number of pixels whose brightness difference between the plurality of bird's-eye view images present in the cluster is less than a predetermined value.

2. The image processing apparatus according to claim 1, characterized in that, The cluster combination determination unit determines the combination result of the cluster combination unit by processing the data within the bounding box corresponding to the size and position of the cluster group combined in the bird's-eye view image through a recognizer used to identify the object.

3. The image processing apparatus according to claim 1, characterized in that, If the joining operation performed at the cluster joining section is valid, the cluster joining determination section maintains the joining of the clusters joined by the cluster joining section. If the joining operation performed at the cluster joining section is cancelled, the cluster joining determination section separates the clusters joined by the cluster joining section back to their original state.

Images