Clutter determination processing device, information processing device, program, and clutter determination processing method

Abstract

A clutter determination processing unit (12) according to the present disclosure comprises: a clutter image processing unit (128) that includes a provisional size determining unit (122) for determining a provisional size of a target object corresponding individually to each detection point acquired by a distance measuring sensor (2) within a preset clutter determination region, on the basis of a reception signal from the distance measuring sensor (2) for each detection point, and that, on the basis of the provisional size and the position of each detection point, analyzes an image including a provisional region projected onto a captured image from an imaging sensor (3), as a target object region corresponding to each detection point of the distance measuring sensor (2); a clutter determining unit (126) that uses the analysis results from the clutter image processing unit (128) to determine whether or not each detection point of the distance measuring sensor (2) is clutter; and a target output unit (127) that removes the detection points that the clutter determining unit (126) has determined to be clutter from all of the detection results acquired by the distance measuring sensor (2), and outputs the result.

Description

Clutter determination processing device, information processing device, program, and clutter determination processing method

[0001] The present disclosure relates to a clutter determination processing device, an information processing device, a program, and a clutter determination processing method for determining clutter in a distance measuring sensor.

[0002] 2. Description of the Related Art Conventionally, there are known techniques for detecting objects using sensors, such as radar, light detection and ranging (LiDAR), and time-of-flight (TOF) sensors, and imaging sensors for acquiring images.

[0003] When using a ranging sensor, not only the target object but also reflected waves from the ground, sea surface, rain, snow, and other background elements are observed as unwanted waves. These unwanted waves, called clutter, interfere with and disrupt target detection and observation. Therefore, removing clutter is a challenge for improving the accuracy of object detection using a ranging sensor. Furthermore, in object detection for purposes such as safe driving assistance and collision prevention, it is desirable to exclude the background from the scope of warnings or alerts. However, because ranging sensors cannot acquire color information such as the brightness, edges, and image gradients of the object and background, it is difficult to distinguish the background. Furthermore, clutter caused by stationary ground objects, such as static ground objects, has the same speed as the stationary target object, making it difficult to distinguish between clutter and the target object even using speed information obtained from the ranging sensor. For example, Figure 21 shows an example in which ground clutter (false detection) and an obstacle (correct detection) are detected at the same position in the same environment, and the ranging sensor is unable to distinguish between the two.

[0004] Meanwhile, there is also known a technology that improves the accuracy of object detection, recognition, and discrimination by combining a distance measurement sensor and an imaging sensor and using them as a composite sensor. Patent Document 1 discloses a technology that improves the accuracy of object detection, recognition, and discrimination by using a radar and a stereo camera. Patent Document 1 generates a composite map by combining a camera distance map obtained from three-dimensional information acquired by the stereo camera with a radar distance map obtained from radar detection results, and analyzes the composite map information to remove noise, i.e., clutter, caused by unnecessary reflected waves from the ground or walls.

[0005] JP 2012-154731 A

[0006] However, the technology described in Patent Document 1 requires expensive sensors such as stereo cameras and LiDAR to generate three-dimensional images for identifying and removing clutter, which results in a high processing load.

[0007] The present disclosure has been made in view of the above, and has an object to provide a clutter determination processing device that can determine clutter with a simple configuration and low-load processing.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the clutter determination processing device disclosed herein has a provisional size determination unit that determines the provisional size of an individual corresponding target for each detection point acquired by a ranging sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the ranging sensor, a clutter image processing unit that analyzes an image including the provisional area projected onto an image captured by the imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point, a clutter determination unit that uses the analysis results of the clutter image processing unit to determine whether each detection point of the ranging sensor is clutter, and a target output unit that removes detection points that the clutter determination unit has determined to be clutter from all detection results acquired by the ranging sensor and outputs them.

[0009] The clutter determination processing device according to the present disclosure has an effect of being able to determine clutter with a simple configuration and low-load processing.

[0010] FIG. 1 is a diagram showing an example of the functional configuration of an object detection system according to a first embodiment. FIG. 2 is a diagram showing an example of the functional configuration of a clutter determination processing unit according to a first embodiment. FIG. 1 is a diagram showing the results of a deep learning (Deep Learning Transform) process; FIG. 2 is a diagram illustrating an example of an area on a captured image where radar clutter detection is expected; FIG. 3 is a diagram showing an example of correspondence information between received signal levels and provisional sizes in embodiment 1; FIG. 4 is a diagram showing an example of a provisional area in which provisional sizes and positions of detection points are projected onto a captured image in embodiment 1; FIG. 5 is a flowchart showing an example of a clutter discrimination processing procedure in a clutter determination processing unit of embodiment 1; FIG. 6 is a diagram for explaining the effect of using deep learning; FIG. 7 is a diagram for explaining class classification in embodiment 1; FIG. 8 is a diagram showing an example of area division in embodiment 1;

[0011] A clutter determination processing device, an information processing device, a program, and a clutter determination processing method according to embodiments will be described in detail below with reference to the accompanying drawings.

[0012] First Embodiment. FIG. 1 is a diagram illustrating an example of the functional configuration of an object detection system according to a first embodiment. The object detection system of this embodiment includes an information processing device 1, a ranging sensor 2, and an imaging sensor 3. Hereinafter, a combination of the ranging sensor 2 and the imaging sensor 3 will also be referred to as a composite sensor. The object detection system detects surrounding objects and transmits object information (integrated information, described later) including the position, speed, and type information of the detected object, a target ID (identifier), and the like, to a display control unit 5. The display control unit 5 generates a display image for displaying the object information on a display device (not shown) and outputs the generated display image to the display device. Furthermore, without being limited to the above, the object detection system may output the detection information to a vehicle control device or the like via an interface (I / F) such as CAN FD (registered trademark) (Controller Area Network with Flexible Data rate).

[0013] The object detection system is mounted on, for example, a vehicle. When the object detection system is mounted on a vehicle and used for vehicle control, the information processing device 1 transmits object information to a vehicle control unit 4. The vehicle control unit 4 controls the vehicle using the object information. Note that the object detection system may select whether or not to output, depending on the needs of the system. In this case, the object information does not need to be transmitted to the vehicle control unit 4, and the vehicle control unit 4 does not need to be provided. In this way, the object detection system is used, for example, for vehicle control in a vehicle, but the use of the object detection system is not limited to this. Furthermore, the object detection system may be mounted on a vehicle and used for purposes other than vehicle control, such as displaying object information.

[0014] The distance measurement sensor 2 detects the position of an object by emitting microwaves, millimeter waves, infrared rays, visible light, or the like to the object and receiving a reflected signal from the object. The distance measurement sensor 2 is, for example, a pulse-type radar, a radar that measures distance from the reflection propagation time such as ToF, or a radar such as an FMCW (Frequency Modulated Continuous Wave) or FCM (Fast Chirp Modulation) type that can also detect Doppler velocity, but is not limited to these.

[0015] The ranging sensor 2 includes a sensor unit 21 and a control unit 22. The sensor unit 21 includes, for example, a transmitting antenna and a receiving antenna. The transmitting antenna emits electromagnetic waves toward an object and the receiving antenna receives reflected waves from the object. The control unit 22 performs signal processing on the reflected waves received by the sensor unit 21 to detect the position of a detection point where an object is assumed to exist and the object's speed, and transmits the detection results to the information processing device 1. In the following description, it is assumed that one detection point corresponds to one target detected by the ranging sensor 2. The detection results may include, in addition to distance, direction, and speed, target ID, tracking information, and received signal level. The received signal level reflects the reflection intensity corresponding to the target's distance and radar cross section (RCS), and can therefore be used as a rough indicator for classifying the type of target. Furthermore, by observing fluctuations in this information over a certain period of time, it is possible to capture the dynamic characteristics of the target. Since a general ranging sensor can be used as the ranging sensor 2, details of the configuration and operation of the ranging sensor 2 will be omitted. In FIG. 1, the control unit 22 that performs these signal processes is shown inside the distance measuring sensor 2, but the same function may be provided inside the information processing device 1, and is not limited to the illustration.

[0016] The imaging sensor 3 is an imaging device capable of capturing images over a range corresponding to the coverage area (field of view / detection range) of the ranging sensor 2, i.e., the detection range in which the ranging sensor 2 can detect an object. The imaging sensor 3 includes an imaging unit 31 and a control unit 32. The imaging unit 31 is, for example, a monocular camera equipped with a lens, an imaging element, etc. The imaging element of the imaging unit 31 detects light, converts the detected light into an electrical signal, and outputs the electrical signal to the control unit 32. The control unit 32 generates a two-dimensional image (hereinafter also referred to as a captured image) based on the electrical signal and transmits the generated captured image to the information processing device 1. The imaging unit 31 detects, for example, visible light, but the wavelength of light detected by the imaging unit 31 is not limited to this. Furthermore, although not shown, the image quality of the image is controlled and adjusted as necessary depending on the imaging environment and conditions, such as illuminance. These controls and adjustments may be performed either within the imaging sensor 3 or within the information processing device 1.

[0017] The information processing device 1 detects, recognizes, and distinguishes objects using the detection results received from the ranging sensor 2 and the captured images received from the imaging sensor 3, and performs necessary processing such as merging and integrating each piece of detection information, and transmits object information such as the position, speed, and type information of the detected, recognized, and distinguished objects to the display control unit 5.

[0018] The information processing device 1 includes a first transmitting / receiving unit 11, a clutter determination processing unit 12, a sensor processing unit 13, a second transmitting / receiving unit 14, a vehicle communication unit 15, and a storage unit 16. Note that if the information processing device 1 is not mounted on a vehicle, the vehicle communication unit 15 does not need to be provided.

[0019] The first transmitter / receiver 11 receives detection results from the ranging sensor 2 and outputs the received detection results to the clutter determination processor 12. It also receives captured images from the imaging sensor 3 and outputs the received captured images to the clutter determination processor 12. These input / output interfaces may use serial communications such as I2C (registered trademark), SPI (Serial Peripheral Interface), or USB (registered trademark) (Universal Serial Bus), or MIPI-CSI, depending on the detection information and configuration of each sensor. The first transmitter / receiver 11 may control the ranging sensor 2 and the imaging sensor 3 based on the output results from the ranging sensor 2 and the imaging sensor 3. For example, the first transmitter / receiver 11 controls sensor operating conditions such as the transmission output and receiving sensitivity of the ranging sensor 2, and imaging conditions such as the exposure time, gain, and shutter timing of the imaging sensor 3. The control function of the imaging sensor 3 may be implemented by, for example, an image signal processor (ISP).

[0020] The clutter determination processing unit 12 performs class classification to classify the detected points of the ranging sensor 2 into background and non-background, using the two-dimensional captured image associated with the detected points of the ranging sensor 2, and thereby removes from the detection target those detected points included in the detection results received from the ranging sensor 2 that correspond to the background. When the object detection system of this embodiment is used to detect objects for vehicle control, the background includes, for example, the ground, a snowy surface, a manhole, grass, trees, etc., but is not limited to these and may include the surface of the sea, etc., and may be determined depending on the application of the object detection system of this embodiment.

[0021] The clutter determination processing unit 12 outputs the detection results (distance, direction, speed, target ID, tracking information, received signal level, etc.) of the detection point corresponding to the detection target, i.e., the target, from the detection results of the distance measurement sensor 2 to the sensor processing unit 13 as first object information. The clutter determination processing unit 12 also outputs the captured image acquired by the imaging sensor 3 and input from the first transmitting / receiving unit 11 to the sensor processing unit 13. The captured image may also be input to the sensor processing unit 13 via a route other than this.

[0022] The sensor processing unit 13 performs an object detection process to detect the type and position of an object using the captured image, and generates the position and type of the object as second object information as a result of the object detection process. The object detection process is a process of detecting the position and type of an object from the captured image. The object detection process is performed using machine learning, such as a convolutional neural network (CNN), a random forest, or a support vector machine, but is not limited to these. Note that, while an example in which the sensor processing unit 13 performs the object detection process to generate the second object information is described here, the object detection process may also be performed by the image analysis unit 125, an object recognition unit (not shown) separate from the image analysis unit 125, or the control unit 32 of the imaging sensor 3.

[0023] The sensor processing unit 13 also integrates the first object information and the second object information, outputs the integrated information resulting from the integration to the second transceiver unit 14 and the vehicle communication unit 15, and stores the integrated information in the storage unit 16. For example, the sensor processing unit 13 uses the positions in the first object information and the positions in the second object information to extract identical objects from among the objects detected by the ranging sensor 2 and the objects detected in the captured image, and associates the first object information and the second object information corresponding to the identical objects for each extracted object. The sensor processing unit 13 outputs the associated first object information and second object information as integrated information to the second transceiver unit 14 and the vehicle communication unit 15 and stores the combined information in the storage unit 16. Note that the integrated information is not limited to the above example, and may be generated using the first object information and the second object information according to the application. Furthermore, depending on the situation or operation, the first object information and the second object information may be output separately without combining the information. In addition, the memory unit 16 may be used not only to store the above-mentioned integrated information, but also as a buffer for temporarily storing input information for processing and information generated during the process, or for storing history and log information recorded during processing and operations.

[0024] In addition, in the configuration of Figure 1, the clutter determination processing unit 12 removes from the detection target those detection points that correspond to the background among the detection points included in the detection results received from the ranging sensor 2, and then inputs the first object information and the second object information to the sensor processing unit 13 to perform integration processing.However, after the sensor processing unit 13 extracts and matches identical objects from the first object information and the second object information, it is also possible to perform clutter determination processing on the remaining detection points of the ranging sensor 2 for which a determination of identity or match cannot be made.

[0025] The second transmitter / receiver 14 outputs the integrated information as object information to the display controller 5. The display controller 5 outputs a display image generated to display the object information on a display device (not shown) to the display device.

[0026] The vehicle communication unit 15 transmits the integrated information as object information to the vehicle control unit 4. The vehicle control unit 4 controls the vehicle using the object information. As described above, the object detection system does not need to be used for vehicle control, and the vehicle communication unit 15 and the vehicle control unit 4 may not be provided.

[0027] Next, details of the clutter determination processing unit 12, which is the clutter determination processing device of this embodiment, will be described. Fig. 2 is a diagram showing an example of the functional configuration of the clutter determination processing unit 12 of this embodiment. In addition to the clutter determination processing unit 12, Fig. 2 also shows the distance measurement sensor 2, the image sensor 3, and the sensor processing unit 13. Note that, as shown in Fig. 1, for example, a first transmission / reception unit 11 is provided between the distance measurement sensor 2 and the image sensor 3 and the clutter determination processing unit 12, but the first transmission / reception unit 11 is not shown in Fig. 2.

[0028] The clutter determination processing unit 12 includes a clutter region determination unit 121, a provisional size determination unit 122, a region projection unit 123, an image extraction unit 124, an image analysis unit 125, a clutter determination unit 126, and a target output unit 127. The clutter region determination unit 121, the provisional size determination unit 122, the region projection unit 123, the image extraction unit 124, and the image analysis unit 125 constitute a clutter image processing unit 128.

[0029] The clutter region determination unit 121 receives the detection results of the ranging sensor 2 from the first transceiver 11 and uses the detection results to determine, for each detection point, whether the detection point detected by the ranging sensor 2 is within a clutter discrimination region (hereinafter also referred to as a discrimination region) that has been set in advance as a region where clutter detection is expected. If there is no detection point detected by the ranging sensor 2 within the discrimination region, the clutter region determination unit 121 outputs the detection result as is to the sensor processing unit 13 as first object information. If there is a detection point detected by the ranging sensor 2 within the discrimination region, the clutter region determination unit 121 notifies the provisional size determination unit 122 of all detection points present within the discrimination region, and outputs the detection results of the ranging sensor 2 to the provisional size determination unit 122 and the clutter determination unit 126.

[0030] The discrimination region is a portion of the entire area detectable by the ranging sensor 2, and is an area where clutter, i.e., unwanted waves due to the background, are expected to be detected. For example, when the ranging sensor 2 is mounted on a vehicle, the discrimination region is a portion of the entire pixel area of ​​the captured image where the background, such as the ground, is expected to be present, and is set based on, for example, the height, depression angle, horizontal orientation, etc. of the composite sensor.

[0031] Furthermore, without being limited to the above, if at least one of the conditions of the height and depression angle of the composite sensor, i.e., a condition that affects the detection range of the ranging sensor 2, is set as an operating condition and the operating condition is changeable, the clutter area determination unit 121 may determine the above area according to the operating condition. Note that, hereinafter, a detection point at which the ranging sensor 2 determines that an object exists in the detection result due to the ranging sensor 2 receiving clutter, i.e., unwanted waves, may also be referred to as clutter.

[0032] For example, Figure 3 shows the distance FFT results of the detection signals of the ranging sensor (radar) 2 when the composite sensor is set at a certain height and depression angle from the ground. In Figure 3, areas such as area A where the level is high compared to the noise level of the radar receiver represent a group of received signals due to reflections from the ground. When the level of these received signals exceeds a certain value, a peak is detected, similar to the reflected signal from the target, and this becomes a detection point (clutter detection point).

[0033] As the depression angle increases, the distribution of area A tends to shift toward the shorter distance and the level to increase. Because the clutter profile (distance direction and level distribution) also changes depending on the ground conditions (asphalt, dirt, grass, undulations, etc.), it is desirable to appropriately set the clutter profile based on the installation conditions of the composite sensor and the environmental conditions. Furthermore, the area where radar clutter detection is expected is defined in physical coordinates, for example, X = ±2 m, Y = 2 to 6 m, with the sensor installation position as the origin. However, when defining the area on the captured image, it is set as area B, which is a sector-shaped rectangle due to the effects of perspective and optical lens distortion, as shown in Figure 4. Figure 4 illustrates an example of an area on the captured image where radar clutter detection is expected. However, as long as the physical discrimination area is covered, the above is not the only limitation, and for convenience, it is acceptable to treat the pixels as a rectangle that encompasses area B.

[0034] The provisional size determination unit 122 determines the provisional size of the range measurement target, which is an object detected by the range measurement sensor 2, using the detection points notified by the clutter region determination unit 121 and the detection result of the range measurement sensor 2. The provisional size determination unit 122 notifies the area projection unit 123 of the provisional size, which is the area where the range measurement target is expected to exist, using the determined provisional size, and outputs the detection result of the range measurement sensor 2 (distance, direction: position information) to the area projection unit 123. In detail, the provisional size determination unit 122 may set a predetermined fixed size as the provisional size, or may determine the provisional size according to a reflection cross-section area such as an RCS expected from a received signal level, or may use information such as the received signal of the range measurement sensor 2 or the speed obtained as a processing / detection result to determine the provisional size. That is, the provisional size determination unit 122 may determine the provisional size by referring to the received signal (raw data) itself, or based on detection information such as speed and signal strength obtained by analyzing the raw data, or further based on time variation information of these received signals and detection results, target ID information obtained by target tracking processing, etc. The provisional size may also be determined by methods other than these.

[0035] Furthermore, the provisional size determination unit 122 may classify the target into levels according to the reflection cross section of the RCS or the like estimated from the received signal level and determine a different provisional size for each level, or may set a provisional size assuming that the target is a moving target (car, bicycle, person) when the target has a relative speed with respect to the ranging sensor 2, or may exclude the target from discrimination as a target different from ground clutter. For example, correspondence information between the received signal level and the provisional size may be determined in advance, and the provisional size determination unit 122 may determine the provisional size using the received signal level and the correspondence information.

[0036] Fig. 5 is a diagram showing an example of correspondence information between received signal levels and provisional sizes in this embodiment. In the example shown in Fig. 5, the correspondence information includes received signal levels and provisional sizes. Note that, for reference, Fig. 5 also shows examples of discrimination targets, but the correspondence information used in processing does not necessarily include the examples of discrimination targets. Note that, in Fig. 5, received signal levels are indicated as large, medium, and small, but these may be defined as specific ranges of numerical values, for example.

[0037] Furthermore, the provisional size determination unit 122 may determine the provisional size by assuming the type of target based on time variation information of the received signal and target tracking processing. Alternatively, the provisional size determination unit 122 may determine the provisional size using multiple pieces of information based on the received signal, such as determining the provisional size using the received signal level and time variation information of the received signal.

[0038] The area projection unit 123 determines a provisional area, which is an area obtained by projecting onto the captured image a provisional area indicating the virtual size and area of ​​the target corresponding to the detection point within the discrimination area. Specifically, the area projection unit 123 identifies the position of the target on the captured image by performing coordinate transformation of the position of the detection point of the ranging sensor 2 using the provisional size input from the provisional size determination unit 122 and the detection result (distance, direction: position information) of the ranging sensor 2. The provisional size is an area in which the target is assumed to exist in the coordinate system of the ranging sensor 2. Specifically, the area projection unit 123 performs coordinate transformation of the position of the detection point to determine an area (provisional area) on a two-dimensional plane corresponding to the captured image when the provisional size of the target is projected onto the two-dimensional plane. This coordinate transformation is a coordinate transformation from the coordinate system of the ranging sensor 2 to the coordinate system of the imaging sensor 3, i.e., the coordinate system of the captured image. For example, the area projection unit 123 performs coordinate transformation on the position of the detection point, and determines a provisional area, which is a position on the captured image, using the result of the coordinate transformation and the provisional size. Through this coordinate transformation, the position and area of ​​the detection point are expressed as a position and area on the captured image corresponding to the target detected by the ranging sensor 2. The area projection unit 123 notifies the image extraction unit 124 of the provisional area, which is obtained by projecting the position and provisional size of the target, expressed in the coordinate system of the captured image, onto the captured image.

[0039] FIG. 6 shows an example of a provisional area in which the provisional size and the positions of the detection points are projected onto a captured image in this embodiment. The left side of FIG. 6 shows an example of a captured image in which an area including a road is captured, and the right side of FIG. 6 shows the detection points of the ranging sensor 2 and the provisional area in the same range as the captured image shown in FIG. 6. The captured image shown in FIG. 6 includes objects other than the background, such as vehicles, cardboard boxes, and traffic cones, as well as the background road (ground, white lines). The ranging sensor 2 receives and detects reflected waves from these objects. Points 501-1 to 501-4 indicate the detection points of the ranging sensor 2, and areas 502-1 to 502-4 indicate provisional areas corresponding to the provisional sizes corresponding to points 501-1 to 501-4, respectively. For example, area 502-1 corresponds to a traffic cone, area 502-2 corresponds to a car, area 502-3 corresponds to a cardboard box, and area 502-4 corresponds to the ground, i.e., ground clutter. 5 and 6 , the shape of the provisional area is determined as, for example, a square, a rectangle, or the like. The shape of the provisional area is not limited to these. As shown in FIG. 6 , the provisional area may be determined with the base starting point so that the center of the base of the provisional area is the detection point of the ranging sensor 2. The provisional area may also be determined with the central starting point so that the center of the provisional area is the detection point of the ranging sensor 2. The correspondence between the detection points of the ranging sensor 2 and the provisional area is not limited to these and may be set according to the operation.

[0040] The image extraction unit 124 receives the captured image captured by the imaging sensor 3 from the first transmission / reception unit 11, and extracts an image of the provisional area, i.e., an image corresponding to the provisional area, from the captured image using the provisional area notified by the area projection unit 123, and outputs the extracted image to the image analysis unit 125. For example, the image extraction unit 124 cuts out (crops) an image portion corresponding to the provisional area in the clutter determination area from the captured image, and further resizes the cropped image if necessary to generate an extracted image.

[0041] The image analysis unit 125 sets a provisional size of the target at the detection point acquired by the ranging sensor 2 in the clutter discrimination area, and analyzes an image including a provisional area corresponding to the provisional size of the target at each detection point of the ranging sensor 2 projected onto the image captured by the imaging sensor 3. For example, the image analysis unit 125 classifies the image corresponding to the provisional area input from the image extraction unit 124 (a processing target area, which is at least a part of the captured image). In detail, the image analysis unit 125 classifies the image received from the image extraction unit 124 into background and non-background, and outputs the classification result to the clutter determination unit 126. Note that the classification result includes information indicating the position of the classified area. The image analysis unit 125 classifies the image into two or more classes, including at least background and non-background classes. For example, the classification result indicates a binary classification result, i.e., a background class corresponding to the background and a foreground class corresponding to non-background. Note that this is not limited to this, and the detection target (background and non-background) may be further classified into multiple classes. However, the present invention is not limited to this, and the background may be further classified into a plurality of classes. That is, classes corresponding to the background and non-background may be defined as a plurality of classes according to the types of the background and non-background, and the image analysis unit 125 may classify the image received from the image extraction unit 124 into two or more classes including a plurality of classes corresponding to the background and non-background. For example, the background may be classified into two or more classes of ground, manhole, grass, and trees. Details of the processing by the image analysis unit 125 will be described later. Each class corresponds to a type of the background or non-background.

[0042] The image extraction unit 124 may also extract from the captured image an image including not only the provisional area in which the target is captured but also a predetermined area including the periphery of the target outside the provisional area, and output these extracted images to the image analysis unit 125. For example, if the provisional area obtained by projecting the provisional size of the detection point of the ranging sensor 2 determined by the provisional size determination unit 122 or the area projection unit 123 onto the captured image is misaligned with the actual target (the obstacle captured in the captured image = the correctly detected target), the image analysis unit 125 may erroneously determine the target as background and classify it because only a portion of the target is captured in the image. Furthermore, if the provisional area obtained by projecting the provisional size of the detection point of the ranging sensor 2 is small compared to the actual target (the entire view of the obstacle captured in the captured image), the image analysis unit 125 may not be able to capture the outline or boundary of the obstacle target, resulting in an incorrect classification. In contrast to the above, by adding an area including the surrounding area outside the provisional area to the extracted image and inputting it into the image analysis unit 125, it is possible to capture a larger image of the actual target (obstacle = correct detection), and by extracting the boundary between the obstacle (correct detection) and the surrounding area such as the ground (false detection), it is possible to avoid incorrect judgment and make correct class classification decisions.

[0043] The clutter determination unit 126 determines whether each detection point of the ranging sensor 2 is clutter using the analysis results of the image analysis unit 125. For example, the clutter determination unit 126 determines whether the detection point detected by the ranging sensor 2 is clutter using the class classification results by the image analysis unit 125. In detail, the clutter determination unit 126 determines whether each detection point belongs to the background class based on the class classification results received from the image analysis unit 125, determines that the detection point corresponding to the extracted image determined to be the background (background class) by the image analysis unit 125 is clutter, and notifies the target output unit 127 of the detection point determined to be clutter. Note that the clutter determination unit 126 may determine whether the detection point is clutter using the detection results detected by the ranging sensor 2 (such as the analysis results of the received signal processed by the ranging sensor 2 and the information determination results such as speed) in addition to the class classification results received from the image analysis unit 125. The detection result used for this determination may be, for example, at least one of distance, speed, and received signal level (assumed RCS), or a combination thereof. The image analysis unit 125 may perform analysis using the captured image and the detection result detected by the ranging sensor 2, and output one analysis result, i.e., one class classification result, to the clutter determination unit 126. For example, the image analysis unit 125 may perform class classification using a trained model that receives the captured image and the detection result detected by the ranging sensor 2 as input. The clutter determination unit 126 also assigns the determination result of the clutter determination unit 126 and classification class information (background: clutter or foreground information) to the detection result received from the clutter region determination unit 121 (detection result of the ranging sensor 2), and outputs the result to the target output unit 127.

[0044] The clutter determination unit 126 may also determine whether a detected point is clutter using multiple class classification results corresponding to multiple captured images captured at different times, i.e., multiple shots or multiple frames of captured images. For example, if a target is moving, it is possible that the background and targets other than the background may not be steadily observed in the observed captured images. Multiple class classification results are obtained by performing the above-described processing on multiple frames of captured images using the image extraction unit 124 and image analysis unit 125. The clutter determination unit 126 may tentatively determine whether a detected point is clutter based on each of the multiple class classification results, and may determine the detected point as clutter if the multiple tentative determination results corresponding to the multiple class classification results are equal to or greater than a certain number, i.e., if the detected point is tentatively determined to be clutter more than a certain number of times. As described above, by making a determination using the results of multiple frames, clutter determination can also be performed for non-stationary targets. The image analysis unit 125 may also take multiple captured images captured by the imaging sensor 3 over multiple shots or multiple frames as input images and output a single class classification result. In this case, the image analysis unit 125 performs image analysis based on multiple input images, classifies the images into at least two or more classes, background and non-background, and outputs one classification result as a comprehensive result of the multiple captured images. Furthermore, the image analysis unit 125 may take in the captured images acquired by the image sensor 3 as input images and the detection results acquired by the distance measurement sensor 2 as input information, and output one object detection result analyzed based on each piece of information.

[0045] The target output unit 127 removes detection points that the clutter determination unit 126 has determined to be clutter from all detection results acquired by the ranging sensor 2 and outputs the results. That is, the target output unit 127 removes information corresponding to detection points that the clutter determination unit 126 has determined to be of the background class from the detection results of the ranging sensor 2 received from the clutter determination unit 126 (all detection results acquired by the ranging sensor 2). The target output unit 127 outputs the detection results from which information corresponding to detection points determined to be of the background class has been removed to the sensor processing unit 13 as first object information. That is, the target output unit 127 removes excluded detection points, which are detection points that the clutter determination unit 126 has determined to be clutter, from the detection results detected by the ranging sensor 2, and outputs the detection results from which the excluded detection points have been removed to the sensor processing unit 13.

[0046] When the sensor processing unit 13 performs class classification other than background, the captured image is input to the sensor processing unit 13, and the sensor processing unit 13 generates second object information by detecting an object from the captured image. As described above, the sensor processing unit 13 integrates the first object information and the second object information, but since the first object information is information from which information corresponding to clutter has been removed, the information corresponding to clutter is not included in the integrated information.

[0047] Next, the operation of the clutter determination processing unit 12 of this embodiment will be described. Fig. 7 is a flowchart showing an example of a clutter discrimination processing procedure in the clutter determination processing unit 12 of this embodiment. Note that Fig. 7 shows processing related to clutter discrimination, and does not show processing related to the second object information, but the second object information is generated by, for example, the sensor processing unit 13, as described above.

[0048] 7 , the clutter determination processing unit 12 acquires a captured image (step S10) and acquires the detection result of the distance measurement sensor 2 (step S11). Note that the order of steps S10 and S11 is not limited to this order, and they may be performed in reverse or simultaneously. In detail, the image extraction unit 124 receives the captured image from the first transmission / reception unit 11, and the clutter region determination unit 121 receives the detection result of the distance measurement sensor 2 from the first transmission / reception unit 11.

[0049] Next, the clutter determination processing unit 12 determines whether a detection point of the ranging sensor 2 is present within the set discrimination region (step S12). More specifically, the clutter region determination unit 121 uses the detection result of the ranging sensor 2 to determine whether a certain detection point detected by the ranging sensor 2 is present within the discrimination region. Note that, since the detection point to be processed is changed sequentially as described below, the detection point to be processed in step S12 may be determined in any manner.

[0050] If there is no detection point of the distance measuring sensor 2 within the discrimination region (No in step S12), the clutter determination processing unit 12 ends the clutter determination process. More specifically, if the clutter region determination unit 121 determines that there is no detection point of the distance measuring sensor 2 within the discrimination region, the clutter determination processing unit 12 ends the clutter determination process. In this case, the clutter determination processing unit 12 outputs the detection result as is to the sensor processing unit 13 as first object information, and ends the clutter determination process.

[0051] If a detection point of the ranging sensor 2 is present within the discrimination region (step S12: Yes), the clutter determination processing unit 12 determines a provisional size of the ranging target (step S13). Specifically, if a detection point of the ranging sensor 2 is present within the discrimination region, the clutter region determination unit 121 outputs position information of the detection point of the ranging sensor 2 corresponding to the detection point to the provisional size determination unit 122, and outputs the detection result of the ranging sensor 2 to the provisional size determination unit 122 and the clutter determination unit 126. The provisional size determination unit 122 determines a provisional size of the ranging target, which is the object detected by the ranging sensor 2, using the detection result received from the clutter region determination unit 121, and notifies the region projection unit 123 of the determined provisional size, and outputs the position information of the detection point of the ranging sensor 2 received from the clutter region determination unit 121 to the region projection unit 123.

[0052] Next, the clutter determination processing unit 12 performs coordinate conversion on the detection point positions (detection point positions) and provisional size of the ranging sensor 2 to identify the position and area (size) of the ranging target on the captured image as a provisional area (step S14). In detail, the area projection unit 123 uses the provisional size notified by the provisional size determination unit 122 and the detection result of the ranging sensor 2 received from the provisional size determination unit 122 to perform coordinate conversion on the position and area of ​​the detection points of the ranging sensor 2 into the coordinate system of the captured image, thereby identifying the position and area of ​​the ranging target on the captured image as a provisional area. The area projection unit 123 notifies the image extraction unit 124 of the provisional area of ​​the ranging target displayed on the captured image.

[0053] Next, the clutter determination processing unit 12 extracts an image corresponding to the provisional area of ​​the target from the captured image (step S15). Specifically, the image extraction unit 124 extracts an image of the provisional area notified by the area projection unit 123 from the captured image, and outputs the extracted image to the image analysis unit 125. The above steps S12 to S15 are preprocessing for class classification in this embodiment.

[0054] Next, the clutter determination processing unit 12 classifies the extracted images into background and non-background (step S16). Specifically, the image analysis unit 125 classifies the images received from the image extraction unit 124 into background and non-background, and outputs the classification results to the clutter determination unit 126.

[0055] Next, the clutter determination processing unit 12 determines whether or not the class classification result of the detection point currently being processed is the background class (step S17). In detail, the clutter determination unit 126 determines whether or not the class classification result of the detection point currently being processed is the background class based on the classification result received from the image analysis unit 125, and notifies the target output unit 127 of the determination result and outputs the detection result of the distance measuring sensor 2 to the target output unit 127.

[0056] If the detected point is in the background class (step S17: Yes), the clutter determination processing unit 12 excludes the detected point from the detection target (step S18). Specifically, the clutter determination unit 126 notifies the target output unit 127 of the detected point determined to be in the background class. The target output unit 127 excludes the detected point notified by the clutter determination unit 126 from the detection results of the distance measuring sensor 2 received from the clutter determination unit 126. In other words, the target output unit 127 excludes the detected point notified by the clutter determination unit 126 from the target detection target. If the detected point is not in the background class (step S17: No), the clutter determination processing unit 12 proceeds to step S19.

[0057] After step S18, or if step S17 is No, the clutter determination processing unit 12 determines whether or not the determination processing has been performed for all detection points of the distance measurement sensor 2 (step S19). More specifically, the clutter region determination unit 121 determines whether or not the processing of steps S12 to S18 has been performed for all detection points detected by the distance measurement sensor 2. This determination may be made by the clutter determination unit 126 or an overall control unit (not shown) that controls the processing of the clutter determination processing unit 12.

[0058] If the determination process has been performed on all detection points of the ranging sensor 2 (Yes in step S19), the clutter determination processor 12 terminates the clutter discrimination process. In this case, the clutter determination processor 12 removes information corresponding to detection points determined by the clutter determination unit 126 to belong to the background class from the detection results of the ranging sensor 2, and outputs the result to the sensor processing unit 13 as first object information. If the determination process has not been performed on all detection points of the ranging sensor 2 (No in step S19), that is, if there are detection points for which determination process has not been performed, the clutter determination processor 12 performs the process again from step S12 on the detection points for which determination process has not been performed. In this way, if multiple detection points are detected, the clutter region determination unit 121, the provisional size determination unit 122, the region projection unit 123, the image extraction unit 124, the image analysis unit 125, and the clutter determination unit 126 may perform the processes for each detection point. In other words, if multiple detection points are detected, the clutter image processing unit 128 and the clutter determination unit 126 may perform the processes for each detection point. In addition, processing of multiple detection points may be performed by repeating steps S12 to S19 in order for each detection point, or processing of multiple detection points may be performed collectively at the end of each processing step as appropriate, and multiple results may be output.

[0059] In the above example, the clutter region determination unit 121 determines whether or not there is a detection point within the discrimination region, but the discrimination region may be an area corresponding to the entire detection range of the distance measuring sensor 2. In this case, the clutter region determination unit 121 does not need to be provided, and the processing from step S13 onwards may be performed for each detection point.

[0060] Furthermore, the determination of whether or not an object is clutter may be performed for a detection point where first detection information, which is the result of object detection using the ranging sensor 2, does not match second detection information detected from the captured image acquired by the imaging sensor 3. The second detection information corresponds to the second object information described above, and the first detection information corresponds to the detection result of the ranging sensor 2 described above. For example, the sensor processing unit 13 may previously use the first detection information and the second detection information to determine whether or not the target detected by the ranging sensor 2 matches the object detected using the captured image acquired by the imaging sensor 3, and then the clutter determination processing unit 12 may perform the processes of steps S12 to S19 for a detection point where the target detected by the ranging sensor 2 does not match the object detected using the captured image acquired by the imaging sensor 3. The determination of whether or not the target detected by the ranging sensor 2 matches the object detected using the captured image may be performed by an overall control unit (not shown) that controls the processing of the clutter determination processing unit 12.

[0061] Through the above processing, the clutter determination processing unit 12 of this embodiment uses the captured image, which is a two-dimensional image, to determine clutter in the detection results of the distance measuring sensor 2. As a result, the clutter determination processing unit 12 of this embodiment can determine clutter with a simple configuration and low-load processing.

[0062] Furthermore, in general object detection, the targets that can be recognized are targets (people, animals, vehicles, fruits, miscellaneous goods, etc.) that have a regular shape, size, color, etc., and whose features can be extracted. Therefore, in general object detection, it is difficult to distinguish backgrounds, which often have indefinite shapes, uniform colors and patterns, and whose distributions may change. Furthermore, in a method that distinguishes between targets and backgrounds by edge detection using brightness gradients and variances, if the background has a pattern, the brightness variance of the background increases, and the background may not be determined to be background. In this embodiment, deep learning is performed on the area corresponding to the target detected by the ranging sensor 2 using various background states as learning data, thereby performing class classification that can extract multidimensional features such as pixel profiles (brightness), edges, size, and gradients. Therefore, even if an object has an indefinite shape and a uniform color and pattern, it is possible to distinguish it as background or foreground. Conversely, in conditions where the background image contains shadows, white lines, grass, etc. on the ground, i.e., there are variations in the colors and patterns contained in the background image, and discrimination using brightness gradient and variance would result in a misclassification (mistakenly determining that it is not ground), it is still possible to distinguish it from the background (ground) using deep learning for class classification.

[0063] FIG. 8 is a diagram illustrating the effects of using deep learning. Images 511 to 516 in FIG. 8 all depict ground, but they also depict backgrounds that are prone to misclassification when using edge detection based on brightness gradient and variance. As shown in FIG. 8 , when images 511 to 516 contain a white line 521, a manhole 522, a shadow 523, a part of a building 524, and a support pillar 525, or when a grassy area is present in the background as in image 515, the brightness variance of the background increases, making it more likely that brightness gradient and variance will misclassify the object as an obstacle rather than the ground. In contrast, deep learning and classification, which can extract multidimensional features such as pixel profiles (brightness), edges, size, and gradients, can identify even non-uniform images like the above as background (ground).

[0064] Next, the classification of this embodiment will be described in detail. FIG. 9 is a diagram for explaining the classification of this embodiment. The image shown in FIG. 9 is an example of an image captured by the imaging sensor 3, and extraction frames 201-1 and 201-n (n is an integer equal to or greater than 2) indicate provisional regions projected onto the captured image of the target identified in step S14 described above. In the example shown in FIG. 9, n targets are included in the discrimination region. Note that although n is 3 or greater, only extraction frames 201-1 and 201-n are shown in FIG. 9, extracted images 202-1 to 202-n corresponding to extraction frames 201-1 to 201-n, respectively, are extracted in step S15. Note that, as described above, these extraction frames 201-1 and 201-n are provisional regions projected onto a two-dimensional plane as positions and regions on the captured image corresponding to targets detected by the ranging sensor 2. Strictly speaking, the region in question may include distortions of the imaging optical system used, but as long as it covers the region, this is not limited to the above. The provisional region may be a square of a standard size, as shown in FIG. 9 , for example, regardless of the shape of the region converted into the coordinate system of the captured image. For example, for convenience in handling pixels, the extraction frames 201-1, 201-n may be simplified in terms of pixels, and the smallest square or circle that includes the provisional region. For example, the region projection unit 123 may define, as the provisional region, a simplified region (e.g., square, circle, etc.) in terms of image pixels for a non-rectangular region obtained by projecting the region of the target corresponding to the detection point of the ranging sensor 2 onto the captured image through coordinate transformation.

[0065] As shown in FIG. 9 , extracted images 202-1 to 202-n are input one by one to the image analysis unit 125. Therefore, if n images are extracted, the images are input to the image analysis unit 125 n times. The image analysis unit 125 may also process multiple images at once. For each image, the image analysis unit 125 outputs one class classification result indicating whether the image belongs to the background class, i.e., (0, 1): background (clutter) or foreground (obstacle). Furthermore, for example, if the image analysis unit 125 uses a CNN or the like, it may output an intermediate output of the model rather than a binary value of 0 or 1. For example, the output of the image analysis unit 125 may be a probability value for each class or other intermediate value (classification intermediate value). When the probability value is output, the clutter determination unit 126 determines that the background class probability value exceeds a certain threshold value to be clutter. Conversely, the image analysis unit 125 may output a probability value or other intermediate value (classification intermediate value) corresponding to the foreground (obstacle) class, and the clutter determination unit 126 may determine whether the target object is in the foreground (obstacle) class. These may be set appropriately depending on the recognition method such as class classification or object detection, and the operation of the learning model. The image analysis unit 125 may output only a probability value, or may output multiple values ​​such as binary values ​​of 0 or 1 or other intermediate values.

[0066] The image extraction unit 124 may extract an image of a region corresponding to the discrimination region (a region where clutter is expected to be present in the detection results of the ranging sensor 2), and output to the image analysis unit 125 a divided image in which the detection point of the ranging sensor 2 exists among a plurality of divided images obtained by dividing the extracted image by a predetermined number. FIG. 10 is a diagram showing an example of region division according to this embodiment. In the example shown in FIG. 10 , the image extraction unit 124 extracts from the captured image a region corresponding to the discrimination region described above, i.e., an image within an image frame 301 indicating the region obtained by converting the discrimination region into the coordinate system of the captured image, and divides the extracted extracted image 302. In the example shown in FIG. 10 , the extracted image 302 is divided into nine equal parts, but the division method is not limited to equal division, and the number of divisions (the number of divided images obtained by division) is also not limited to nine.

[0067] For example, in processing the first distance measurement target, the image extraction unit 124 inputs to the image analysis unit 125 a divided image corresponding to the divided area including the detection point of the distance measurement sensor 2 notified by the area projection unit 123. Similarly, in processing the second distance measurement target, the image extraction unit 124 inputs to the image analysis unit 125 a divided image corresponding to the divided area including the detection point of the distance measurement sensor 2 notified by the area projection unit 123. That is, among the image areas obtained by dividing the discrimination area, if the divided images in which the detection point of the distance measurement sensor 2 is included are divided image 302-3 (obstacle) and divided image 302-5 (ground clutter), these are input to the image analysis unit 125. If there are n distance measurement targets detected by the distance measurement sensor 2, divided images corresponding to the n distance measurement targets are input to the image analysis unit 125 one by one. Note that here, division is performed with respect to the area within the image frame 301, but the captured image itself may also be divided into multiple divided areas. Alternatively, all divided regions may be classified, and then the detected points may be associated with the classified regions and the results of the discrimination may be output.

[0068] Alternatively, instead of extracting a provisional area from the acquired image by the image extraction unit 124, the angle of view of the imaging sensor 3 may be adjusted so that only the calculated provisional area is captured based on the detection results of the ranging sensor 2, and then an image may be captured. That is, the imaging sensor 3 may capture an image by adjusting the angle of view of the imaging sensor 3 so that the detection points of the ranging sensor 2 correspond to the provisional area whose coordinates have been transformed by the area projection unit 123. This method makes it possible to eliminate image extraction processes such as cropping and resizing, and by adjusting the provisional area so that a provisional area that previously captured only a portion of the image is captured so that the entire image is captured, the resolution of the captured image is increased, which is expected to improve the accuracy of clutter discrimination.

[0069] The clutter determination unit 126 determines for each divided region whether the divided region is clutter, as in the example described above. If the provisional region described above exists within a divided region determined to be clutter, the clutter determination unit 126 determines that the provisional region is clutter. Note that although the image is divided into nine regions in FIG. 10 , the number of divisions is not limited to nine. In this way, by dividing the image input to the image analysis unit 125 into divided images, the processing load per cycle on the image analysis unit 125 can be reduced.

[0070] The image analysis unit 125 performs class classification using machine learning such as CNN, random forest, or support vector machine, but is not limited to these, and the image analysis unit 125 may use any method for class classification.

[0071] 11 is a diagram showing a functional configuration of the image analysis unit 125 according to this embodiment and an example of creating a learned model. Fig. 11 shows an example in which the image analysis unit 125 creates a learned model by machine learning such as CNN, random forest, or support vector machine, and performs class classification using the learned model.

[0072] In the example shown in FIG. 11 , the image analysis unit 125 includes a trained model storage unit 252 and an image classification unit 253. The model creation unit 251 creates a trained model, for example, through supervised learning, and stores it in the trained model storage unit 252. In detail, the model creation unit 251 creates a trained model through machine learning using a training dataset 250 consisting of training images and ground truth data of class classification results corresponding to the images. Note that the training images may be images cut out from captured images captured by the imaging sensor 3, images cut out from images captured by an imaging device other than the imaging sensor 3, or a mixture of these. Furthermore, images may be edited by inverting the orientation of the target object or by performing color conversion on the captured images, thereby increasing and incorporating more content and variation for learning.

[0073] The trained model storage unit 252 stores the trained model. The image classification unit 253 inputs the image received from the image extraction unit 124 into the trained model, thereby obtaining a class classification result as an output from the trained model. This trained model is a trained model for inferring a class corresponding to an input image from the image. Note that the model creation unit 251 may be provided outside the image analysis unit 125 in the information processing device 1, may be provided within the image analysis unit 125, or may be provided in a device separate from the information processing device 1.

[0074] 12 and 13 are diagrams showing examples of training images according to the present embodiment. In the example shown in FIG. 12 , an image is extracted from an image capturing an area including a road so as to include both an object other than the background and the background, and an object image 304 and a background image 305 are used as training images. Note that the training images shown in FIG. 12 are the same as the captured image shown on the left side of FIG. 6 . In the training dataset, the correct answer data, i.e., label, corresponding to the object image 304 is assigned the foreground class (1: class other than the background), and the correct answer data, i.e., label, corresponding to the background image 305 is assigned the background class (0). In this way, the training images include background images, non-background images, and images in which the background and non-background are mixed. In this way, the accuracy of class classification can be improved by including images of various types of objects in the training images.

[0075] Furthermore, as shown in FIG. 13 , for one object image 304, by cutting out the image using multiple image frames in which the positions of the cut-out regions are randomly varied as illustrated by image frames 306-1 to 306-3, images in which the object exists in various positions within the image are generated and used as training images. Images containing the object are assigned a foreground class as correct answer data. In this way, by using images in which the object exists in various positions within the image as training images, it is possible to respond to a variety of situations and improve the accuracy of class classification. The above images are merely examples, and training images are not limited to the above examples.

[0076] Next, the hardware configuration of the object detection system of this embodiment will be described. FIG. 14 is a diagram showing an example of the hardware configuration of the object detection system of this embodiment. As shown in FIG. 14 , the distance measurement sensor 2 includes the above-mentioned sensor unit 21, which is composed of a transmitting antenna, a receiving antenna, an electrical circuit, etc., and a distance measurement sensor ECU (Electronic Control Unit) 23, which functions as a control unit 22. The distance measurement sensor ECU 23 includes a processing circuit, for example, a processor and a memory. The imaging sensor 3 includes the above-mentioned imaging unit 31, which is composed of a lens, an imaging element, etc., and an imaging sensor ECU 33, which functions as a control unit 32. The imaging sensor ECU 33 includes a processing circuit, for example, a processor and a memory. The vehicle control unit 4 is realized by a vehicle control ECU 41. The vehicle control ECU 41 includes a processing circuit, for example, a processor and a memory.

[0077] The information processing device 1 is realized by, for example, a computer system including a first communication I / F 101, a vehicle communication I / F 102, a second communication I / F 103, a processor 104, and a memory 105. The first communication I / F 101, the vehicle communication I / F 102, the second communication I / F 103, the processor 104, and the memory 105 are connected via a system bus.

[0078] The first communication I / F 101 is a transmitter and receiver that comply with a communication standard for communicating with the distance measurement sensor 2 and the imaging sensor 3. The vehicle communication I / F 102 is a transmitter and receiver that comply with a communication standard for communicating with the vehicle control ECU 41, such as, but not limited to, CAN FD (registered trademark) (Controller Area Network with Flexible Data Rate). The second communication I / F 103 is a transmitter and receiver that comply with a communication standard for communicating with the display control unit 5.

[0079] The processor 104 may be a microcontroller unit (MCU), a central processing unit (CPU), a graphics processing unit (GPU), or the like. The memory 105 may include various types of memory, such as non-volatile or volatile semiconductor memory, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically programmable read-only memory (EEPROM), as well as storage devices such as a hard disk. The memory 105 stores programs to be executed by the processor 104, necessary data obtained during processing, and the like. The memory 105 is also used as a temporary storage area for programs. The memory 105 and the processor 104 constitute a processing circuit. The processing circuit may be a single circuit or multiple circuits.

[0080] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. A computer system having the above configuration is provided with, for example, a reading unit (not shown) that reads a recording medium on which the program is recorded, and the program read from the recording medium is installed in memory 105. Then, when the program is executed, the program read from memory 105 is stored in the main storage area of ​​memory 105. In this state, processor 104 executes processing as information processing device 1 of this embodiment in accordance with the program stored in memory 105.

[0081] In the above description, a program describing the processing in the information processing device 1 is provided as a recording medium, but this is not limited to this. For example, a program provided via a transmission medium such as the Internet via the first communication I / F 101, the second communication I / F 103, or another transceiver not shown in the figure may also be used.

[0082] The program of this embodiment causes a computer system to perform, for example, functional processing of setting a provisional size of the target at the detection point acquired by the ranging sensor 2 in a clutter discrimination area, and analyzing an image including a provisional area corresponding to the provisional size of the target at each detection point of the ranging sensor 2 projected onto the image captured by the imaging sensor 3; functional processing of using the analysis results to determine whether each detection point of the ranging sensor 2 is clutter or not; and functional processing of removing detection points determined to be clutter from all detection results acquired by the ranging sensor 2 and outputting the results.

[0083] The clutter determination processing unit 12 and the sensor processing unit 13 shown in FIG. 1 are realized by the processor 104 shown in FIG. 14 executing a program stored in the memory 105 shown in FIG. 14. The memory 105 is also used to realize the clutter determination processing unit 12 and the sensor processing unit 13. The first transceiver unit 11 shown in FIG. 1 is realized by the first communication I / F 101 shown in FIG. 14. The second transceiver unit 14 shown in FIG. 1 is realized by the second communication I / F 103 shown in FIG. 14. The vehicle communication unit 15 shown in FIG. 1 is realized by the vehicle communication I / F 102 shown in FIG. 14. The storage unit 16 shown in FIG. 1 is part of the memory 105 shown in FIG. 14. The information processing device 1 may be realized by multiple computer systems.

[0084] The display control unit 5 is a computer system including a memory 51, a processor 52, and a communication I / F 53, all of which are connected via a system bus. The processor 52 is an MCU, CPU, GPU, etc., similar to the processor 104, and the memory 51 is a storage device such as a hard disk, etc., similar to the memory 105. The communication I / F 53 is a transmitter and receiver that comply with a communication standard for communicating with the information processing device 1.

[0085] As described above, the clutter determination processing unit 12 of this embodiment uses the captured image to perform class classification to classify the captured image into background and non-background, thereby removing from the detection target those detection points that correspond to the background among the detection points included in the detection result received from the distance measuring sensor 2. Therefore, the clutter determination processing unit 12 of this embodiment can determine clutter with a simple configuration and low-load processing.

[0086] Second Embodiment Fig. 15 is a diagram showing an example of the functional configuration of a clutter determination processing unit 12a according to a second embodiment. The object detection system of this embodiment is similar to the object detection system of the first embodiment, except that it includes a clutter determination processing unit 12a instead of the clutter determination processing unit 12. As shown in Fig. 1, for example, a first transceiver 11 is provided between the distance measuring sensor 2 and the image sensor 3 and the clutter determination processing unit 12a, but the first transceiver 11 is not shown in Fig. 15. Components having the same functions as those of the first embodiment are assigned the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0087] As in the first embodiment, the clutter determination processing unit 12a extracts detection results of detection points corresponding to detection targets, i.e., targets other than clutter, from the detection results of the distance measuring sensor 2 and outputs them as first object information to the sensor processing unit 13. The captured image acquired by the imaging sensor 3 is input from the first transmission / reception unit 11 to the sensor processing unit 13, which generates second object information. As described in the first embodiment, the second object information may be generated by a device other than the sensor processing unit 13. The clutter determination processing unit 12a includes a clutter region determination unit 121, a provisional size determination unit 122, a region projection unit 123, an image extraction unit 124a, an image analysis unit 125a, a clutter determination unit 126a, and a target output unit 127. The clutter region determination unit 121, the provisional size determination unit 122, the region projection unit 123, the image extraction unit 124a, and the image analysis unit 125a constitute a clutter image processing unit 128a.

[0088] In this embodiment, the clutter determination processing unit 12a extracts an image of a discrimination region from the captured image, which is a region in the captured image where the ranging sensor 2 is expected to detect clutter, and determines, by segmentation, which of multiple classes the pixels in the extracted image belong to, including at least background and non-background. Examples of segmentation include, but are not limited to, semantic segmentation, instance segmentation, and panoptic segmentation. The clutter determination processing unit 12a then uses the class determination result to determine whether the detection point of the ranging sensor 2 is clutter. As in the first embodiment, the discrimination region is a region in the coordinate system of the captured image where the ranging sensor 2 is expected to detect clutter, i.e., unwanted waves due to the background.

[0089] The operations of the clutter region determination unit 121, the provisional size determination unit 122, and the region projection unit 123 are the same as in the first embodiment, and the processing target is the detection point of the ranging sensor 2 within the set clutter discrimination region, as in the first embodiment. The provisional size determination unit 122 determines the provisional region by determining the provisional size of the ranging target using the detection result of the ranging sensor 2, as in the first embodiment. The region projection unit 123, as in the first embodiment, uses the provisional size and the detection result (position and type prediction result) to perform coordinate conversion of the position of the detection point of the ranging sensor 2, thereby identifying the position and region on the captured image corresponding to the provisional size of the ranging target, and notifying the clutter determination unit 126a of the provisional region. The detection result of the ranging sensor 2 is also input to the clutter determination unit 126a, along with the provisional size determination unit 122 and the region projection unit 123.

[0090] The clutter discrimination region may be set for each operating condition, similar to the discrimination region in Embodiment 1. For example, the image extraction unit 124a or the clutter region determination unit 121 may determine the discrimination region according to the operating condition.

[0091] The image extraction unit 124a extracts an image corresponding to the clutter discrimination region set by the clutter region determination unit 121 from the captured image received from the imaging sensor 3, and outputs the extracted image to the image analysis unit 125a. Note that the image extraction unit 124a may extract an image of the discrimination region by cropping a portion of the captured image corresponding to the discrimination region and resizing the cropped image. Furthermore, the image extraction unit 124a may simplify the image on a pixel basis for convenience in handling pixels, and extract, for example, an image of the smallest rectangle that includes the discrimination region.

[0092] The image analysis unit 125a performs segmentation to classify the pixels of the image received from the image extraction unit 124a into two or more classes, including a background and a non-background class. For example, the image analysis unit 125a performs segmentation to classify the image received from the image extraction unit 124a pixel by pixel, and outputs a pixel classification result indicating the class of each pixel to the clutter determination unit 126a. Note that the segmentation may be performed on a pixel-by-pixel basis, or on a multiple-pixel basis. Alternatively, the image analysis unit 125a may perform segmentation on pixels thinned out at regular intervals. That is, the segmentation is performed on a predetermined pixel basis. The predetermined pixels are specified, for example, by the number of vertical and horizontal pixels.

[0093] The clutter determination unit 126a maps a provisional region corresponding to the detection points from the region projection unit 123 onto the image of the extracted clutter discrimination region, calculates and analyzes the frequency distribution of each class of the mapped image region from the class classification results for each pixel received from the image analysis unit 125a, and determines whether the region is classified as a background class. The clutter determination unit 126a attaches the determination result and classification class information of the clutter determination unit 126a to the detection result of the ranging sensor 2 and notifies the target output unit 127. Specifically, the clutter determination unit 126a determines whether the provisional region is background based on the class classification results and frequency distribution of each pixel in the provisional region, for example, whether the background class frequency is higher than that of the foreground, or whether the frequency is above a certain level. This allows the clutter determination unit 126a to determine whether the detection points of the ranging sensor 2 corresponding to the provisional region are clutter. The clutter determination unit 126a may determine whether or not a detection point is clutter using the pixel classification result received from the image analysis unit 125a and the provisional area notified from the area projection unit 123, as well as the detection result detected by the distance measurement sensor 2. The detection result used for this determination may be, for example, at least one of distance, speed, and received signal level (assumed RCS), or a combination thereof.

[0094] The clutter determination unit 126a may also determine whether a detection point is clutter using multiple pixel classification results corresponding to multiple captured images captured at different times, i.e., multiple frames of captured images. For example, multiple pixel classification results are obtained by performing the above-described image extraction unit 124a and image analysis unit 125a on each of the captured images in multiple frames. The clutter determination unit 126a may provisionally determine whether a detection point is clutter based on each of the multiple pixel classification results, and determine the detection point to be clutter if a certain number of provisional determination results corresponding to the multiple pixel classification results are met, i.e., if the detection point is provisionally determined to be clutter more than a certain number of times. Note that the image extraction unit 124a and the image analysis unit 125a may also take multiple captured images acquired over multiple shots or multiple frames by the imaging sensor 3 as input images and output a single pixel classification result. In this case, the image analysis unit 125a performs image analysis based on the multiple input images and outputs a single pixel classification result as a comprehensive result of the multiple captured images. Furthermore, the image analysis unit 125a may take in the captured image acquired by the imaging sensor 3 as an input image and the detection result acquired by the ranging sensor 2 as input information, and output the result of one object detection analyzed based on each piece of information.

[0095] The target output unit 127 removes the detection points determined to be clutter by the clutter determination unit 126a from the detection results of the distance measurement sensor 2, and outputs the detection results from which the detection points determined to be clutter have been removed as first object information to the sensor processing unit 13. Note that the second object information is generated by the sensor processing unit 13 as in the first embodiment, but is not limited to this, and may be generated by, for example, the image analysis unit 125a, an object recognition unit (not shown), or the control unit 32 of the imaging sensor 3.

[0096] Next, the operation of the clutter determination processing unit 12a of this embodiment will be described. Fig. 16 is a flowchart showing an example of a clutter discrimination processing procedure in the clutter determination processing unit 12a of this embodiment. Note that Fig. 16 shows processing related to clutter discrimination, and does not show processing related to second object information.

[0097] Steps S10, S11, and S12 are the same as those in embodiment 1. If step S12 is No, the clutter determination processing unit 12a ends the clutter determination process, as in embodiment 1.

[0098] If the answer is Yes in step S12, the clutter determination processing unit 12a extracts the captured image of the discrimination region (step S32). Specifically, the image extraction unit 124a extracts the image of the discrimination region from the captured image and outputs the extracted image to the image analysis unit 125a.

[0099] Next, the clutter determination processing unit 12a segments the extracted image into background and non-background parts in predetermined pixel units (step S33). Specifically, the image analysis unit 125a classifies the image received from the image extraction unit 124a in pixel units determined by the segmentation, and outputs the class of each pixel to the clutter determination unit 126a.

[0100] Steps S13 and S14 are the same as in the first embodiment, and as described above, the processing target is the detection point of the distance measuring sensor 2 within the discrimination area. As in the first embodiment, the clutter determination processing unit 12a may determine a provisional size for the detection point within the discrimination area and convert the coordinates of the provisional area into the coordinate system of the captured image. In Fig. 16, steps S13 and S14 are described after step S33, but the timing at which steps S13 and S14 are performed is not limited to this, and it is sufficient that the processing corresponding to the target object to be processed is performed between step S11 and step S34, which will be described later.

[0101] The clutter determination processing unit 12a determines whether the provisional region is located within the background class region (step S34). Specifically, the clutter determination unit 126a determines whether the provisional region is located within the background class region using the pixel classification result received from the image analysis unit 125a and the provisional region notified from the region projection unit 123, and then adds the determination result and classification class information of the clutter determination unit 126a to the detection result of the ranging sensor 2 and notifies the target output unit 127. For example, as will be described later, the clutter determination processing unit 12a determines whether the provisional region is located within the background class region based on a pixel frequency distribution indicating the number of pixels determined to be the background class and the number of pixels determined to be the foreground class.

[0102] If the provisional region is present within the background class region (step S34: Yes), the clutter determination processing unit 12a excludes the detection point from the detection targets (step S18), similar to step S18 in embodiment 1. Step S19 is the same as in embodiment 1. If the provisional region is not present within the background class region (step S34: No), the clutter determination processing unit 12a proceeds to step S19. As in embodiment 1, if multiple detection points are detected, steps S13 to S19 may be repeated in order for each detection point, or multiple detection points may be processed together at the end of each processing step, as appropriate, and multiple results may be output.

[0103] Furthermore, as in the first embodiment, the determination of whether or not an object is clutter may be performed for a detection point where the first detection information, which is the result of object detection using the ranging sensor 2, and the second detection information, which is the result of object detection using the captured image, do not match. For example, the sensor processing unit 13 may previously determine, using the first detection information and the second detection information, whether or not the target detected by the ranging sensor 2 matches the object detected using the captured image. Then, if there is a detection point where the target detected by the ranging sensor 2 does not match the object detected using the captured image, the clutter determination processing unit 12a may perform the processing of steps S32 to S19 shown in FIG. 16. The determination of whether or not the target detected by the ranging sensor 2 matches the object detected using the captured image may be performed by an overall control unit (not shown) that controls the processing of the clutter determination processing unit 12a.

[0104] Next, clutter discrimination using segmentation according to this embodiment will be described in detail. FIG. 17 is a diagram for explaining clutter discrimination using segmentation according to this embodiment. A captured image is shown in the upper left of FIG. 17, and an image within an image frame 401 indicating a discrimination region is extracted from the captured image. The extracted image 402 is input to the image analysis unit 125a, which performs segmentation. The image analysis unit 125a classifies the image into, for example, a background class and a foreground class in predetermined pixel units. The first region 403-1 and the second region 403-2 shown in FIG. 17 are each assumed to be the provisional regions described above.

[0105] Here, the class is determined on a pixel-by-pixel basis, and the clutter determination unit 126a aggregates the results of the pixel-by-pixel classification within each of the first region 403-1 and the second region 403-2. The lower right of FIG. 17 shows the number of pixels determined to be the background class and the number of pixels determined to be the foreground class within each of the first region 403-1 and the second region 403-2 as frequencies (pixel frequency distributions). Because the first region 403-1 contains a cardboard box (an actual obstacle), i.e., an object other than the background, there are pixels of the background class and pixels of the foreground class. Because the second region 403-2 is the ground, i.e., the background, all of the pixels within the second region 403-2 are determined to be the background class. Using these aggregation results, the clutter determination unit 126a determines that the provisional area corresponding to the first area 403-1 is not clutter, and that the provisional area corresponding to the second area 403-2 is clutter (the second area 403-2 exists within the area of ​​the background class).

[0106] Although FIG. 17 illustrates an example in which the class of a provisional region is determined based on whether the background or foreground class distribution is more prevalent, the method for determining whether a provisional region is clutter is not limited to this. For example, a threshold value for the ratio of the number of pixels in the foreground class to the number of pixels in the background class may be determined, and the clutter determination unit 126a may determine that a provisional region is not clutter if this ratio exceeds the threshold value. For example, the clutter determination unit 126a may determine that a provisional region is not clutter if the number of pixels in the foreground class exceeds X% of the number of pixels in the background class. This X% corresponds to the threshold value. If X is set to 0, the provisional region is determined to be clutter if all pixels are in the background class. Even when segmentation is performed in units of multiple pixels rather than in units of one pixel, the clutter determination unit 126a similarly determines whether a provisional region is clutter using the results of classifying each provisional region by a predetermined pixel unit, i.e., the frequency corresponding to each class. Also, although an example in which the image is divided into two classes, background and foreground, has been shown here, the background and non-background may be further divided into multiple classes, as in the first embodiment.

[0107] The image analysis unit 125a performs segmentation using machine learning such as CNN, random forest, or support vector machine, but is not limited to this and any segmentation method may be used in the image analysis unit 125a.

[0108] 18 is a diagram showing a functional configuration of the image analysis unit 125a according to the present embodiment and an example of creating a learned model. Fig. 18 shows an example in which the image analysis unit 125a creates a learned model by machine learning such as CNN, random forest, or support vector machine, and performs segmentation using the learned model.

[0109] In the example shown in FIG. 18 , the image analysis unit 125a includes a trained model storage unit 252a and a pixel classification unit 253a. The model creation unit 251a creates a trained model, for example, through supervised learning and stores it in the trained model storage unit 252a. Specifically, the model creation unit 251a creates a trained model through machine learning using a training dataset 250a consisting of training images and ground truth data of class classification results for a predetermined pixel unit corresponding to the training images. Note that the training images may be images cut out from captured images captured by the imaging sensor 3, images cut out from images captured by an imaging device other than the imaging sensor 3, or a mixture of these. Furthermore, images that have undergone color conversion or the orientation of the target object in the captured image may be edited to increase and incorporate more content and variation for training.

[0110] The trained model storage unit 252a stores the trained model. The pixel classification unit 253a inputs the image received from the image extraction unit 124a into the trained model, thereby obtaining the result of classification in a predetermined pixel unit as output from the trained model. This trained model is a trained model for inferring the result of classification in a predetermined pixel unit from the input image. Note that the model creation unit 251a may be provided outside the image analysis unit 125a in the information processing device 1, may be provided within the image analysis unit 125a, or may be provided in a device separate from the information processing device 1.

[0111] The hardware configuration of the object detection system of this embodiment is the same as that of embodiment 1. Like the clutter determination processing unit 12 of embodiment 1, the clutter determination processing unit 12a of this embodiment is realized by the processor 104 shown in Fig. 14 executing a program stored in the memory 105 shown in Fig. 14. The memory 105 is also used to realize the clutter determination processing unit 12a.

[0112] As described above, in this embodiment, the clutter determination processing unit 12a extracts an image of a discrimination region from a captured image, determines through segmentation which of a plurality of classes, including background and non-background, each pixel in the extracted image belongs to, and determines whether or not the detected point of the ranging sensor 2 is clutter using the determination result. Therefore, the clutter determination processing unit 12a of this embodiment can determine clutter with a simple configuration and low-load processing. Furthermore, by narrowing the target of segmentation to a discrimination region where the ranging sensor 2 is expected to detect clutter, the processing load can be reduced.

[0113] Third Embodiment Fig. 19 is a diagram showing an example of the functional configuration of a clutter determination processing unit 12b according to a third embodiment. The object detection system of this embodiment is similar to the object detection system of the first embodiment, except that it includes a clutter determination processing unit 12b instead of the clutter determination processing unit 12. As shown in Fig. 1, for example, a first transceiver 11 is provided between the distance measuring sensor 2 and the image sensor 3 and the clutter determination processing unit 12b, but the first transceiver 11 is not shown in Fig. 19. Components having the same functions as those of the first embodiment are assigned the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0114] The clutter determination processing unit 12b includes a clutter region determination unit 121, a provisional size determination unit 122, a region projection unit 123, an image analysis unit 125b, a clutter determination unit 126b, and a target output unit 127. The clutter region determination unit 121, the provisional size determination unit 122, the region projection unit 123, and the image analysis unit 125b constitute a clutter image processing unit 128b.

[0115] The clutter area determination unit 121, provisional size determination unit 122, and area projection unit 123 are the same as those in embodiment 1, but the area projection unit 123 notifies the clutter determination unit 126b of the position and size (area) of the target object on the captured image as a provisional area.

[0116] The image analysis unit 125b performs an object detection process using the captured image received from the first transceiver 11 and outputs the position, area, and type of the object obtained by the object detection process to the clutter determination unit 126b. The object detection process is similar to the object detection process for generating the second object information described in the first embodiment and is performed using machine learning such as, for example, CNN, random forest, or support vector machine, but is not limited thereto. For example, the image analysis unit 125b extracts an area in the captured image that is likely to be an object using machine learning, and recognizes and classifies the area to estimate the position, area rectangle (bounding box), and type of the object in the captured image. The object detected by the image analysis unit 125b is an object other than the background. For example, the image analysis unit 125b recognizes and classifies objects into multiple classes, including people and vehicles, but the classes recognized and classified by the image analysis unit 125b, i.e., the types of objects determined by the image analysis unit 125b, are not limited thereto. The image analysis unit 125b may output only the position of the detected object without determining the type of object.

[0117] In this embodiment, as in the first embodiment, the clutter determination processing unit 12b extracts the detection results of the detection targets, i.e., detection points corresponding to targets other than clutter, from the detection results of the distance measurement sensor 2, and outputs them as first object information to the sensor processing unit 13. The captured image acquired by the imaging sensor 3 is input from the first transmission / reception unit 11 to the sensor processing unit 13, and the sensor processing unit 13 generates second object information. As described in the first embodiment, the second object information may be generated by a unit other than the sensor processing unit 13. Furthermore, the result of the object detection processing by the image analysis unit 125b may be used as the second object information.

[0118] The clutter determination unit 126b determines whether the detected objects match using the object detection results (class, position, object detection area rectangle: hereinafter referred to as detection frame) output from the image analysis unit 125b and the detection results (position, provisional size) of the ranging sensor 2 notified from the area projection unit 123. For example, the clutter determination unit 126b compares and collates the object detection results (class, position, detection frame) output from the image analysis unit 125b with the detection results (position, provisional size) of the ranging sensor 2 notified from the area projection unit 123, and determines that an object whose detection frame overlaps with a certain position error or less is a detected target (other than the background). Furthermore, if no object detection result is obtained as a detection target and only the detection result of the ranging sensor 2 is obtained, the clutter determination unit 126b determines that the object is clutter (other than the detected target). The clutter determination unit 126b notifies the target output unit 127 of the detection points corresponding to the ranging targets determined to be clutter as detection points corresponding to clutter. The clutter determination unit 126b may determine whether or not a detection point is clutter by using the object detection results (class, position, detection frame) output from the image analysis unit 125b and the detection results (position, provisional size) of the ranging sensor 2 notified from the area projection unit 123, as well as the detection results detected by the ranging sensor 2. The detection results used for this determination may be, for example, at least one of distance, speed, and received signal level (assumed RCS), or a combination thereof.

[0119] The clutter determination unit 126b may also determine whether a detection point is clutter using object positions (object detection results) corresponding to multiple captured images captured at different times, i.e., multiple frame captured images. For example, multiple object detection results are obtained by the image analysis unit 125b performing the above-described processing on each of the multiple frame captured images. The clutter determination unit 126b may tentatively determine whether a detection point is clutter based on each of the multiple object detection results, and determine that the detection point is clutter if a certain number of multiple tentative determinations corresponding to the multiple object detection results are met, i.e., if the detection point is tentatively determined to be clutter more than a certain number of times. Note that the image analysis unit 125b may also take multiple captured images acquired over multiple shots or multiple frames by the imaging sensor 3 as input images and output a single object detection result. In this case, the image analysis unit 125b performs image analysis based on the multiple input images and outputs a single object detection result as a comprehensive result of the multiple captured images. Furthermore, the image analysis unit 125b may take in the captured image acquired by the image sensor 3 as an input image and the detection result acquired by the ranging sensor 2 as input information, and output the result of one object detection analyzed based on each piece of information.

[0120] Next, the operation of the clutter determination processing unit 12b of this embodiment will be described. Fig. 20 is a flowchart showing an example of a clutter discrimination processing procedure in the clutter determination processing unit 12b of this embodiment. Note that Fig. 20 shows processing related to clutter discrimination, and does not show processing related to second object information, but as described above, the processing result of the image analysis unit 125b may be used as the second object information.

[0121] Steps S10 to S14 are the same as those in the first embodiment. The clutter determination processing unit 12b performs object detection of targets (other than the background) in the captured image (step S41). Specifically, the image analysis unit 125b performs object detection processing using the captured image. Next, the clutter determination processing unit 12b acquires type, position, and area information of the captured target (target detected using the captured image) (step S42). Specifically, the image analysis unit 125b outputs the type, position, and area information of the object (target) obtained by the object detection processing to the clutter determination unit 126b.

[0122] The clutter determination processing unit 12b determines whether the detected object (imaged object) matches the position and area of ​​the ranging object (step S43). Specifically, the clutter determination unit 126b determines whether the detected object (imaged object) matches the position and area of ​​the ranging object using the object detection result (class, position, detection frame) output from the image analysis unit 125b and the detection result (position, provisional size) of the ranging sensor 2 notified from the area projection unit 123.

[0123] If the detected object (imaged object) matches the position and area of ​​the distance measurement object (Yes in step S43), the clutter determination processing unit 12b proceeds to step S19. If the detected object (imaged object) does not match the position and area of ​​the distance measurement object (No in step S43), the clutter determination processing unit 12b proceeds to step S18. Steps S18 and S19 are the same as in the first embodiment. As in the first embodiment, if multiple detection points are detected, steps S12 to S19 may be repeated in order for each detection point, or multiple detection points may be processed together at the end of each processing step as appropriate, and multiple results may be output.

[0124] The hardware configuration of the object detection system of this embodiment is the same as that of embodiment 1. Like the clutter determination processing unit 12 of embodiment 1, the clutter determination processing unit 12b of this embodiment is realized by the processor 104 shown in Fig. 14 executing a program stored in the memory 105 shown in Fig. 14. The memory 105 is also used to realize the clutter determination processing unit 12b.

[0125] As described above, in this embodiment, the clutter determination processing unit 12b uses the result of object detection processing using the captured image to determine that the detected object does not match the target of distance measurement by the distance measuring sensor 2 is clutter. Therefore, the clutter determination processing unit 12b in this embodiment can determine clutter with a simple configuration and low-load processing.

[0126] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0127] Various aspects of the present disclosure are summarized below as appendices.

[0128] (Supplementary Note 1) A clutter determination processing device comprising: an image analysis unit that sets a provisional size of a target at a detection point detected by a ranging sensor in an assumed clutter detection area, and analyzes an image including an area corresponding to the provisional target size of each detection point of the ranging sensor projected onto an image captured by an imaging sensor; a clutter determination unit that uses the analysis results of the image analysis unit to determine whether each detection point of the ranging sensor is clutter; and a target output unit that removes detection points determined to be clutter by the clutter determination unit from all detection results obtained by the ranging sensor and outputs the results. (Supplementary Note 2) The clutter determination processing device according to Supplementary Note 1, comprising: a clutter area determination unit that determines whether a detection point of the distance measuring sensor is within a predetermined clutter determination area; an area projection unit that determines, as a provisional area, an area obtained by projecting onto the captured image an area of ​​a target corresponding to the detection point within the clutter determination area; and an image extraction unit that extracts an image of the provisional area from the captured image, wherein the image analysis unit classifies the image of the provisional area extracted by the image extraction unit into two or more classes including at least a background and a non-background class, and the clutter determination unit determines, as clutter, the detection point corresponding to the provisional area determined to be the background by the image analysis unit. (Supplementary Note 3) The clutter determination processing device according to Supplementary Note 2, characterized in that the image extraction unit extracts an image of the clutter determination area from the captured image, and outputs, to the image analysis unit, one of a plurality of divided images obtained by dividing the image by a predetermined number, the divided image including the provisional area coordinate-transformed by the area projection unit. (Supplementary Note 4) The clutter determination processing device according to Supplementary Note 2, wherein the image sensor captures an image by adjusting the angle of view of the image sensor so that the detection points of the distance measuring sensor correspond to the provisional area obtained by coordinate transformation by the area projection unit. (Supplementary Note 5) The clutter determination processing device according to any one of Supplementary Notes 2 to 4, wherein the clutter area determination unit determines the clutter discrimination area according to operational conditions including at least one of the height and depression angle of the distance measuring sensor.(Supplementary Note 6) The clutter determination processing device according to any one of Supplements 2 to 5, wherein the classes corresponding to the background and non-background are defined as a plurality of classes according to the types of the background and non-background, and the image analysis unit classifies the image of the provisional region into the two or more classes including the plurality of classes corresponding to the background and non-background. (Supplementary Note 7) The clutter determination processing device according to any one of Supplements 2 to 6, wherein the image analysis unit performs the class classification using a trained model for inferring the class corresponding to the input image from the image, and the trained model is created by training using, as training images, an image of the background, an image of the non-background, and an image in which the background and non-background are mixed. (Supplementary Note 8) The clutter determination processing device according to any one of Supplements 2 to 7, wherein, when a plurality of the detection points are detected, the clutter region determination unit, the region projection unit, and the image extraction unit perform processing for each of the detection points. (Supplementary Note 9) The clutter determination processing device according to any one of Supplements 2 to 8, characterized in that the image extraction unit extracts an image of the provisional area together with an image of a predetermined area surrounding the provisional area from the captured image, and inputs the extracted images to the image analysis unit. (Supplementary Note 10) The clutter determination processing device according to any one of Supplements 2 to 9, characterized in that it determines whether the target detected by the ranging sensor matches the object detected by the image sensor using first detection information acquired by the ranging sensor and second detection information acquired by the image sensor, and executes processing by the clutter area determination unit, the area projection unit, the image extraction unit, the image analysis unit, and the clutter determination unit for a detection point of the ranging sensor where the target detected by the ranging sensor does not match the object detected by the image sensor.(Supplementary Note 11) A clutter determination processing device according to Supplementary Note 1, comprising: a clutter area determination unit that pre-sets a clutter discrimination area in which the ranging sensor is expected to detect clutter, and determines whether the detection point of the ranging sensor is within the set clutter discrimination area; an image extraction unit that extracts an image corresponding to the clutter discrimination area from the captured image when the detection point of the ranging sensor is within the clutter discrimination area; and an area projection unit that determines, as a provisional area, an area in which the area of ​​the target corresponding to the detection point of the ranging sensor is projected onto an image of the clutter discrimination area; wherein the image analysis unit classifies the image of the clutter discrimination area extracted by the image extraction unit into two or more classes including at least a background and a non-background class by segmenting each pixel; and the clutter determination unit determines whether the detection point corresponding to the provisional area is clutter based on the class for each pixel in the provisional area. (Supplementary Note 12) The clutter determination processing device according to Supplementary Note 11, wherein the clutter region determination unit determines the clutter discrimination region according to operational conditions including at least one of the height and depression angle of the ranging sensor. (Supplementary Note 13) The clutter determination processing device according to Supplementary Note 11 or 12, wherein the image analysis unit performs the segmentation in units of multiple pixels. (Supplementary Note 14) The clutter determination processing device according to Supplementary Note 11 or 12, wherein the image analysis unit performs the segmentation on pixels thinned out at regular intervals. (Appendix 15) A clutter determination processing device described in any one of Appendices 11 to 14, characterized in that it uses first detection information acquired by the ranging sensor and second detection information acquired by the imaging sensor to determine whether the target detected by the ranging sensor matches the object detected by the imaging sensor, and executes processing by the clutter area determination unit, the area projection unit, the image extraction unit, the image analysis unit and the clutter determination unit for the detection points of the ranging sensor where the target detected by the ranging sensor does not match the object detected by the imaging sensor.(Supplementary Note 16) A clutter determination processing device according to Supplementary Note 1, comprising: a clutter area determination unit that determines whether the detection point of the ranging sensor is within a predetermined clutter determination area; and an area projection unit that determines an area obtained by projecting onto the captured image an area of ​​a target corresponding to the detection point within the clutter determination area as a provisional area, wherein the image analysis unit performs an object detection process that detects objects other than the background, and the clutter determination unit determines whether the target corresponding to the provisional area matches the object detected by the object detection process based on the position of the detection point of the ranging sensor on the captured image and the provisional area, the position of the object detected by the object detection process and the object detection area rectangle, and determines that the detection point corresponding to the target of the ranging sensor that does not match the object detected by the object detection process is clutter. (Supplementary Note 17) The clutter determination processing device according to any one of Supplements 2 to 16, further comprising: a provisional size determination unit that determines a provisional size of the target based on received signal information acquired by the distance measurement sensor, wherein the area projection unit determines the provisional area using the provisional size and the positions of the detection points. (Supplementary Note 18) The clutter determination processing device according to Supplementary Note 17, wherein the provisional size determination unit determines the provisional size to be a predetermined fixed size. (Supplementary Note 19) The clutter determination processing device according to Supplementary Note 17, wherein the provisional size determination unit determines the provisional size based on a received signal level of each detection point obtained from the distance measurement sensor. (Supplementary Note 20) The clutter determination processing device according to any one of Supplements 2 to 19, wherein the area projection unit determines a simplified area on image pixels as a provisional area for a non-rectangular area obtained by projecting an area of ​​the target corresponding to the detection point of the distance measurement sensor onto the captured image by coordinate transformation, and outputs the simplified area to the clutter determination unit. (Supplementary Note 21) The clutter determination processing device according to any one of Supplementary Notes 2 to 20, wherein the image analysis unit outputs a probability value or a plurality of classification intermediate values ​​of each class corresponding to the input image, and the clutter determination unit determines that the clutter is a background class when the probability value of the background class exceeds a certain threshold value.(Supplementary Note 22) The clutter determination processing device according to any one of Supplements 2 to 21, characterized in that the image analysis unit takes in a plurality of the captured images acquired by the imaging sensor over a plurality of shots or a plurality of frames as input images, performs the analysis, and outputs the single analysis result. (Supplementary Note 23) The clutter determination processing device according to any one of Supplements 2 to 22, characterized in that the image analysis unit performs the analysis using detection results from the distance measurement sensor in addition to the image to be analyzed, and outputs the single analysis result. (Supplementary Note 24) The clutter determination processing device according to any one of Supplements 2 to 21, characterized in that the image analysis unit performs processing on each of the captured images of a plurality of shots or a plurality of frames, and the clutter determination unit determines whether the detection point is clutter based on a plurality of analysis results from the image analysis unit corresponding to each of the plurality of shots or a plurality of frames. (Appendix 25) The clutter determination processing device described in any one of Appendices 1 to 24, characterized in that the clutter determination unit determines whether the detection point is clutter or not using the result of the analysis by the image analysis unit and at least one or a combination of the distance, speed, and received signal level detected by the ranging sensor. (Supplementary Note 26) An information processing device comprising: a transceiver unit that receives object detection results from a ranging sensor, receives captured images from an imaging sensor that captures the detection range in which the ranging sensor can detect objects, and controls the ranging sensor and the imaging sensor; an image analysis unit that sets a provisional size of the target at the detection point detected by the ranging sensor in an assumed clutter detection area, and analyzes an image including an area corresponding to the provisional target size of each detection point of the ranging sensor projected onto the captured image of the imaging sensor; a clutter determination unit that uses the analysis results of the image analysis unit to determine whether each detection point of the ranging sensor is clutter; a target output unit that removes detection points determined to be clutter by the clutter determination unit from all detection results obtained by the ranging sensor and outputs them; and a sensor processing unit that generates integrated information that integrates the detection results output from the target output unit and object detection results using the captured images.(Appendix 27) A program that causes a computer system to execute the following functional processes: setting a provisional size of a target at a detection point detected by a ranging sensor in an assumed clutter detection area, and analyzing an image including an area corresponding to the provisional target size at each detection point of the ranging sensor projected onto an image captured by an imaging sensor; using the results of the analysis, determining whether each detection point of the ranging sensor is clutter or not; and removing the detection points determined to be clutter from all detection results obtained by the ranging sensor and outputting them. (Appendix 28) A clutter determination processing method in a clutter determination processing device, comprising: a functional process of setting a provisional size of a target at a detection point detected by a ranging sensor in an assumed clutter detection area, and analyzing an image including an area corresponding to the provisional target size of each detection point of the ranging sensor projected onto an image captured by an imaging sensor; a functional process of using the results of the analysis to determine whether each detection point of the ranging sensor is clutter; and a functional process of removing the detection points determined to be clutter from all detection results obtained by the ranging sensor and outputting them. (Supplementary Note 101) A clutter determination processing device characterized by comprising: a provisional size determination unit that determines the provisional size of a corresponding target object individually for each detection point acquired by a ranging sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the ranging sensor; a clutter image processing unit that analyzes an image including the provisional area projected onto an image captured by an imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; a clutter determination unit that determines whether each detection point of the ranging sensor is clutter using the analysis results of the clutter image processing unit; and a target output unit that removes detection points determined to be clutter by the clutter determination unit from all detection results acquired by the ranging sensor and outputs them.(Supplementary Note 102) The clutter determination processing device according to Supplementary Note 101, wherein the clutter image processing unit comprises: a clutter area determination unit that determines whether a detection point of the distance measuring sensor is within the predetermined clutter determination area; an area projection unit that determines, as the provisional area, an area obtained by projecting onto the captured image an area of ​​a target corresponding to the detection point within the clutter determination area; an image extraction unit that extracts an image of the provisional area from the captured image; and an image analysis unit that classifies the image of the provisional area extracted by the image extraction unit into two or more classes including at least a background and a non-background class, wherein the clutter determination unit determines, as clutter, the detection point corresponding to the provisional area determined to be the background by the image analysis unit. (Supplementary Note 103) The clutter determination processing device according to Supplementary Note 102, wherein the image extraction unit extracts an image of the clutter determination area from the captured image, and outputs, from a plurality of divided images obtained by dividing the image by a predetermined number, the divided image including the provisional area coordinate-transformed by the area projection unit to the image analysis unit. (Supplementary Note 104) The clutter determination processing device according to Supplementary Note 102, wherein the imaging sensor captures an image by adjusting the angle of view of the imaging sensor so that the detection points of the distance measuring sensor correspond to the provisional area whose coordinates have been transformed by the area projection unit. (Supplementary Note 105) The clutter determination processing device according to any one of Supplementary Notes 102 to 104, wherein the image extraction unit extracts an image of the provisional area together with an image of a predetermined area surrounding the provisional area from the captured image, and inputs the extracted images to the image analysis unit.(Supplementary Note 106) The clutter image processing device according to Supplementary Note 101, wherein the clutter image processing unit comprises: a clutter area determination unit that determines whether the detection point of the ranging sensor is within the predetermined clutter determination area; an image extraction unit that extracts an image corresponding to the clutter determination area from the captured image when the detection point of the ranging sensor is within the clutter determination area; an area projection unit that projects an area of ​​the target corresponding to the detection point of the ranging sensor onto the image of the clutter determination area and determines the area as the provisional area; and an image analysis unit that classifies each pixel into two or more classes including at least a background and a non-background class by segmenting the image of the clutter determination area extracted by the image extraction unit, wherein the clutter determination unit determines whether the detection point corresponding to the provisional area is clutter based on the frequency of each class in the provisional area. (Supplementary Note 107) The clutter determination processing device according to Supplementary Note 106, wherein the image analysis unit performs the segmentation in units of multiple pixels. (Supplementary Note 108) The clutter determination processing device according to Supplementary Note 106, wherein the image analysis unit performs the segmentation on pixels thinned out at regular intervals. (Supplementary Note 109) The clutter determination processing device according to any one of Supplements 102 to 108, wherein the classes corresponding to the background and non-background are defined as a plurality of classes according to the types of the background and non-background, and the image analysis unit classifies the image extracted by the image extraction unit into the two or more classes including the plurality of classes corresponding to the background and non-background. (Supplementary Note 110) The clutter determination processing device according to any one of Supplements 102 to 109, wherein the image analysis unit classifies the classes using a trained model for inferring the corresponding class from an input image, and the trained model is created by training using, as training images, a background image, a non-background image, and an image in which the background and non-background are mixed.(Appendix 111) A clutter determination processing device described in any one of Appendices 102 to 110, characterized in that it uses first detection information acquired by the ranging sensor and second detection information acquired by the imaging sensor to determine whether the target detected by the ranging sensor matches the object detected by the imaging sensor, and executes processing by the clutter image processing unit and the clutter determination unit for the detection points of the ranging sensor where the target detected by the ranging sensor does not match the object detected by the imaging sensor. (Appendix 112) The clutter image processing unit comprises: a clutter area determination unit that determines whether the detection point of the ranging sensor is within the predetermined clutter discrimination area; an area projection unit that determines the area obtained by projecting the area of ​​the target corresponding to the detection point within the clutter discrimination area onto the captured image as the provisional area; and an image analysis unit that performs object detection processing that detects objects other than the background from the captured image acquired by the image sensor, wherein the clutter determination unit determines whether the target corresponding to the provisional area matches the object detected by the object detection processing based on the position of the detection point of the ranging sensor on the captured image and the provisional area, the position of the object other than the background detected by the object detection processing and the object detection area rectangle, and determines that the detection point corresponding to the target of the ranging sensor that does not match the object detected by the object detection processing is clutter. (Supplementary Note 113) The clutter determination processing device according to any one of Supplementary Notes 102 to 112, characterized in that the area projection unit determines a simplified area on image pixels as a provisional area for a non-rectangular area obtained by projecting an area of ​​a target corresponding to a detection point of the distance measuring sensor onto the captured image by coordinate transformation, and outputs the simplified area. (Supplementary Note 114) The clutter determination processing device according to any one of Supplementary Notes 102 to 113, characterized in that the image analysis unit outputs a probability value or a plurality of classification intermediate values ​​for each class corresponding to the input image from the input image, and the clutter determination unit determines whether or not there is clutter using at least one of the probability values ​​and the classification intermediate values.(Supplementary Note 115) The clutter determination processing device according to any one of Supplements 102 to 114, wherein the clutter region determination unit determines the clutter discrimination region according to operational conditions including at least one of the height and depression angle of the ranging sensor. (Supplementary Note 116) The clutter determination processing device according to any one of Supplements 102 to 115, wherein, when multiple detection points are detected by the ranging sensor, the clutter image processing unit and the clutter determination unit perform processing for each of the detection points. (Supplementary Note 117) The clutter determination processing device according to any one of Supplements 102 to 116, wherein the image analysis unit classifies input images or pixels into two or more classes including at least a background and a non-background class, or performs object detection processing to detect objects other than the background from the input image, using a trained model trained and generated in advance by deep learning. (Supplementary Note 118) The clutter determination processing device according to Supplementary Note 101, wherein the provisional size determination unit determines the provisional size to a predetermined fixed size. (Supplementary Note 119) The clutter determination processing device according to Supplementary Note 101, wherein the provisional size determination unit determines the provisional size based on a received signal level of each of the detection points obtained from the distance measurement sensor. (Supplementary Note 120) The clutter determination processing device according to any one of Supplementary Notes 101 to 119, wherein the clutter determination unit determines whether or not the detection point is clutter using the result of the analysis by the clutter image processing unit and at least one or a combination of the distance, speed, and received signal level detected by the distance measurement sensor. (Supplementary Note 121) The clutter determination processing device according to any one of Supplementary Notes 102 to 117, wherein the image analysis unit performs the analysis using the detection result of the distance measurement sensor in addition to the image to be analyzed, and outputs one of the analysis results.(Supplementary Note 122) The clutter determination processing device according to any one of Supplementary Notes 101 to 121, characterized in that processing by the clutter image processing unit is performed on a plurality of frames, and the clutter determination unit determines whether or not the detection point is clutter based on analysis results of a plurality of the clutter image processing units corresponding to each of the plurality of frames. (Supplementary Note 123) The clutter determination processing device according to any one of Supplementary Notes 102 to 117, 121, characterized in that the image analysis unit takes in a plurality of the captured images acquired over a plurality of shots or a plurality of frames as input images, performs the analysis, and outputs one of the analysis results. (Supplementary Note 124) An information processing device comprising: a transceiver unit that receives object detection results from a ranging sensor, receives captured images from an imaging sensor that captures an image of a detection range in which the ranging sensor can detect objects, and controls the ranging sensor and the imaging sensor; a clutter image processing unit that determines a provisional size of a corresponding target object for each detection point acquired by the ranging sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the ranging sensor, and analyzes an image including the provisional area projected onto the captured image of the imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; a clutter determination unit that uses the analysis results of the clutter image processing unit to determine whether each detection point of the ranging sensor is clutter; a target output unit that removes detection points determined to be clutter by the clutter determination unit from all detection results acquired by the ranging sensor and outputs them; and a sensor processing unit that generates integrated information that integrates the detection results output from the target output unit and object detection results using the captured images.(Appendix 125) A program that causes a computer system to execute the following functional processes: for each detection point acquired by a ranging sensor within a predetermined clutter discrimination area, determining a provisional size of the corresponding target based on the received signal of each detection point of the ranging sensor, and analyzing an image including the provisional area projected onto an image captured by an imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; using the results of the analysis, determining whether each detection point of the ranging sensor is clutter; and removing the detection points determined to be clutter from all detection results acquired by the ranging sensor and outputting them. (Appendix 126) A clutter determination processing method in a clutter determination processing device, comprising: a functional process of determining a provisional size of an individual corresponding target for each detection point acquired by a ranging sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the ranging sensor, and analyzing an image including the provisional area projected onto an image captured by an imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; a functional process of determining whether each detection point of the ranging sensor is clutter using the results of the analysis; and a functional process of removing the detection points determined to be clutter from all detection results acquired by the ranging sensor and outputting them.

[0129] REFERENCE SIGNS LIST 1 Information processing device, 2 Distance measurement sensor, 3 Imaging sensor, 4 Vehicle control unit, 5 Display control unit, 11 First transmission / reception unit, 12, 12a, 12b Clutter determination processing unit, 13 Sensor processing unit, 14 Second transmission / reception unit, 15 Vehicle communication unit, 16 Storage unit, 21 Sensor unit, 22, 32 Control unit, 23 Distance measurement sensor ECU, 31 Imaging unit, 33 Imaging sensor ECU, 41 Vehicle control ECU, 51, 105 Memory, 52, 104 Processor, 53 Communication I / F, 101 First communication I / F, 102 Vehicle communication I / F, 103 Second communication I / F, 121 Clutter area determination unit, 122 Provisional size determination unit, 123 Area projection unit, 124, 124a Image extraction unit, 125, 125a, 125b Image analysis unit, 126, 126a, 126b clutter determination unit, 127 target output unit, 128, 128a, 128b clutter image processing unit, 251, 251a model creation unit, 252, 252a learned model storage unit, 253 image classification unit, 253a pixel classification unit.

Claims

1. A clutter determination processing device comprising: a provisional size determination unit that determines the provisional size of a corresponding target object for each detection point acquired by a ranging sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the ranging sensor, and a clutter image processing unit that analyzes an image including the provisional area projected onto an image captured by an imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; a clutter determination unit that uses the analysis results of the clutter image processing unit to determine whether each detection point of the ranging sensor is clutter; and a target output unit that removes detection points that the clutter determination unit has determined to be clutter from all detection results acquired by the ranging sensor and outputs them.

2. The clutter image processing unit comprises: a clutter area determination unit that determines whether the detection point of the ranging sensor is within the predetermined clutter discrimination area; an area projection unit that determines the area obtained by projecting the area of ​​the target corresponding to the detection point within the clutter discrimination area onto the captured image as the provisional area; an image extraction unit that extracts an image of the provisional area from the captured image; and an image analysis unit that classifies the image of the provisional area extracted by the image extraction unit into two or more classes including at least a background and a non-background class, and the clutter determination unit determines that the detection point corresponding to the provisional area determined to be the background by the image analysis unit is clutter.

3. The clutter determination processing device according to claim 2, characterized in that the image extraction unit extracts an image of the clutter determination area from the captured image, and outputs to the image analysis unit, among a plurality of divided images obtained by dividing the image by a predetermined number, the divided image that includes the provisional area that has been coordinate-transformed by the area projection unit.

4. The clutter determination processing device according to claim 2, characterized in that the imaging sensor adjusts the angle of view of the imaging sensor so that the detection point of the ranging sensor corresponds to the provisional area after coordinate transformation by the area projection unit, and captures an image.

5. A clutter determination processing device as described in any one of claims 2 to 4, characterized in that the image extraction unit extracts an image of the provisional area as well as an image of a specified area surrounding the provisional area from the captured image and inputs them to the image analysis unit.

6. The clutter image processing unit comprises: a clutter area determination unit that determines whether the detection point of the ranging sensor is within the predetermined clutter discrimination area; an image extraction unit that extracts an image corresponding to the clutter discrimination area from the captured image when the detection point of the ranging sensor is within the clutter discrimination area; an area projection unit that determines the provisional area as an area obtained by projecting the area of ​​the target corresponding to the detection point of the ranging sensor onto the image of the clutter discrimination area; and an image analysis unit that performs segmentation on the image of the clutter discrimination area extracted by the image extraction unit to classify each pixel into two or more classes including at least a background and a non-background class, wherein the clutter determination unit determines whether the detection point corresponding to the provisional area is clutter based on the frequency of each class in the provisional area.

7. The clutter determination processing device according to claim 6, wherein said image analysis section performs said segmentation in units of a plurality of pixels.

8. The clutter determination processing device according to claim 6, wherein said image analysis section performs said segmentation on pixels thinned out at regular intervals.

9. A clutter determination processing device as described in any one of claims 2 to 8, characterized in that each of the classes corresponding to the background and non-background is defined as a plurality of classes according to the type of the background and non-background, and the image analysis unit classifies the image extracted by the image extraction unit into the two or more classes including the plurality of classes corresponding to the background and non-background.

10. A clutter determination processing device as described in any one of claims 2 to 9, characterized in that the image analysis unit classifies the classes using a trained model for inferring the corresponding classes from the input images, and the trained model is created by learning using, as training images, background images, non-background images, and images in which background and non-background are mixed.

11. A clutter determination processing device as described in any one of claims 2 to 10, characterized in that it uses first detection information acquired by the ranging sensor and second detection information acquired by the imaging sensor to determine whether the target detected by the ranging sensor matches the object detected by the imaging sensor, and executes processing by the clutter image processing unit and the clutter determination unit for the detection points of the ranging sensor where the target detected by the ranging sensor does not match the object detected by the imaging sensor.

12. The clutter image processing unit comprises: a clutter area determination unit that determines whether the detection point of the ranging sensor is within the predetermined clutter discrimination area; an area projection unit that determines the area obtained by projecting the area of ​​the target corresponding to the detection point within the clutter discrimination area onto the captured image as the provisional area; and an image analysis unit that performs object detection processing that detects objects other than the background from the captured image acquired by the image sensor; and the clutter determination unit determines whether the target corresponding to the provisional area matches the object detected by the object detection processing based on the position of the detection point of the ranging sensor on the captured image and the provisional area, the position of the object other than the background detected by the object detection processing and the object detection area rectangle, and determines that the detection point corresponding to the target of the ranging sensor that does not match the object detected by the object detection processing is clutter.

13. A clutter determination processing device as described in any one of claims 2 to 12, characterized in that the area projection unit defines a simplified area on the image pixels as a provisional area for a non-rectangular area obtained by projecting the area of ​​the target corresponding to the detection point of the ranging sensor onto the captured image by coordinate transformation, and outputs the simplified area.

14. A clutter determination processing device as described in any one of claims 2 to 13, characterized in that the image analysis unit outputs a probability value or a plurality of classification intermediate values ​​for each class corresponding to the input image, and the clutter determination unit determines whether or not the image is clutter using at least one of the probability values ​​and the classification intermediate values.

15. A clutter determination processing device as described in any one of claims 2 to 14, characterized in that the clutter area determination unit determines the clutter discrimination area according to operational conditions including at least one of the height and depression angle of the ranging sensor.

16. A clutter determination processing device as described in any one of claims 2 to 15, characterized in that when the distance measuring sensor detects multiple detection points, processing by the clutter image processing unit and the clutter determination unit is carried out for each detection point.

17. A clutter determination processing device as described in any one of claims 2 to 16, characterized in that the image analysis unit uses a trained model that has been trained and generated in advance by deep learning to classify input images or pixels into two or more classes including at least a background and a non-background class, or performs object detection processing to detect objects other than the background from the input image.

18. The clutter determination processing device according to claim 1, wherein said provisional size determination unit determines said provisional size to be a predetermined fixed size.

19. The clutter determination processing device according to claim 1, wherein said provisional size determination unit determines said provisional size based on the received signal level at each of said detection points obtained from said distance measurement sensor.

20. A clutter determination processing device as described in any one of claims 1 to 19, characterized in that the clutter determination unit determines whether the detection point is clutter or not using the results of the analysis by the clutter image processing unit and at least one or a combination of the distance, speed and received signal level detected by the ranging sensor.

21. A clutter determination processing device as described in any one of claims 2 to 17, characterized in that the image analysis unit performs the analysis using the detection results of the ranging sensor in addition to the image of the object of analysis, and outputs one analysis result.

22. A clutter determination processing device as described in any one of claims 1 to 21, characterized in that processing by the clutter image processing unit is performed on multiple frames, and the clutter determination unit determines whether the detected point is clutter or not based on the analysis results of multiple clutter image processing units corresponding to each of the multiple frames.

23. A clutter determination processing device as described in any one of claims 2 to 17 and 21, characterized in that the image analysis unit takes in multiple captured images acquired over multiple shots or multiple frames as input images, performs the analysis, and outputs one analysis result.

24. An information processing device comprising: a transceiver unit that receives object detection results from a ranging sensor, receives captured images from an imaging sensor that captures the detection range in which the ranging sensor can detect objects, and controls the ranging sensor and the imaging sensor; a clutter image processing unit that determines a provisional size of each corresponding target object for each detection point acquired by the ranging sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the ranging sensor, and analyzes an image including the provisional area projected onto the image captured by the imaging sensor as a target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; a clutter determination unit that uses the analysis results of the clutter image processing unit to determine whether each detection point of the ranging sensor is clutter; a target output unit that removes detection points determined to be clutter by the clutter determination unit from all detection results acquired by the ranging sensor and outputs them; and a sensor processing unit that generates integrated information that integrates the detection results output from the target output unit and object detection results using the captured images.

25. A program for causing a computer system to execute the following functional processes: for each detection point acquired by a ranging sensor within a predetermined clutter discrimination area, determining the provisional size of the corresponding target based on the received signal at each detection point of the ranging sensor, and analyzing an image including the provisional area projected onto the image captured by the imaging sensor as the target area corresponding to each detection point of the ranging sensor based on the provisional size and the position of the detection point; using the results of the analysis, determining whether each detection point of the ranging sensor is clutter; and removing the detection points determined to be clutter from all detection results acquired by the ranging sensor and outputting them.

26. A clutter determination processing method in a clutter determination processing device, comprising: a functional process of determining a provisional size of an individual corresponding target for each detection point acquired by a distance measurement sensor within a predetermined clutter discrimination area based on the received signal of each detection point of the distance measurement sensor, and analyzing an image including the provisional area projected onto an image captured by an imaging sensor as a target area corresponding to each detection point of the distance measurement sensor based on the provisional size and the position of the detection point; a functional process of determining whether each detection point of the distance measurement sensor is clutter using the results of the analysis; and a functional process of removing the detection points determined to be clutter from all detection results acquired by the distance measurement sensor and outputting them.

Images