Object recognition device, object recognition processing method, and recording medium

Abstract

This object recognition device for executing object recognition processing on the basis of measurement data from a distance measurement sensor and image data from a camera executes: a prescribed first object recognition process for a short-distance region in which the distance from a prescribed base point is within a first distance set as the shortest distance at which an object can be recognized using an object recognition process based on image data; a prescribed second object recognition process, different from the first object recognition process, for a medium-distance region in which the distance from the prescribed base point exceeds the first distance and is within a second distance set as the upper-limit value of the distance at which an object can be recognized using the object recognition process based on image data; and a prescribed third object recognition process, different from the first object recognition process and the second object recognition process, for a long-distance region in which the distance from the prescribed base point exceeds the second distance.

Description

Object recognition device, object recognition processing method, and recording medium

[0001] The present disclosure relates to an object recognition device, an object recognition processing method, and a recording medium.

[0002] In the field of autonomous driving technology for vehicles and advanced driver assistance systems, development is underway of object detection technology that utilizes imaging devices such as monocular cameras or stereo cameras, and ranging sensors that measure distances based on a group of reflected points that reflect irradiated waves such as LiDAR (Light Detection and Ranging or Laser Imaging Detection and Ranging) or millimeter-wave radar.In recent years, devices have been proposed that detect objects by fusing image data generated by the imaging device with measurement data from a ranging sensor.

[0003] For example, Patent Document 1 proposes an apparatus for recognizing a preceding vehicle ahead of a host vehicle using sensor fusion, in which image processing is performed with simple calculations that require a small amount of calculation. Specifically, Patent Document 1 discloses a technology in which a preceding vehicle region determining means determines the preceding vehicle region based on clustering of distance measurement results from a scanning laser radar performed by a clustering processing means, an edge image calculation processing means processes an image captured by a monocular camera of at least the preceding vehicle region into compressed edge binary image information, an edge binary image information collecting means collects this edge binary image information as image feature amount information, and a recognition determination means recognizes the preceding vehicle region as a preceding vehicle by comparing this information with image feature amount information of a determination criterion without performing complex image processing that requires a large amount of calculation, such as correlation calculation of the captured image or contour extraction, and further a determination criterion update means updates the image feature amount information of the determination criterion, and a predicted position update means updates a predicted position of the preceding vehicle.

[0004] Furthermore, Patent Document 2 proposes a device that, when performing vehicle recognition using a laser radar, combines vehicle recognition using an image sensor to eliminate reflections from vehicles other than the vehicle ahead, roadside objects, etc., thereby achieving highly accurate recognition. Specifically, Patent Document 2 discloses a technology in which, based on the positions of each reflection point identified by a laser radar module, a CPU determines a group of reflection points that exist at approximately equidistant positions within a range that is approximately the same width as the vehicle as a vehicle candidate point group, coordinate-transforms this vehicle candidate point group into a camera coordinate system of a CCD camera, and compares it with a rectangular area extracted by the camera module; and if the vehicle candidate point group after coordinate transformation approximately matches the rectangular area, the CPU determines that the vehicle candidate point group is a vehicle ahead.

[0005] JP 2005-090974 A JP 2003-084064 A

[0006] However, the imaging device and the ranging sensor have different characteristics in terms of temporal resolution and spatial resolution. Therefore, it is difficult to vary the temporal resolution and spatial resolution of the distance range to be recognized for each of the imaging device and ranging sensor. In the above-mentioned Patent Documents 1 and 2, the range of the distance to be recognized is defined as a predetermined recognition range in which a preceding vehicle or a vehicle ahead exists. However, if processing can be performed that takes advantage of the respective characteristics of the imaging device and ranging sensor depending on the range of distance from the vehicle, the accuracy of recognizing objects around the vehicle can be improved.

[0007] The present disclosure has been made in consideration of the above problems, and an object of the present disclosure is to provide an object recognition device, an object recognition processing method, and a recording medium that enable object recognition processing that makes use of the respective characteristics of an imaging device and a ranging sensor, thereby improving object recognition accuracy.

[0008] In order to solve the above problem, according to one aspect of the present disclosure, there is provided an object recognition device comprising: a ranging sensor that measures at least the distance to a reflection point based on the reflected wave of an irradiated irradiation wave; a camera that generates image data of a shooting range; and one or more processing devices that perform object recognition processing based on the measurement data of the ranging sensor and the image data of the camera, wherein the one or more processing devices perform a predetermined first object recognition processing for a short-distance area where the distance from a predetermined base point is within a first distance set as the shortest distance at which an object can be recognized by the object recognition processing based on the image data; a predetermined second object recognition processing different from the first object recognition processing for a medium-distance area where the distance from the predetermined base point is within a second distance that exceeds the first distance and is set as the upper limit of the distance at which an object can be recognized by the object recognition processing based on the image data; and a predetermined third object recognition processing different from the first object recognition processing and the second object recognition processing for a long-distance area where the distance from the predetermined base point exceeds the second distance.

[0009] Furthermore, in order to solve the above problem, according to another aspect of the present disclosure, there is provided an object recognition processing method that performs object recognition processing based on measurement data from a distance measuring sensor that measures at least the distance to a reflection point based on the reflected wave of an irradiated irradiation wave, and image data from a camera that generates image data of a shooting range, in which the computer performs a predetermined first object recognition processing for a short-distance area where the distance from a predetermined base point is within a first distance set as the shortest distance at which an object can be recognized by the object recognition processing based on the image data, a predetermined second object recognition processing different from the first object recognition processing for a medium-distance area where the distance from the predetermined base point is greater than the first distance and is within a second distance set as the upper limit of the distance at which an object can be recognized by the object recognition processing based on the image data, and a predetermined third object recognition processing different from the first object recognition processing and the second object recognition processing for a long-distance area where the distance from the predetermined base point is greater than the second distance.

[0010] In addition, in order to solve the above problem, according to another aspect of the present disclosure, there is provided a non-transitory tangible recording medium having recorded thereon a computer program that causes a computer to perform object recognition processing based on measurement data from a distance measuring sensor that measures at least the distance to a reflection point based on the reflected wave of an irradiated irradiation wave, and image data from a camera that generates image data of a shooting range, wherein the computer program causes the computer to perform a predetermined first object recognition processing for a short-distance area where the distance from a predetermined base point is within a first distance set as the shortest distance at which an object can be recognized by the object recognition processing based on the image data, a predetermined second object recognition processing different from the first object recognition processing for a medium-distance area where the distance from the predetermined base point is greater than the first distance and is within a second distance set as the upper limit of the distance at which an object can be recognized by the object recognition processing based on the image data, and a predetermined third object recognition processing different from the first object recognition processing and the second object recognition processing for a long-distance area where the distance from the predetermined base point is greater than the second distance.

[0011] As described above, according to the present disclosure, it is possible to improve the object recognition accuracy by taking advantage of the respective features of the imaging device and the distance measuring sensor.

[0012] FIG. 1 is a schematic diagram showing an example configuration of a vehicle equipped with an object recognition device according to an embodiment of the present disclosure. FIG. 2 is an explanatory diagram showing the spatial resolution of LiDAR and a camera. FIG. 3 is an explanatory diagram showing the minimum distance at which stereo matching is possible for a stereo camera. FIG. 4 is a block diagram showing an example configuration of an object recognition device according to the same embodiment. FIG. 5 is a flowchart showing a routine of point cloud data processing by a point cloud data processing unit in object recognition processing according to the same embodiment. FIG. 6 is a flowchart showing a routine of image data processing by an image data processing unit in object recognition processing according to the same embodiment. FIG. 7 is a flowchart showing a routine of first object recognition processing for a short-distance area by an object recognition processing unit in object recognition processing according to the same embodiment. FIG. 8 is a flowchart showing a routine of second object recognition processing for a medium-distance area by an object recognition processing unit in object recognition processing according to the same embodiment. FIG. 9 is a flowchart showing a routine of third object recognition processing for a long-distance area by an object recognition processing unit in object recognition processing according to the same embodiment. FIG. 10 is a flowchart showing an object recognition processing method by an object recognition device according to the same embodiment. FIG. 11 is a flowchart showing an object recognition processing method by an object recognition device according to a modified example of the present disclosure. FIG. 12 is a flowchart showing a routine of driving environment determination processing by an object recognition device according to a modified example.

[0013] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0014] 1. Overall Configuration of Object Recognition Device First, an example of the overall configuration of a vehicle will be described as an example of a moving body equipped with an object recognition device according to an embodiment of the present disclosure. In the following embodiment, an example of an object recognition device using LiDAR as an example of a ranging sensor will be described.

[0015] Fig. 1 is a schematic diagram showing an example of the configuration of a vehicle 1 equipped with an object recognition device 30 according to this embodiment. The vehicle 1 shown in Fig. 1 is configured as a front-wheel drive vehicle in which a driving torque output from a driving force source 3 that generates driving torque for the vehicle is transmitted to the front wheels. The driving force source 3 may be an internal combustion engine such as a gasoline engine or a diesel engine, a driving motor, or both an internal combustion engine and a driving motor.

[0016] The vehicle 1 is not limited to a combination of drive wheels or a drive method, and may be, for example, a rear-wheel drive vehicle, a four-wheel drive vehicle, or an electric vehicle equipped with a drive motor for each wheel. Furthermore, if the vehicle 1 is an electric vehicle or a hybrid electric vehicle, the vehicle 1 is equipped with a secondary battery that stores power supplied to the drive motor, and a motor or a generator such as a fuel cell that generates power to charge the battery.

[0017] The vehicle 1 is equipped with a driving force source 3, an electric steering device 11, and brake devices 7LF, 7RF, 7LR, and 7RR (hereinafter collectively referred to as "brake devices 7" unless a distinction is required) as devices used to control the operation of the vehicle 1. The driving force source 3 outputs driving torque that is transmitted to a front-wheel drive shaft 5 via a transmission and a front-wheel differential mechanism (not shown). The operation of the driving force source 3 and the transmission is controlled by a vehicle control unit 20 that includes one or more electronic control units (ECUs: Electronic Control Units).

[0018] An electric steering device 11 is provided on the front-wheel drive shaft 5. The electric steering device 11 includes an electric motor and a gear mechanism (not shown), and is controlled by a vehicle control unit 20 to adjust the steering angle of the left and right front wheels. During manual driving, the vehicle control unit 20 controls the electric steering device 11 based on the steering angle of the steering wheel 13 operated by the driver. During autonomous driving, the vehicle control unit 20 controls the electric steering device 11 based on a target steering angle set according to the planned driving trajectory.

[0019] The brake devices 7LF, 7RF, 7LR, and 7RR apply braking forces to the front, rear, left, and right wheels, respectively. The brake devices 7 are configured as hydraulic brake devices, for example, and generate predetermined braking forces by controlling the hydraulic pressure supplied to each brake device 7 by a hydraulic pressure control unit 9. If the vehicle 1 is an electric vehicle or a hybrid electric vehicle, the brake devices 7 are used in combination with regenerative braking using a drive motor.

[0020] The vehicle control unit 20 includes one or more electronic control devices that control the operation of the driving force source 3, the electric steering device 11, and the hydraulic control unit 9. If the vehicle 1 is equipped with a transmission, the vehicle control unit 20 may have a function to control the operation of the transmission. The vehicle control unit 20 is configured to be able to perform automatic driving control and emergency braking control of the vehicle 1 using information about objects recognized by the object recognition device 30.

[0021] The object recognition device 30 includes a LiDAR 31, a camera 33, and a processing device 50. The LiDAR 31 and the camera 33 are installed, for example, on the upper part of the front window facing the passenger compartment inside the vehicle or on the front part of the vehicle body, with the measurement direction or shooting direction facing forward.

[0022] The LiDAR 31 is a type of distance measurement sensor that measures at least the distance to a reflection point based on the reflected wave of an emitted irradiation wave. The LiDAR 31 emits laser light (optical waves) as a type of irradiation wave in multiple directions ahead of the vehicle 1 and receives reflected light (reflected waves) of the laser light, and acquires data (hereinafter also referred to as "point cloud data") of the positions of the reflection points in three-dimensional space based on the laser light and reflected light. The point cloud data of the LiDAR 31 corresponds to measurement data of a distance measurement sensor.

[0023] For example, the LiDAR 31 may be a time-of-flight (ToF) LiDAR that calculates the position of a reflection point in three-dimensional space based on data on the direction in which the reflected light is received and data on the time between emitting laser light and receiving the reflected light. The LiDAR 31 may also calculate the position of the reflection point based on information on the intensity of the reflected light. Alternatively, the LiDAR 31 may be a frequency-modulated continuous wave (FMCW) LiDAR that irradiates laser light whose frequency is linearly changed and calculates the position of the reflection point in three-dimensional space based on data on the direction in which the reflected light is received and data on the phase difference between the frequency of the irradiated laser light and the frequency of the reflected light.

[0024] The LiDAR 31 may be a so-called scanning LiDAR that scans a plurality of laser beams arranged in a line along the vertical or horizontal direction in the horizontal or vertical direction. Alternatively, the LiDAR 31 may be a LiDAR that irradiates a wide area with laser beams, captures the light reflected by an object using a three-dimensional distance image sensor, and analyzes the positions of the reflection points in three-dimensional space to generate point cloud data of the reflection points. The LiDAR 31 is communicably connected to the processing device 50 via wired or wireless communication means and transmits the generated point cloud data to the processing device 50.

[0025] The point cloud data generated by the LiDAR 31 may be, for example, data on the coordinate positions of each reflection point in a three-dimensional coordinate system (also referred to as the "LiDAR coordinate system") with three orthogonal axes and with the LiDAR 31 itself as the origin. When the LiDAR 31 measures the area ahead of the vehicle 1, the LiDAR 31 may be installed so that the three axes of the LiDAR coordinate system are aligned with the longitudinal, transverse, and height directions of the vehicle 1, but they may be different. The coordinate positions of the reflection points in the point cloud data generated by the LiDAR 31 are converted by the processing device 50 into coordinate positions in a three-dimensional coordinate system (also referred to as the "vehicle coordinate system") with three orthogonal axes and with the origin at a predetermined position of the vehicle 1 and aligned with the longitudinal, transverse, and height directions of the vehicle 1.

[0026] In general, the LiDAR 31 has a fixed total energy that can be irradiated onto a unit virtual plane in space where the laser light is irradiated. Therefore, the LiDAR 31 has a characteristic that the spatial resolution decreases proportionally depending on the distance from the light-emitting surface that emits the laser light. On the other hand, the LiDAR 31 can extend the measurement distance by limiting the irradiation range where the laser light is irradiated, or can increase the spatial resolution by reducing the frame rate, which is the number of processing times per unit time.

[0027] The distance measurement sensor is not limited to the LiDAR 31, but may be a radar sensor such as a millimeter wave radar.

[0028] The camera 33 is an imaging device equipped with an imaging element such as a CCD (Charged-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor). In this embodiment, the vehicle 1 is equipped with a pair of left and right stereo cameras 33LF, 33RF that capture images of the front. The stereo cameras 33LF, 33RF are communicably connected to the processing device 50 via wired or wireless communication means and transmit the generated image data to the processing device 50.

[0029] 1, the camera 33 is configured as a pair of left and right stereo cameras 33LF, 33RF, but may also be a monocular camera consisting of a single imaging camera. In addition to the front-facing camera, the vehicle 1 may also be equipped with a camera that captures images behind the vehicle 1, or a camera mounted on a side mirror or the like that captures images of the left or right rear.

[0030] The shooting direction and angle of view indicating the shooting range of the camera 33 are defined, for example, by a three-dimensional coordinate system (also referred to as a "camera coordinate system") with three orthogonal axes having the camera 33 itself as the origin. When the camera 33 is a stereo camera 33LF, 33RF, the three-dimensional coordinate system may have the center point of the pair of stereo cameras 33LF, 33RF as the origin. When the camera 33 measures the area ahead of the vehicle 1, the camera 33 may be installed so that the three axes of the camera coordinate system are aligned with the longitudinal direction, vehicle width direction, and height direction of the vehicle 1, but may be different. Information on the shooting range of the image data generated by the camera 33 is converted into information in the vehicle coordinate system by the processing device 50.

[0031] The camera 33 generally has a fixed number of frames per second (fps). The camera 33 has a characteristic that the spatial resolution of the generated image data is low because it cannot focus in areas closer than the focal length. Furthermore, the camera 33 has a characteristic that the spatial resolution in a virtual plane of space peaks at the focal length and decreases at areas farther than the focal length depending on the resolution of the image sensor. Furthermore, in the case of the stereo cameras 33LF and 33RF, a minimum distance at which the image data generated by the left and right cameras can be matched (stereo matching) is separately defined. The focal length of the camera 33 or the minimum distance (first distance L1) described above is the shortest distance at which an object can be recognized by object recognition processing based on image data.

[0032] FIG. 2 is a diagram illustrating the spatial resolution of the LiDAR 31 and the camera 33. In the illustrated example, the LiDAR 31 and the camera 33 are installed at the upper portion of the front windshield of the vehicle 1 on the passenger compartment side. In the region (hereinafter also referred to as the "short distance region") from the installation position (base point) L0 of the camera 33 to a first distance L1, which is defined as, for example, the focal length of the camera 33 or the minimum distance at which stereo matching is possible, the spatial resolution Re_C of the image data generated by the camera 33 is low. As shown in FIG. 3, when the camera 33 is a stereo camera 33LF, 33RF, the first distance L1 is defined as the minimum distance at which stereo matching is possible between the image data generated by the stereo cameras 33LF, 33RF. The spatial resolution Re_C of the image data generated by the camera 33 peaks at the first distance L1 and decreases proportionally as the distance from the first distance L1 increases.

[0033] On the other hand, the spatial resolution Re_Li of the LiDAR 31 decreases proportionally as the distance from the installation position (base point) L0 of the LiDAR 31 increases. In the illustrated example, in the short-distance region (third region) where the distance from the installation position (base point) L0 of the LiDAR 31 and the camera 33 is within a first distance L1, the spatial resolution Re_Li of the LiDAR 31 is higher than the spatial resolution Re_C of the camera 33. Furthermore, at a second distance L2 that is farther than the first distance L1, the spatial resolution Re_Li of the LiDAR 31 and the spatial resolution Re_C of the camera 33 intersect. In other words, in the region where the distance from the installation position (base point) L0 of the LiDAR 31 and the camera 33 exceeds the first distance L1 and extends to the second distance L2 (hereinafter also referred to as the "medium-distance region"), the spatial resolution Re_C of the camera 33 is higher than the spatial resolution Re_Li of the LiDAR 31. Furthermore, in the area where the distance from the installation position (base point) L0 of the LiDAR 31 and the camera 33 exceeds the second distance L2 (hereinafter also referred to as the ``long distance area''), the spatial resolution Re_Li of the LiDAR 31 is again higher than the spatial resolution Re_C of the camera 33.

[0034] In the present disclosure, the object recognition device is configured to perform highly accurate object recognition in each of the short-distance, medium-distance, and long-distance ranges based on the characteristics of the spatial resolutions Re_Li and Re_C of the LiDAR 31 and camera 33, respectively, shown in Figure 2.

[0035] In this embodiment, the first distance L1 is the boundary between the short distance region and the medium distance region, and the second distance L2 is the boundary between the medium distance region and the long distance region. However, the boundaries do not have to coincide with the first distance L1 or the second distance L2. In particular, the boundary between the medium distance region and the long distance region may be set based on the accuracy of distance measurement by parallax detection of the stereo cameras 33LF and 33RF, or a comparison between the minimum detection size at an arbitrary distance based on the minimum scanning angle between the irradiation points of the laser light of the LiDAR 31 and the detection size equivalent to one pixel at an arbitrary distance of the camera 33. Furthermore, the boundaries of the respective regions may be gradated (gradually changing) or may overlap.

[0036] The processing device 50 functions as a device that recognizes objects by having one or more processors, such as a central processing unit (CPU) or a graphics control unit (GPU), execute a computer program. The computer program is a computer program that causes the processor to execute the operations to be performed by the processing device 50, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) provided in the processing device 50, or may be recorded on a recording medium built into the processing device 50 or any recording medium that can be externally attached to the processing device 50.

[0037] The recording medium for recording a computer program may be a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape; an optical recording medium such as a CD-ROM, a DVD, or a Blu-ray (registered trademark); a magneto-optical medium such as a floptical disk; a memory element such as a RAM or a ROM; a flash memory such as a USB memory or an SSD; or any other medium capable of storing a program.

[0038] The processing device 50 is connected to the LiDAR 31, the camera 33, the vehicle control unit 20, and the notification device 40 via a dedicated line or via communication means such as a controller area network (CAN) or local internet (LIN). The notification device 40 notifies the occupants of various information by means of image display, audio output, or the like based on a drive signal generated by the processing device 50. The notification device 40 includes, for example, a display device provided in the instrument panel and a speaker provided in the vehicle 1. The display device may be a display device of a navigation system or a head-up display (HUD) that displays information on the windshield.

[0039] 2. Processing Device Next, the processing device 50 of the object recognition device 30 according to this embodiment will be described in detail.

[0040] (2-1. Configuration Example) FIG. 4 is a block diagram showing a configuration example of the processing device 50. The processing device 50 includes a processing unit 51 and a storage unit 53. The processing unit 51 is configured to include one or more processors. A part or all of the processing unit 51 may be configured as an updatable component such as firmware, or may be a program module executed by commands from a CPU or the like. The processing device 50 may be configured as a single device, or may be configured as multiple devices connected to each other so that they can communicate with each other.

[0041] The storage unit 53 is configured with one or more storage elements (memories), such as RAM (Random Access Memory) or ROM (Read Only Memory), communicably connected to the processing unit 51. However, the number and type of storage units 53 are not particularly limited. The storage unit 53 stores computer programs executed by the processing unit 51, various parameters used in arithmetic processing, detection data, calculation results, and other data. A part of the storage unit 53 functions as a work area for the processing unit 51.

[0042] In addition, the processing device 50 is equipped with one or more communication interfaces (not shown) for sending and receiving data between the LiDAR 31, the camera 33, the vehicle control unit 20, and the notification device 40.

[0043] (2-2. Configuration of Processing Unit) The processing unit 51 of the processing device 50 performs object recognition processing based on the point cloud data transmitted from the LiDAR 31 and the image data transmitted from the camera 33. In the technology disclosed herein, the processing unit 51 performs different object recognition processing for each of the short-distance area, the medium-distance area, and the long-distance area according to the distance from the installation position (base point) L0 of the LiDAR 31 and the camera 33.

[0044] 4 , the processing unit 51 of the processing device 50 includes an acquisition unit 61, a ranging sensor drive control unit 62, a point cloud data processing unit 63, an imaging device drive control unit 64, an image data processing unit 65, an object recognition processing unit 67, and a response control unit 69. The acquisition unit 61, the ranging sensor drive control unit 62, the point cloud data processing unit 63, the imaging device drive control unit 64, the image data processing unit 65, the object recognition processing unit 67, and the response control unit 69 are functions realized by the execution of a computer program by one or more processors. Note that some or all of the acquisition unit 61, the ranging sensor drive control unit 62, the point cloud data processing unit 63, the imaging device drive control unit 64, the image data processing unit 65, the object recognition processing unit 67, and the response control unit 69 may be configured by hardware such as an analog circuit.

[0045] (2-2-1. Acquisition Unit) The acquisition unit 61 acquires point cloud data transmitted at a predetermined cycle from the LiDAR 31 and image data transmitted at a predetermined cycle from the camera 33. The point cloud data of the LiDAR 31 includes information on the coordinate positions of each reflection point on the LiDAR coordinate system. The image data of the camera 33 is image data of the shooting range generated by the image sensor.

[0046] (2-2-2. Distance measurement sensor drive control unit) The distance measurement sensor drive control unit 62 controls the drive of the distance measurement sensor. In this embodiment, the distance measurement sensor drive control unit 62 controls the irradiation of laser light by the LiDAR 31. Specifically, the distance measurement sensor drive control unit 62 controls the irradiation energy and irradiation position of the laser light irradiated from the LiDAR 31. The irradiation energy and irradiation position of the laser light can change at predetermined time intervals or randomly.

[0047] In this embodiment, the distance measurement sensor drive control unit 62 sets the minimum scanning angle and irradiation energy between irradiation points of the laser light so that the minimum detection target size between irradiation points on a virtual unit plane in each distance range of the short distance area, medium distance area, and long distance area is less than a predetermined value.

[0048] (2-2-3. Point Cloud Data Processing Unit) The point cloud data processing unit 63 performs predetermined data processing based on the point cloud data acquired from the LiDAR 31. In this embodiment, the point cloud data processing unit 63 calculates the distance from a predetermined base point to each reflection point included in the acquired point cloud data. The point cloud data processing unit 63 also extracts clusters, which are groups of reflection points whose inter-reflection point distances are within a predetermined distance (clustering). Furthermore, the point cloud data processing unit 63 calculates a motion vector of the center of a cluster between frames in a three-dimensional map (hereinafter also referred to as a "frame") containing clusters extracted from each point cloud data acquired in time series. The center of a cluster is, for example, the coordinate position at which the sum of the distances from the reflection points constituting the extracted cluster is minimum, but the method for calculating the center of the cluster is not particularly limited. The motion vector indicates the moving speed and moving direction of the objects constituting the cluster.

[0049] 5 shows a flowchart of a routine for processing point cloud data by the point cloud data processing unit 63. The point cloud data processing unit 63 acquires point cloud data at time t_n transmitted from the LiDAR 31 (step S11). Next, the point cloud data processing unit 63 calculates the distance to each reflection point included in the point cloud data (step S13). For example, for each reflection point, the point cloud data processing unit 63 converts the coordinate position in the LiDAR coordinate system to the coordinate position in the vehicle coordinate system and calculates the distance from the origin of the vehicle coordinate system to each reflection point.

[0050] The vehicle coordinate system is a three-dimensional coordinate system with three orthogonal axes, with the origin being the base point (L0) of distance measurement in the object recognition process, and the three axes being the longitudinal, transverse, and height directions of the vehicle 1. In this embodiment, an example will be described in which the center point in the longitudinal direction of the vehicle 1 at the installation positions of the LiDAR 31 and the camera 33 is set as the base point (L0) and the distance to each reflection point is calculated. However, the position of the base point may be set at any position, such as the front part of the vehicle 1.

[0051] The method for calculating the distance to each reflection point is determined by an appropriate method depending on the type and specifications of the LiDAR 31.

[0052] Next, the point cloud data processing unit 63 extracts clusters, which are groups of reflection points whose distances between them are within a predetermined distance (clustering) (step S15). For example, the point cloud data processing unit 63 extracts clusters by grouping reflection points whose distances between them are equal to or less than a preset processing threshold into the same group. Through the clustering process, the point cloud data processing unit 63 generates data of a frame (three-dimensional map) including data of the clusters in three-dimensional space.

[0053] The distance between the reflection points may be, for example, Euclidean distance, but may be any other distance. The clustering process is not limited to the above example, and any method may be used.

[0054] Next, the point cloud data processing unit 63 determines whether the number of generated frames is equal to or greater than a predetermined threshold value N (step S17). The predetermined threshold value N is set in advance to an arbitrary value equal to or greater than 2 as the number of time-series frames required to calculate the movement vector (movement speed and movement direction) of the object indicated by each cluster.

[0055] If the point cloud data processing unit 63 does not determine that the number of frames is greater than or equal to the predetermined threshold value N (S17 / No), it returns to step S11 and repeats the process of extracting clusters from the point cloud data at time t_n+1 and generating frames.

[0056] On the other hand, if the point cloud data processing unit 63 determines that the number of frames is equal to or greater than the predetermined threshold value N (S17 / Yes), it calculates the motion vector of the center of the cluster of reflection points caused by the same detection target included in each frame (step S19). For example, the point cloud data processing unit 63 calculates the center of each cluster included in each frame based on the coordinate positions of the multiple reflection points that make up the cluster. Furthermore, the point cloud data processing unit 63 identifies clusters of reflection points caused by the same detection target from among the clusters included in each frame, based on the position and shape of each cluster included in the time-series frames or information on the motion vector of the cluster identified up to the previous calculation cycle.

[0057] The point cloud data processing unit 63 then calculates a movement vector on the vehicle coordinate system of the center of the cluster of reflection points caused by the same detection target. The direction of the movement vector indicates the movement direction of the detection target. The magnitude of the movement vector indicates the distance moved by the detection target in the time corresponding to the difference in the times at which the reflection points of the clusters included in the multiple frames were acquired, i.e., the movement speed. The point cloud data processing unit 63 records the data on the position, movement direction, and movement speed of the cluster obtained by the above point cloud data processing in the memory unit 53.

[0058] Next, the point cloud data processing unit 63 determines whether each cluster may be an obstacle to the vehicle 1 (step S21). For example, if the movement direction of the cluster intersects with the planned driving route of the vehicle 1, the point cloud data processing unit 63 determines that the cluster may be an obstacle to the vehicle 1. Furthermore, if the cluster is located within the lane in which the vehicle 1 is traveling, the point cloud data processing unit 63 determines that the cluster may be an obstacle to the vehicle 1. The planned driving route or lane of the vehicle 1 can be determined based on information about lane markings detected by the image data processing unit 65, for example.

[0059] Next, the point cloud data processing unit 63 records information about the clusters determined to be potential obstacles to the vehicle 1 in the data recorded in the storage unit 53 (step S23). As a result, the data of the clusters extracted from the point cloud data of the LiDAR 31 is recorded in the storage unit 53 together with information about whether or not the clusters are potential obstacles to the vehicle 1.

[0060] (2-2-4. Imaging Device Drive Control Unit) The imaging device drive control unit 64 controls the driving of the camera 33. In this embodiment, the imaging device drive control unit 64 generates image data of the imaging range photographed by the camera 33 at predetermined time intervals.

[0061] (2-2-5. Image Data Processing Unit) The image data processing unit 65 performs predetermined data processing based on the image data acquired from the camera 33. In this embodiment, the image data processing unit 65 detects lane markings such as white lines based on the acquired image data.

[0062] 6 shows a flowchart of an image data processing routine performed by the image data processing unit 65. The image data processing unit 65 acquires image data at time t_n transmitted from the camera 33 (step S31). Next, the image data processing unit 65 detects lane markings based on the image data (step S33). For example, the image data processing unit 65 detects lane markings by performing a process (edge ​​detection process) to detect edges in the image data where the amount of change in brightness exceeds a predetermined threshold, and a process (feature point matching process) to identify lane markings based on the edge pattern. However, the method for detecting lane markings based on image data is not particularly limited.

[0063] The image data processing unit 65 also determines the relative position of the lane markings with respect to the vehicle 1. If the cameras 33 are stereo cameras 33LF, 33RF, the image data processing unit 65 determines the position of the lane markings in the vehicle coordinate system based on parallax information of the image data generated by the left and right stereo cameras 33LF, 33RF. If the camera 33 is a monocular camera, the image data processing unit 65 determines the position of the lane markings in the vehicle coordinate system based on changes in the lane markings in multiple image data acquired in time series.

[0064] Next, the image data processing unit 65 records the detected lane marking data in the storage unit 53 (step S35).

[0065] (2-2-6. Object Recognition Processing Unit) The object recognition processing unit 67 performs object recognition processing for each of the short-distance, medium-distance, and long-distance regions using different methods. Each of the object recognition processing operations will be described below, divided into a first object recognition processing operation performed for the short-distance region, a second object recognition processing operation performed for the medium-distance region, and a third object recognition processing operation performed for the long-distance region.

[0066] (First object recognition process) In the first object recognition process for the short distance region, the object recognition processing unit 67 executes a process of recognizing an object using only the point cloud data acquired from the LiDAR 31. Specifically, the object recognition processing unit 67 recognizes an object based on a cluster in the short distance region (short distance cluster) among the clusters extracted by the point cloud data processing unit 63.

[0067] The short-distance region is a region in which the spatial resolution of the LiDAR 31 is higher than the spatial resolution of the camera 33 (see FIG. 2). In the short-distance region, it is not possible to obtain image data that can recognize an object through object recognition processing. Furthermore, in the case of the stereo cameras 33LF and 33RF, it is not possible to measure distances based on image data in regions where stereo matching is not possible. Furthermore, the short-distance region is a region close to the vehicle 1, and is a region in which the presence of an object poses a high level of urgency, such as collision avoidance. For this reason, in the short-distance region, the object recognition processing unit 67 recognizes objects using only the point cloud data of the LiDAR 31, which has a high spatial resolution.

[0068] The object recognition processing unit 67 may estimate the type of object and the distance to the object in a short distance area where the distance from the vehicle 1 is short, but may not estimate the object's moving speed, size, etc. This reduces the load or time required for the object recognition processing in the short distance area, and allows the object recognition processing unit 67 to quickly detect objects.

[0069] 7 is a flowchart showing the first object recognition process performed by the object recognition processing unit 67. The object recognition processing unit 67 identifies short-distance clusters that exist in the short-distance region from among the clusters recorded by the point cloud data processing unit 63 in step S23 described above as clusters that may pose obstacles to the vehicle 1 (step S31).

[0070] For example, the object recognition processing unit 67 identifies as a short-distance cluster a cluster in which the distance from the base point L0 of the vehicle coordinate system to the center of the cluster is less than a first distance L1, which is set as the shortest distance at which an object can be recognized by object recognition processing based on image data. The object recognition processing unit 67 may identify as a short-distance cluster a cluster in which the distance from the base point L0 of the vehicle coordinate system to the reflection point that is the smallest among the reflection points that make up the cluster is less than the first distance L1. Alternatively, the object recognition processing unit 67 may identify as a short-distance cluster a cluster in which the distance from the base point L0 of the vehicle coordinate system to the reflection point that is the greatest among the reflection points that make up the cluster is less than the first distance L1.

[0071] Next, the object recognition processing unit 67 recognizes the object to be detected based on the identified short-distance clusters (step S33). For example, the object recognition processing unit 67 performs pattern matching processing using the short-distance clusters to identify the type of the object to be detected.

[0072] Next, the object recognition processing unit 67 records information on the object recognition results in the storage unit 53 (step S45). For example, the object recognition processing unit 67 records information on the type of recognized object, the object's location (direction), the distance to the object, and the object's movement direction and movement speed for each short distance cluster. The object's location may be, for example, the direction in which the center of the corresponding short distance cluster is located relative to the origin (base point L0) of the vehicle coordinate system. The distance to the object may be the distance to the reflection point that is the shortest distance from the origin of the vehicle coordinate system among the reflection points that make up the short distance cluster. The object's movement direction and movement speed may be movement vector data calculated by the point cloud data processing unit 63 in step S19 described above.

[0073] (Second Object Recognition Process) In the second object recognition process for the mid-distance region, the object recognition processing unit 67 performs the object recognition process using the point cloud data of the LiDAR 31 and the image data of the camera 33. In this embodiment, the object recognition processing unit 67 sets a determination region in the image data that includes clusters in the mid-distance region (mid-distance clusters) from the clusters extracted by the point cloud data processing unit 63, and performs edge detection processing and feature point matching processing based on the image data of the determination region to recognize the object.

[0074] The medium-distance region is a region where the spatial resolution of both the LiDAR 31 and the camera 33 is relatively high (see FIG. 2 ). However, because the spatial resolution of the camera 33 is higher than the spatial resolution of the LiDAR 31, the object recognition processing unit 67 performs object recognition processing using image data. Here, performing object recognition processing using image data for the entire image data of the shooting range increases the computational load on the processor and lengthens the processing time. Because the medium-distance region is a region closer to the vehicle 1 than the long-distance region among the regions in which objects can be recognized by the camera 33, it is desirable that the time required for object recognition processing be relatively short.

[0075] For this reason, in this embodiment, the object recognition processing unit 67 primarily identifies the detection target present in the mid-distance area based on the point cloud data of the LiDAR 31, narrows down the area (determination area) where the detection target exists, and performs object recognition processing using the image data. As a result, the object recognition processing unit 67 performs object recognition processing in the mid-distance area with high accuracy, while reducing the load or time required for the object recognition processing.

[0076] 8 is a flowchart showing the second object recognition process by the object recognition processing unit 67. The object recognition processing unit 67 identifies (step S51) medium-distance clusters that exist in the medium-distance region from among the clusters recorded by the point cloud data processing unit 63 in step S23 described above as clusters that may pose obstacles to the vehicle 1. The identification of medium-distance clusters may be performed in accordance with the method for identifying short-distance clusters described in step S31 described above.

[0077] Next, the object recognition processing unit 67 sets a predetermined area including the coordinate positions of the mid-distance clusters in the image data as the judgment area (step S53). For example, the object recognition processing unit 67 sets the smallest rectangular area that includes all of the reflection points included in each mid-distance cluster as the judgment area. The set rectangular area may be a vertical and horizontal area that is parallel to two sides of the image data generated by the camera 33. Alternatively, the object recognition processing unit 67 may set the judgment area by adding a preset margin to the smallest rectangular area that includes all of the reflection points included in each mid-distance cluster. The object recognition processing unit 67 trims the image data to fit the judgment area set for each mid-distance cluster.

[0078] The judgment area that is set is not limited to a rectangular area that is set to include the group of reflection points of the mid-distance cluster, but may also be a circular or elliptical area, or an area of ​​any other appropriate shape.

[0079] Next, the object recognition processing unit 67 performs object recognition processing using the image data of the trimmed determination areas (step S55). For example, the object recognition processing unit 67 performs edge detection processing and feature point matching processing on the image data of each determination area to identify the type of object to be detected. The object recognition processing unit 67 also determines the distance to the object to be detected and the object's location (direction) based on the image data of the determination areas. If the camera 33 is a stereo camera 33LF, 33RF, the object recognition processing unit 67 determines the distance to the object based on parallax information of the image data generated by the left and right stereo cameras 33LF, 33RF. If the camera 33 is a monocular camera, the object recognition processing unit 67 determines the distance to the object based on changes in the same detected object in multiple image data acquired in time series. The object recognition processing unit 67 determines the object's location (direction) based on the location (range) of the detected object in the image data.

[0080] The object recognition processing unit 67 also determines the moving direction and moving speed of the object to be detected based on changes in the distance to the object to be detected and the position (direction) of the object to be detected determined from each of the time-series image data. The object recognition processing unit 67 determines the relative speed and moving direction of the object to be detected with respect to the vehicle 1 from changes in the distance and position of the object to be detected in the vehicle coordinate system, and calculates the moving speed and moving direction of the object to be detected based on the relative speed and relative moving direction of the object and the speed and moving direction of the vehicle 1.

[0081] Next, the object recognition processing unit 67 records information on the object recognition result in the storage unit 53 (step S57). For example, the object recognition processing unit 67 records information on the type of recognized object, the position (direction) of the object, the distance to the object, and the moving direction and moving speed of the object for each detection target corresponding to the medium-distance cluster.

[0082] (Third object recognition processing) In the third object recognition processing for the long-distance region, the object recognition processing unit 67 performs the object recognition processing using the point cloud data of the LiDAR 31 and the image data of the camera 33, as in the second object recognition processing for the medium-distance region. However, in the long-distance region, the spatial resolution of the LiDAR 31 and the spatial resolution of the camera 33 are lower than in the medium-distance region, and the spatial resolution of the camera 33 is lower than the spatial resolution of the LiDAR 31 (see FIG. 2 ).

[0083] Therefore, in the long-distance region, super-resolution processing is performed on the image data to suppress a decrease in the accuracy of the object recognition processing based on the image data. Since the long-distance region is the region farthest from the vehicle 1 among the regions in which the object recognition processing is performed, an increase in the time spent on the object recognition processing is permitted. This allows the object recognition processing unit 67 to perform the object recognition processing in the long-distance region with high accuracy.

[0084] 9 is a flowchart showing the third object recognition process by the object recognition processing unit 67. The object recognition processing unit 67 identifies long-distance clusters that exist in the long-distance region from among the clusters recorded by the point cloud data processing unit 63 in the above-mentioned step S23 as clusters that may pose an obstacle to the vehicle 1 (step S61). Identification of long-distance clusters may be performed in accordance with the method for identifying short-distance clusters described in the above-mentioned step S31.

[0085] Next, the object recognition processing unit 67 sets a predetermined area including the coordinate positions of the long-distance clusters in the image data as a judgment area (step S63). The object recognition processing unit 67 sets the judgment area in the same procedure as step S53 of the second object recognition processing, and trims the image data to fit the judgment area.

[0086] Note that the process of identifying long-distance clusters for the long-distance region (step S61) and setting the judgment region (step S63) may be performed in the same process as the process of identifying medium-distance clusters for the medium-distance region (step S51) and setting the judgment region (step S53). In other words, the object recognition processing unit 67 may identify clusters that exist in the medium-distance region and the long-distance region, and then identify clusters that exist in the long-distance region as long-distance clusters.

[0087] Next, the object recognition processing unit 67 performs super-resolution processing on the image data of the trimmed determination area (step S65). If the camera 33 is a stereo camera 33LF, 33RF, the object recognition processing unit 67 aligns the image data of the determination area generated by the stereo camera 33LF, 33RF and restores it as a high-resolution image. If the camera 33 is a monocular camera, the object recognition processing unit 67 restores it as a high-resolution image by, for example, emphasizing edges by reconstructing the pixel values ​​of the image data of the determination area.

[0088] Next, the object recognition processing unit 67 performs object recognition processing using the high-resolution image data that has been subjected to the super-resolution processing (step S67). The object recognition processing unit 67 performs object recognition processing using the same procedure as step S55 of the second object recognition processing. Next, the object recognition processing unit 67 records information on the object recognition results in the storage unit 53 (step S69). For example, the object recognition processing unit 67 records information on the type of recognized object, the object's location (direction), the distance to the object, and the object's moving direction and moving speed for each detection target corresponding to the long-distance cluster.

[0089] That is, in the long-distance region, the object recognition processing unit 67 performs super-resolution processing on the image data of the camera 33, and then executes object recognition processing based on the image data. This makes it possible to improve the accuracy of the object recognition processing even in regions where the spatial resolution of the LiDAR 31 and the spatial resolution of the camera 33 are reduced.

[0090] In addition, in the object recognition processing for each of the short-distance area, the medium-distance area, and the long-distance area, the object recognition processing unit 67 may perform a labeling process (a process of associating object information with recognition information) for an object that has been recognized once, and thereafter perform a tracing process using the LiDAR 31 or the camera 33, thereby omitting the process of estimating the type, size, etc. of the object. This makes it possible to reduce the load of calculation processing on the processing device 50.

[0091] (2-2-7. Response Control Unit) The response control unit 69 executes predetermined control to respond to the recognized object based on the result of the object recognition processing by the object recognition processing unit 67. For example, the response control unit 69 transmits information on the object recognition result to the vehicle control unit 20 in order to avoid contact with or approaching the recognized object. The information on the object recognition result includes any one or more pieces of information on the type, position, movement speed, and movement direction of the object recorded in the storage unit 53. The vehicle control unit 20 executes emergency brake control or automatic steering control in order to avoid contact with or approaching the object.

[0092] Alternatively, the response control unit 69 may drive the notification device 40 to notify the driver of the presence of the object in order to avoid contact with or approaching the recognized object. For example, the response control unit 69 may notify the driver of the type or location of the object, or advice on driving operations to avoid contact with or approaching the object, by means of either or both of sound and display.

[0093] 3. Object Recognition Processing Method Next, an object recognition processing method performed by the processing device 50 of the object recognition device 30 according to this embodiment will be described.

[0094] 10 shows a flowchart of an object recognition processing method. The flowchart described below may be executed continuously while the system of the vehicle 1 is running, or may be executed while the object recognition system is running.

[0095] When the processing unit 51 of the processing device 50 detects system startup (step S71), the point cloud data processing unit 63 executes the point cloud data processing illustrated in Fig. 5 based on the point cloud data measured by the LiDAR 31 (step S73). Also, the image data processing unit 65 executes the image data processing illustrated in Fig. 6 based on the image data generated by the camera 33 (step S75).

[0096] Next, the object recognition processing unit 67 executes the first object recognition processing illustrated in Fig. 7, the second object recognition processing illustrated in Fig. 8, and the third object recognition processing illustrated in Fig. 9 for each of the short-distance region, the medium-distance region, and the long-distance region (step S79). As described above, the short-distance region in which the first object recognition processing is executed is a region in which the accuracy of object recognition or distance measurement using image data from the camera 33 is low. For this reason, the object recognition processing for the short-distance region is executed based on the point cloud data of the LiDAR 31, which has high spatial resolution.

[0097] Furthermore, the mid-distance region where the second object recognition process is executed is a region where the spatial resolution of both the LiDAR 31 and the camera 33 is high. For this reason, for the mid-distance region, a process is executed in which clusters that may be obstacles to the vehicle 1 (mid-distance clusters) are primarily detected based on the point cloud data of the LiDAR 31, and the object is analyzed based on image data of a determination region including the mid-distance clusters, which is set in the image data of the camera 33.

[0098] Furthermore, the long-distance region in which the third object recognition process is performed is a region in which the spatial resolution of both the LiDAR 31 and the camera 33 is reduced. For this reason, for the long-distance region, a process is performed in which clusters (long-distance clusters) that may be obstacles to the vehicle 1 are primarily detected based on the point cloud data of the LiDAR 31, and the objects are analyzed based on image data of a determination region including the long-distance clusters, which is set in the image data of the camera 33. At this time, in the third object recognition process, a process is performed in which the resolution of the image data is increased by applying super-resolution processing to the image data.

[0099] Therefore, at least in the medium-distance area and the long-distance area, object recognition processing is performed that takes advantage of the characteristics of the LiDAR 31 and the camera 33, respectively, and the accuracy of the object recognition processing results can be improved. Also, in the short-distance area, object recognition processing is performed based on the point cloud data of the LiDAR 31, and object recognition in areas where the object recognition accuracy by the camera 33 is reduced can be supplemented.

[0100] Next, based on the results of the object recognition process, the response control unit 69 executes either a notification process to avoid contact with or proximity to the object or a process to send information on the object recognition result to the vehicle control unit 20 (step S79).

[0101] Next, the processing unit 51 determines whether the system has stopped (step S81). If the processing unit 51 determines that the system has not stopped (S81 / No), the processing unit 51 returns to step S73 and repeats the object recognition process. On the other hand, if the processing unit 51 determines that the system has stopped (S81 / Yes), the processing unit 51 ends the process.

[0102] As described above, the object recognition device 30 according to this embodiment performs a predetermined first object recognition process for a short-distance region where the distance from the predetermined base point L0 is within a first distance L1, which is set as the shortest distance at which an object can be recognized by object recognition processing based on image data from the camera 33. The object recognition device 30 also performs a predetermined second object recognition process, different from the first object recognition process, for a medium-distance region where the distance from the predetermined base point L0 is greater than the first distance L1 and within a second distance L2, which is set as the upper limit of the distance at which an object can be recognized by object recognition processing based on image data. The object recognition device 30 also performs a predetermined third object recognition process, different from the first object recognition process and the second object recognition process, for a long-distance region where the distance from the predetermined base point L0 is greater than the second distance L2. This makes it possible to utilize the respective characteristics of the LiDAR 31 and the camera 33 to improve the object recognition accuracy in each region.

[0103] Specifically, in a medium-distance area where the spatial resolution of the camera 33 is higher than that of the LiDAR 31, the object recognition device 30 primarily identifies a medium-distance cluster of the detection target based on the point cloud data of the LiDAR 31, and complements the recognition data such as the distance to the object of the detection target, its type, and its moving speed based on image data of a determination area including the medium-distance cluster. This reduces the processing load on the image data while making use of the respective characteristics of the LIDAR 31 and the camera 33 to improve the object recognition accuracy.

[0104] Furthermore, in the long-distance region where the spatial resolution of the camera 33 is lower than that of the LiDAR 31, the object recognition device 30 according to this embodiment performs super-resolution processing on image data of a determination region including a long-distance cluster, and then performs object recognition processing based on the image data. As a result, even in the long-distance region, object recognition processing is performed based on image data with a resolution higher than the spatial resolution of the LiDAR 31, thereby improving object recognition accuracy. Note that, because the long-distance region is a region far from the vehicle 1, even when such super-resolution processing is performed, the risk of approaching or coming into contact with objects present in the long-distance region does not increase significantly.

[0105] Furthermore, in a short-distance area less than the first distance L1 where the object recognition accuracy of the object recognition process based on the image data of the camera 33 is low, the object recognition device 30 according to this embodiment executes the object recognition process based on the point cloud data of the LiDAR 31. Therefore, in an area where the object recognition accuracy of the camera 33 is low, distance recognition and the like are performed based on the point cloud data of the LiDAR 31, and it is possible to take measures such as a highly urgent notification operation or an avoidance operation.

[0106] 4. Other Embodiments Up to this point, an object recognition device according to an embodiment of the present disclosure has been described, but the object recognition device according to the above-described embodiment can be modified in various ways.

[0107] (4-1. Modification 1) In the object recognition device according to the above embodiment, the LiDAR 31 emits laser light so as to recognize detection target objects in all of the short-distance, medium-distance, and long-distance ranges. However, the technology of the present disclosure is not limited to the above example. For example, the ranging sensor drive control unit 62 may emit laser light during a period in which the minimum scanning angle and irradiation energy between the irradiation points of the laser light are set so that the minimum detection target size between the irradiation points on the virtual unit plane of the target distance range is less than a predetermined value, and during a period in which the minimum scanning angle and irradiation energy between the irradiation points of the laser light are set so that the minimum detection target size between the irradiation points on the virtual unit plane of the short-distance range is less than a predetermined value.

[0108] That is, the distance measurement sensor drive control unit 62 may cause the LiDAR 31 to emit laser light in a predetermined energy band capable of detecting the target distance range and laser light in an energy band other than the predetermined energy band, for separate periods of time. This makes it possible to maintain high object recognition accuracy in the short-distance range.

[0109] (4-2. Modification 2) The object recognition device may adjust the time for which the first object recognition process and the second object recognition process are each performed, or the proportion of the range in the measurement range in which the first object recognition process and the second object recognition process are each performed, based on information about the driving environment of the vehicle 1. Specifically, the ranging sensor drive control unit may determine whether or not there is a high need for object recognition in the short-distance area based on information about the driving environment of the vehicle 1, and if it determines that there is a high need for object recognition in the short-distance area, may increase the irradiation time or irradiation range of the laser light for performing object recognition processing for the short-distance area. This allows resources for object recognition processing by the LiDAR 31 to be concentrated and allocated to the short-distance area.

[0110] Fig. 11 shows a flowchart of an object recognition processing method by the processing device 50 of the object recognition device 30 according to Modification 2. The flowchart shown in Fig. 11 is obtained by adding a surrounding situation determination process (step S72) to the flowchart shown in Fig. 10.

[0111] When the processing unit 51 of the processing device 50 detects system startup (step S71), the ranging sensor drive control unit 62 executes a driving environment determination process to determine whether or not there is a high need for object recognition in the short-distance area based on information about the driving environment of the vehicle 1 (step S72). For example, the ranging sensor drive control unit 62 determines whether or not the vehicle 1 is placed in a driving environment in which it is estimated that there are many objects that could be obstacles to the vehicle 1 within a short distance from the vehicle 1. When the ranging sensor drive control unit 62 determines that there is a high need for object recognition in the short-distance area, it concentrates the allocation of resources for object recognition processing by the LiDAR 31 on the short-distance area.

[0112] 12 is a flowchart showing an example of the driving environment determination process. The distance measurement sensor drive control unit 62 acquires information on the vehicle speed of the vehicle 1 at time t_n (step S91). The vehicle speed information may be a sensor signal from a vehicle speed sensor, or may be acquired from another control device that has vehicle speed information.

[0113] Next, the ranging sensor drive control unit 62 acquires road type information at time t_n (step S93). The road type information indicates the type of road on which the vehicle 1 is traveling and is recorded, for example, in map data. The road type information may include, for example, one or more of the following information: a residential road, a shopping street, a main road, an urban expressway, an intercity expressway, a school route, the road width, the presence or absence of a sidewalk, the presence or absence of a guardrail, the presence or absence of a curb, or the traffic time zone (current time). Based on the road type information, the ranging sensor drive control unit 62 can determine whether the road on which the vehicle 1 is traveling is an environment with many pedestrians or bicycles, or whether it is a narrow road.

[0114] For example, the distance measurement sensor drive control unit 62 acquires information on the type of road recorded in the map data based on the traveling position of the vehicle 1 identified by a position detection sensor such as a GPS sensor and the map data. The distance measurement sensor drive control unit 62 may acquire information on the type of road on which the vehicle is traveling by communicating with another vehicle or an external system.

[0115] Next, the distance measurement sensor drive control unit 62 acquires information on the results of the object recognition process up to the previous calculation cycle (time t_n-1) (step S95). Specifically, the distance measurement sensor drive control unit 62 reads out information on the results of the object recognition process recorded in the storage unit 53.

[0116] Next, the distance measurement sensor drive control unit 62 determines whether the driving environment of the vehicle 1 is one that requires attention to the short-distance area (step S97). For example, the distance measurement sensor drive control unit 62 determines that the driving environment requires attention to the short-distance area when it determines that the vehicle 1 is driving in a shopping district with many pedestrians and bicycles, or when it determines that the vehicle 1 is driving on a school route during school arrival or departure times. However, the method for determining whether the driving environment of the vehicle 1 is one that requires attention to the short-distance area is not limited to the above example.

[0117] If the distance measurement sensor drive control unit 62 determines that the driving environment of the vehicle 1 is one that requires attention to the short-distance area (S97 / Yes), it records that the first range including the driver's line of sight is also applied to the measurement range of the short-distance area, and ends the driving environment determination process.On the other hand, if the distance measurement sensor drive control unit 62 does not determine that the driving environment of the vehicle 1 is one that requires attention to the short-distance area (S97 / No), it ends the driving environment determination process.

[0118] 11 , after executing the driving environment determination process, the processing unit 51 executes the processes of step S73 and subsequent steps in the flowchart shown in Fig. 10. At that time, if the processing unit 51 does not determine in step S72 that the driving environment of the vehicle 1 is one that requires attention to the short-distance area (S97 / No), it executes the processes of step S73 and subsequent steps in accordance with the procedure by the object recognition device 30 according to the embodiment described above.

[0119] On the other hand, if the processing unit 51 determines in step S72 that the driving environment of the vehicle 1 is one that requires attention in the short-distance area (S97 / Yes), it also concentrates the resources of the object recognition processing by the LiDAR 31 on the first range in the first object recognition processing for the short-distance area. Specifically, the ranging sensor drive control unit 62 increases the irradiation time or irradiation range of the laser light whose irradiation energy, irradiation density, etc. are adjusted to perform the object recognition processing for the short-distance area. This increases the irradiation density of the laser light for performing the object recognition processing for the short-distance area, or increases the irradiation energy of each laser light.

[0120] 7 is a cluster measured with high accuracy, and the object recognition process is performed based on the first cluster measured with high accuracy. Therefore, when the risk level is high in the short-distance area where the accuracy of the object recognition process using image data is low, the object recognition accuracy in the short-distance area can be increased, and the responsiveness of the notification process, the automatic driving control process, etc. can be improved.

[0121] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art to which the present disclosure pertains can conceive of various modifications or alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0122] For example, in the above embodiment, the short-distance region and the medium-distance region, and the medium-distance region and the long-distance region are clearly separated by boundaries, and predetermined processing is performed in each region. However, the technology disclosed herein is not limited to such an example. For example, when the distance to a detected object varies across the short-distance region and the medium-distance region, or the medium-distance region and the long-distance region, the regions may be gradually changed near the boundaries of the respective regions, or may be overlapped. This prevents a sudden change in the processing method for recognizing an object, which can cause the results of the object recognition processing to become unstable.

[0123] 1: Vehicle 20: Vehicle control unit 30: Object recognition device 33: Camera 33LF: Stereo camera 33RF: Stereo camera 40: Notification device 50: Processing device 51: Processing unit 53: Memory unit 61: Acquisition unit 62: Distance measurement sensor drive control unit 63: Point cloud data processing unit 64: Imaging device drive control unit 65: Image data processing unit 67: Object recognition processing unit 69: Response control unit L0: Base point L1: First distance L2: Second distance Re_C: Spatial resolution of camera Re_Li: Spatial resolution of LiDAR

Claims

1. a distance measuring sensor that measures at least the distance to a reflection point based on a reflected wave of the irradiated irradiation wave; a camera that generates image data of a photographing range; one or more processing devices that perform object recognition processing based on the measurement data of the distance measuring sensor and the image data of the camera; In an object recognition device comprising: The one or more processing devices perform the object recognition processing. a predetermined first object recognition process for a short distance area within a first distance from a predetermined base point, the first distance being set as the shortest distance at which an object can be recognized by the object recognition process based on the image data; a predetermined second object recognition process, different from the first object recognition process, for a middle distance region where the distance from the predetermined base point exceeds the first distance and is within a second distance set as an upper limit value of the distance at which an object can be recognized by the object recognition process based on the image data; a predetermined third object recognition process different from the first object recognition process and the second object recognition process for a long-distance region where the distance from the predetermined base point exceeds the second distance; In the first object recognition process, the object recognition process is performed using only the measurement data; In the second object recognition process, an object recognition process is performed using the measurement data and the image data; An object recognition device that adjusts the time for which the first object recognition process and the second object recognition process are each executed, or the proportion of the range in which the first object recognition process and the second object recognition process are each executed in the measurement range in which the object recognition process is executed, based on information about the running environment of a moving body.

2. The one or more processing devices: In the third object recognition process for the long-distance region, a super-resolution process is performed using the image data acquired in time series to perform object recognition process. The object recognition device according to claim 1 .

3. An object recognition processing method in which a computer performs object recognition processing based on measurement data from a distance measuring sensor that measures at least the distance to a reflection point based on the reflected wave of an irradiated irradiation wave, and image data from a camera that generates image data of a shooting range, The computer The object recognition process includes: a predetermined first object recognition process for a short distance area within a first distance from a predetermined base point, the first distance being set as the shortest distance at which an object can be recognized by the object recognition process based on the image data; a predetermined second object recognition process, different from the first object recognition process, for a middle distance region where the distance from the predetermined base point exceeds the first distance and is within a second distance set as an upper limit value of the distance at which an object can be recognized by the object recognition process based on the image data; a predetermined third object recognition process different from the first object recognition process and the second object recognition process for a long distance region where the distance from the predetermined base point exceeds the second distance; Run In the first object recognition process, the object recognition process is performed using only the measurement data; In the second object recognition process, an object recognition process is performed using the measurement data and the image data; An object recognition processing method that adjusts the time for executing the first object recognition processing and the second object recognition processing, respectively, or the proportion of the range in which the first object recognition processing and the second object recognition processing are executed in a measurement range in which the object recognition processing is executed, based on information about the running environment of a moving body.

4. A computer program that causes a computer to perform object recognition processing based on measurement data from a distance measuring sensor that measures at least a distance to a reflection point based on a reflected wave of an irradiated irradiation wave, and image data from a camera that generates image data of a shooting range, The computer, The object recognition process includes: a predetermined first object recognition process for a short distance area within a first distance from a predetermined base point, the first distance being set as the shortest distance at which an object can be recognized by the object recognition process based on the image data; a predetermined second object recognition process, different from the first object recognition process, for a middle distance region where the distance from the predetermined base point exceeds the first distance and is within a second distance set as an upper limit value of the distance at which an object can be recognized by the object recognition process based on the image data; a predetermined third object recognition process different from the first object recognition process and the second object recognition process for a long distance region where the distance from the predetermined base point exceeds the second distance; Execute In the first object recognition process, the object recognition process is performed using only the measurement data; In the second object recognition process, an object recognition process is performed using the measurement data and the image data; A non-transitory tangible recording medium having recorded thereon a computer program that adjusts the time for executing the first object recognition process and the second object recognition process, respectively, or the proportion of the range in which the first object recognition process and the second object recognition process are executed in the measurement range in which the object recognition process is executed, respectively, based on information about the running environment of a moving body.

Images