Object recognition apparatus

Abstract

To provide an object recognition device capable of reducing the possibility of erroneously determining the type or state of an object.SOLUTION: The object recognition device for recognizing an object existing in a predetermined area includes a communication part 4 capable of communicating with a communication device attached to a mobile body, a distance measuring sensor 2 for detecting an observation point indicating a position where the object exists, and a control part 1 for specifying a kind or a state of the object on the basis of the observation point. The control unit receives position information and size information of a target moving body which is a moving body corresponding to the communication device by using the communication unit, and executes area specifying processing for specifying a target existence area Ar which is an area where the target moving body may exist on the basis of the received position information and size information. The control unit is configured to estimate an observation point present in the target presence region as an observation point related to the target moving object, and specify a type or a state of the target moving object based on information on the observation point present in the target presence region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure herein relates to an object recognition device. [Background technology]

[0002] Patent Document 1 discloses a perimeter monitoring device that determines the type of an object based on an image, and then, for objects whose type has not yet been determined, determines the type based on radar detection data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-209387 Summary of the Invention [Problem to be solved by the invention]

[0004] With a distance measurement sensor, an observation point (for example, a reflection point) for a single object may be split into multiple points (i.e., observed as multiple objects). As a result, the type or state (for example, position) of the object may be misrecognized. To address this issue, a configuration could be considered in which the recognition results of the camera image are used to correct and handle the group of observation points, but camera images can also be misrecognized. For example, even with image recognition, a bus may be mistakenly recognized as a passenger car.

[0005] One disclosed object is to provide an object recognition device that can reduce the possibility of erroneously determining the type or state of an object. [Means for solving the problem]

[0006] The object recognition device disclosed herein is an object recognition device that performs processing to recognize an object present in a predetermined recognition target area, a communication unit (4) configured to be capable of wireless communication with a communication device attached to the mobile object; a ranging sensor (2) for detecting an observation point indicating the position where an object is present; a control unit (1) that identifies the type or state of an object based on an observation point; The control unit Using a communication unit, position information and size information of a target moving object that is a moving object corresponding to the communication device are acquired; Execute an area specification process to specify a target existence area (Ar) that is an area where a target moving object may exist, based on reception position information, which is position information, and reception size information, which is size information, acquired using the communication unit; Among the observation points, the observation points existing within the target existence area are estimated to be observation points related to the target moving object; The system is configured to identify the type or state of a target moving object based on information on observation points present in the target existence area.

[0007] According to this configuration, the target existence area where the target moving object may exist is estimated based on the position information and size information acquired through communication, and the observation points within the target existence area are determined to be observation points for a single object (i.e., the target moving object), and the target moving object is recognized. This makes it possible to properly identify the target moving object. Therefore, the possibility of misjudging the type or state of the target moving object (detected object) can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an overall view of an object recognition system. [Figure 2] 10 is a flowchart illustrating an example of processing by a control unit. [Figure 3] FIG. 10 is a diagram illustrating a method for identifying a target existence region. [Figure 4] FIG. 10 is a diagram illustrating a method for identifying a target existence region. [Figure 5] FIG. 1 illustrates reclustering. [Figure 6] FIG. 10 is a diagram illustrating a process for identifying a center position. [Figure 7] 10 is a flowchart illustrating an example of processing by a control unit. [Figure 8]FIG. 10 is a diagram illustrating a method for identifying a target existence region. [Figure 9] FIG. 1 illustrates reclustering. [Figure 10] FIG. 1 illustrates reclustering. DETAILED DESCRIPTION OF THE INVENTION

[0009] First Embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present disclosure will now be described with reference to the accompanying drawings. Fig. 1 is a diagram showing an example of a schematic configuration of an object recognition system including an object recognition device 10. As shown in Figs.

[0010] The object recognition device 10 is a device that executes a process of recognizing an object present in a predetermined recognition target area that is determined according to the position and orientation of the object recognition device 10. In this embodiment, the roadside device 100 is equipped with the function of the object recognition device 10. In the following, the roadside device 100 may be appropriately read as the object recognition device 10. The roadside device 100 is a wireless device used for vehicle-to-infrastructure (V2I) communication. The roadside device 100 may be placed on a road, near a road, or near an intersection. The roadside device 100 may be attached to a traffic light, a utility pole, a streetlight, or other structure.

[0011] The roadside device 100 including the object recognition device 10 recognizes road users such as vehicles and pedestrians present within a recognition target area and transmits the recognition results to surrounding vehicles. For example, the roadside device 100 checks whether there are pedestrians within the recognition target area and, if necessary, transmits a signal to surrounding vehicles (e.g., buses) to alert the driver. The surrounding vehicles here may be vehicles present in a location where road-to-vehicle communication with the roadside device is possible. Note that vehicles to which the roadside device 100 notifies a message may be limited to vehicles that are capable of wireless communication with the roadside device 100 and that are related to the warning content (e.g., vehicles that satisfy certain conditions).

[0012] <Object recognition device> 1, the object recognition device 10 includes a control unit 1, a distance measurement sensor 2, a camera 3, and a communication unit 4. Each of these will be explained in turn below.

[0013] <Range measuring sensor> The ranging sensor 2 is a sensor for measuring the distance to an object. The ranging sensor 2 detects an observation point that indicates the position where the object is located. The ranging sensor 2 may be a device that transmits electromagnetic waves of a predetermined frequency toward a recognition target area and detects the observation point by receiving reflected waves that are the electromagnetic waves reflected from the object.

[0014] In this embodiment, the ranging sensor 2 is a LiDAR. LiDAR is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging. The ranging sensor 2 may also be a millimeter wave radar. The ranging sensor 2 may also be a UWB (Ultra Wide Band) radar that uses impulse waves used in UWB communication. The ranging sensor 2 may also be a ToF (Time of Flight) camera.

[0015] LiDAR is a device that generates three-dimensional observation point cloud data indicating the positions of reflection points for each irradiation direction by emitting laser light. The reflection points correspond to the observation points. The irradiation range of the laser light defines the detection range of the LiDAR. The LiDAR as the ranging sensor 2 is provided on the roadside device 100 so that its detection range is, for example, a predetermined range in front of the roadside device 100. The LiDAR as the ranging sensor 2 may be installed at a predetermined position on the roadside device 100.

[0016] The LiDAR serving as the ranging sensor 2 transmits the detected observation point cloud to the control unit 1 as observation point cloud data. The observation point cloud data represents the three-dimensional position of each detected observation point. In other words, the observation point cloud data indicates the position information for each observation point. The position information of an observation point indirectly indicates the distance from the LiDAR to the observation point. The ranging sensor 2 emits laser light at a predetermined observation interval to generate observation point cloud data. The observation interval may be 100 milliseconds, 200 milliseconds, or the like.

[0017] The LiDAR used as the ranging sensor 2 may be either a scan type or a flash type. The scan type is a method of generating observation point cloud data by irradiating a laser beam in a sweeping manner. The flash type is a method of generating observation point cloud data by irradiating a predetermined sensing light in a diffuse manner and receiving the reflected light with a two-dimensional array of light-receiving sensors. Observation point cloud data is sometimes called three-dimensional point cloud data, range image, or 3D image.

[0018] The electromagnetic waves transmitted and received by the distance measuring sensor 2 may be in the range of 750 nm to 1100 nm, such as 903 nm or 905 nm. The electromagnetic waves may also be in the range of 1000 nm to 1600 nm, such as 1550 nm. The term "electromagnetic waves of a predetermined frequency" may be replaced with "electromagnetic waves (light) of a predetermined wavelength."

[0019] <Camera> The camera 3 is, for example, a visible light camera. The camera 3 may also be an infrared camera. The camera 3 may be included in other sensors, which will be described later.

[0020] In this embodiment, as an example, the camera 3 captures an image of a predetermined range in front of the roadside device 100. The camera 3 is set so that the recognition target area is included in the image capture range. The image capture range of the camera 3 is also set so that it includes the detection range of the distance measurement sensor 2. The image capture range of the camera 3 may be the same as the detection range of the distance measurement sensor 2, or may be larger than the detection range of the distance measurement sensor 2. The camera 3 is installed at a predetermined position of the roadside device.

[0021] An image of the surroundings captured by the camera 3 will be referred to as a captured image below. The camera 3 transmits captured image data to the control unit 1. Hereinafter, the captured image data may be appropriately interpreted as a captured image. The frame rate of the camera 3 may be 10 fps, 20 fps, 30 fps, or the like. The camera 3 may generate captured image data at a frame rate according to the image recognition processing capability of the processor 11 and output the data to the control unit 1. The camera 3 may output the captured image data to the control unit 1 in the form of a video signal. In this case, the control unit 1 may acquire the captured image data based on the video signal input from the camera 3. Since a captured image corresponds to a camera image, these terms may be appropriately interchanged.

[0022] <Other sensors> The object recognition device 10 may be equipped with sensors other than the distance measurement sensor 2 that detect surrounding objects. Sensors having an object detection function other than the distance measurement sensor 2 will hereinafter also be referred to as other sensors. Examples of other sensors include millimeter-wave radar, infrared sensors, ultrasonic sensors, and illuminance sensors. The other sensors transmit the results of their detection to the control unit 1. Note that the other sensors are optional elements. The object recognition device 10 does not necessarily need to be equipped with other sensors.

[0023] <Communications Department> The communication unit 4 is a wireless communication module configured to be able to perform short-range communication, which is direct wireless communication, with a communication device 21 attached to the mobile body 200. In this embodiment, the mobile body 200 is a vehicle. The communication device 21 may be a communication device (so-called in-vehicle device) mounted on the vehicle as the mobile body 200. The communication device 21 is linked to the mobile body 200. Note that the mobile body 200 may also be a person. In this case, a portable device (such as a smartphone) carried by the person corresponds to the communication device 21. A communication device attached to a mobile body may be interpreted as a communication device mounted on a vehicle as a mobile body, a communication device carried by a pedestrian or the like as a mobile body, or a communication device used in the mobile body.

[0024] The communication method (i.e., short-range communication method) between the communication unit 4 and the communication device 21 may be DSRC (Dedicated Short Range Communications) or Wi-Fi (registered trademark). DSRC is wireless communication that complies with standards such as IEEE802.11p, ARIB STD-T75, or CEN EN12253. The short-range communication method may also be cellular V2X, for example, communication using a PC5 interface. The communication unit 4 may include a circuit that is compatible with the communication method with the communication device 21. The communication unit 4 acquires information necessary for the processor 11 to execute the area identification process described below.

[0025] <Control Unit 1> The control unit 1 is hardware that identifies the type or state of an object based on the observation point acquired from the distance measurement sensor 2. The control unit 1 has a processor 11, RAM 12, storage 13, and I / O 14. The processor 11 is an arithmetic core that performs arithmetic processing based on data received from the distance measurement sensor 2 or other sensors. The processor 11 may be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). RAM is an abbreviation for Random Access Memory.

[0026] The storage 13 is a rewritable nonvolatile memory. The storage 13 may be realized by at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium. The storage 13 may include multiple types of storage media, such as a ROM (Read Only Memory) and a flash memory. The I / O 14 is an input / output circuit.

[0027] The processor 11 identifies the type or state of the object (hereinafter also referred to as the detected object) detected by the ranging sensor 2 based on the observation point cloud data acquired from the ranging sensor 2. The object type may be classified as a vehicle, pedestrian, bicycle, traffic sign, obstacle, building, wild animal, and others. The processor 11 identifies the type based on the characteristics of the detected object. Identifying the type may also be referred to as identifying. Furthermore, if the detected object is a vehicle, the processor 11 is configured to be able to identify the vehicle type as well. For example, the processor 11 may identify whether the detected object may be a passenger car, bus, motorcycle, truck, or the like. The processor 11 identifies the position, speed, orientation, or direction of travel of the detected object as the state of the detected object. The term "type" may be appropriately interpreted as "category."

[0028] Specifically, first, the processor 11 performs clustering on the observation point cloud data acquired from the distance measurement sensor 2. Clustering refers to grouping observation points whose inter-point distance is less than a predetermined value as observation points corresponding to the same object. Hereinafter, a cluster refers to a group (in other words, a collection) of observation points obtained by clustering. Through clustering, the observation point cloud is divided into clusters corresponding to detected objects. For convenience, a set of observation points that make up one cluster will also be referred to as a cluster point cloud.

[0029] The processor 11 may be configured to perform a type identification process and a state identification process. The type identification process is a process for identifying the type of object corresponding to a cluster based on the feature of the cluster. The feature is a state quantity that indicates the characteristics of the cluster point cloud that constitutes one object. A variety of feature quantities can be used as feature quantities for identifying the type of detected object. That is, the number of observation points that constitute the cluster point cloud, the width, height, depth, contour length, intensity mean, intensity variance, slice feature, relative position of the slice feature, local intensity mean, local intensity variance, circularity, linearity, inter-point angle distribution, curvature distribution, etc. can be used.

[0030] The state identification process is a process for identifying the state of an object. As described above, the state of an object may include its position, speed, orientation, or direction of travel. For example, the processor 11 may identify the position of an object based on the cluster point cloud. The processor 11 may perform a type identification process and a state identification process for each cluster.

[0031] The processor 11 acquires position information and size information of the mobile body 200 corresponding to the communication device 21 using the communication unit 4. Hereinafter, the mobile body 200 corresponding to the communication device 21 will also be referred to as a target mobile body. In other words, the target mobile body is the mobile body 200 equipped with the communication device 21 that communicated using the communication unit 4.

[0032] The position information of the target moving object is information indicating the current position of the target moving object. As an example, the target moving object is equipped with a GNSS receiver that receives positioning signals transmitted by positioning satellites that make up the GNSS (Global Navigation Satellite System). The target moving object calculates its position using the positioning signals received by the GNSS receiver. The communication device 21 of the target moving object periodically wirelessly transmits the position information of the target moving object. The communication unit 4 receives the position information periodically transmitted from the target moving object and provides it to the processor 11. The processor 11 uses the communication unit 4 to acquire the position calculated by the target moving object using the GNSS as position information. Note that the mode of transmission of the position information may be any method, such as broadcasting. The processor 11 may also calculate the position information of the target moving object using other methods. The processor 11 sequentially acquires the position information of the target moving object.

[0033] The size information of the target moving object is information indicating the size of the target moving object. Hereinafter, the size information of the target moving object is also simply referred to as size information. As an example, in this embodiment, the size information is information indicating the overall length, overall width, and overall height of the target moving object. Alternatively, the size information may be information that indirectly indicates the size of the target moving object, such as the model or model number of the target moving object. The communication device 21 of the target moving object periodically wirelessly transmits the size information of the target moving object. The communication unit 4 receives the size information periodically transmitted from the target moving object and provides it to the processor 11.

[0034] Note that the location information and size information may be included in one message. The communication device 21 of the target moving object may be configured to periodically transmit a vehicle status message including location information and size information. In addition to location and size, the vehicle status message may include driving speed, acceleration, vehicle body orientation (azimuth angle), shift position, steering angle, etc. The vehicle status message may be a CAM (Cooperative Awareness Message) defined in ETSI TS 102 637-2 or a BSM (Basic Safety Message) defined in SAE J 2735, etc.

[0035] The position information of the target moving object acquired by the processor 11 using the communication unit 4 is also referred to as received position information below. The size information of the target moving object acquired by the processor 11 using the communication unit 4 is also referred to as received size information below.

[0036] The processor 11 executes an area identification process to identify an object existence area Ar, which is an area where an object moving object may exist, based on the reception position information and reception size information. The processor 11 estimates an object outline Co, which is the outline of the object moving object, based on the reception position information and reception size information. The object outline Co defines the object existence area Ar. In other words, the part inside the object outline Co corresponds to the object existence area Ar.

[0037] After performing the above-described clustering, processor 11 may perform reclustering based on the target existence area Ar. Reclustering is a process of updating a set of observation points associated with one object based on reception position information and reception size information acquired through communication. Specifically, processor 11 estimates that observation points present within the target existence area Ar are observation points related to the target moving object. The observation points related to the target moving object are observation points that constitute a cluster related to the target moving object. The cluster related to the target moving object is also referred to as the target cluster hereinafter. Reclustering may be performed on an observation point basis.

[0038] Note that if there is a cluster in which some observation points extend outside the target existence area Ar, the extending observation points may be incorporated into the same group as other observation points that exist within the target existence area Ar. In other words, reclustering may be performed on a cluster-by-cluster basis. A cluster in which some observation points extend outside the target existence area Ar may be understood as a cluster that exists across the target contour Co. On the other hand, a cluster that exists across the target contour Co and in which the majority of the constituent observation points are located outside the target existence area Ar may be considered as a cluster corresponding to an object that is independent (different) from the target moving object.

[0039] With respect to the target moving object, the processor 11 identifies the type or state of the target moving object based on information on the observation points determined by reclustering, i.e., information on the observation points present in the target existence area Ar. After identifying the target cluster corresponding to the target moving object, the processor 11 executes a type identification process for the target cluster. However, the type of the target cluster may be the type acquired through communication. The type identification process for the target cluster may be omitted.

[0040] The processor 11 may acquire orientation information of the target moving object from the communication unit 4, the distance measurement sensor 2, or another sensor. The processor 11 may identify the orientation of the target moving object from time-series position data, or may acquire it from the target moving object through short-range communication. The processor 11 may acquire information indicating the steering angle of the target moving object from the communication device 21 as orientation information. Alternatively, the processor 11 may acquire the orientation of the target moving object using the camera 3. The processor 11 may predict the orientation from the travel trajectory of the target moving object and use the predicted orientation as orientation information.

[0041] In the region identification process, the processor 11 may identify the target existence region Ar and the target contour Co based on orientation information of the target moving object in addition to the reception position information and reception size information. If the orientation of the target moving object changes, the region in which the target moving object exists may change. Therefore, by estimating the target existence region Ar based on orientation information as well, it is expected that a more accurate target existence region Ar can be obtained.

[0042] After identifying the type or state of the object based on the observation point cloud data, the processor 11 performs sensor fusion processing, which is processing for integrating the detection results of the distance measurement sensor 2 and other sensors.

[0043] In this embodiment, as the sensor fusion process, the processor 11 integrates the detection results of the ranging sensor 2 and the camera 3. The processor 11 performs image recognition based on the captured image. The processor 11 integrates the result of identifying the type or state of an object based on the observation point cloud data with the result of performing image recognition based on the captured image. The fusion process finally determines the state and type of each detected object at the current time. The state and type of the detected object determined by the sensor fusion process are also referred to as the integrated result or the final recognition result. Note that the sensor fusion process may also use the prediction result of the current state of the detected object estimated from the tracking result described below. The sensor fusion process may be performed taking into account the characteristics (strengths / weaknesses) of each sensor depending on the situation.

[0044] The processor 11 tracks each detected object based on the final recognition result. Tracking involves comparing the current object position estimated from past detection results with the latest detection result to determine the correspondence between them and identify the current position of previously detected objects. By performing a tracking process each time a detection result is obtained from the ranging sensor 2, it becomes possible to identify the transition of an object's position, which changes over time. Such tracking includes determining the identity of the latest detected object with a previously detected object (so-called identification). If the processor 11 determines that an object detected in the current integration result and an object detected in the previous integration result are the same object, it links the two together. This linking makes it possible to track each object. Hereinafter, the tracking result data for each object will also be referred to as tracking data. A common detected object number or the like may be assigned to the same detected object, and the position data may be stored.

[0045] <Object recognition processing flow> Next, the object recognition process performed by the processor 11 will be described using the flowchart shown in Fig. 2. The object recognition process is repeatedly performed from when the ignition switch is turned on until it is turned off. The object recognition process may be performed sequentially at a predetermined cycle (for example, 100 milliseconds or 200 milliseconds) at which the ranging sensor 2 generates observation point cloud data.

[0046] In S11, the processor 11 acquires observation point cloud data from the ranging sensor 2. The processor 11 then performs clustering based on the observation point cloud data. In S12, the processor 11 determines whether or not it has acquired position information from at least one moving body 200 since the object recognition process was last executed. For ease of explanation, the case where the answer is No in S12, i.e., where no position information has been received from any moving body 200, will be described first. If there is no moving body 200 from which position information has been received, the process proceeds to S17.

[0047] In S17, the processor 11 identifies the type and position of the detected object for each cluster obtained by clustering. The processor 11 may estimate the orientation of the detected object in S17 from the distribution of the observation points that make up the cluster. In S18 following S17, the processor 11 performs sensor fusion processing. In this embodiment, the processor 11 integrates the detection results of the distance measuring sensor 2 and the camera 3, and identifies the identification and state of each detected object. Note that if the object recognition device 10 does not include other sensors such as the camera 3, the fusion processing in S18 may be omitted and S19 may be executed.

[0048] In S19, processor 11 determines the correspondence between the latest detected object and previously detected objects based on the detected object recognition result determined by the above process and the past recognition results. That is, the tracking data for each object is updated. A new detected object ID may be assigned to a newly detected object, and a new record may be created. In addition, in S19, processor 11 may determine the speed, direction of travel, and other conditions of the detected object from the time-series data of the position included in the tracking data.

[0049] After S18, after a predetermined time has elapsed, the process returns to S11, where the processor 11 acquires observation point cloud data from the ranging sensor 2. Then, clustering is performed again based on the observation point cloud data. In S12, the processor 11 again determines whether or not position information has been acquired from at least one mobile object 200. S12 essentially corresponds to determining whether a mobile object 200 capable of short-range communication is present within the communication range of the roadside unit 100. If a mobile object 200 capable of short-range communication is present, the processor 11 acquires size information along with the position information of the mobile object 200. If the answer is Yes in S12, the process proceeds to S13.

[0050] In S13, it is determined whether or not the moving object 200 (i.e., the target moving object) corresponding to the communication device 21 that is the sender of the location information, etc., is currently being tracked. The object being tracked may be interpreted as an object that has been previously detected using the ranging sensor 2 or another sensor. Specifically, the processor 11 determines whether or not there is an object that corresponds to the target moving object among the objects being tracked. The processor 11 determines, for example, from the received location information, whether or not there is an object that corresponds to the target moving object. If the determination in S13 is Yes, the process proceeds to S14. In S14, the processor 11 reads the received size information from the RAM 12. Note that, in addition to the location information, similarities in speed, size, color, type, etc. may also be used to identify the correspondence between the detected object and the target moving object in S13.

[0051] In S15, the processor 11 performs an area identification process. Based on the integration result and the tracking data, the processor 11 estimates the possible location of each object detected in S18 at the current time. FIG. 3 shows the estimated location of each detected object. In FIG. 3, each object is illustrated as targets Ob1 to Ob5. In FIG. 3, each target Ob1 to Ob5 is indicated by a representative position coordinate. Of course, in reality, each detected object is not a point, but has a location range according to its size / type. Note that the positive y-axis direction indicates the forward direction for the roadside device 100, the positive x-axis direction indicates the rightward direction for the roadside device 100, and the negative z-axis direction indicates the direction of gravity. If the roadside device 100 is positioned facing north, the positive x-axis direction corresponds to north and the positive y-axis direction corresponds to east. FIG. 3 also shows the reception position coordinate Lr corresponding to the reception position information.

[0052] The processor 11 determines the target Ob that is closest to the reception position coordinate Lr from among the targets Ob1 to 5 being tracked, and sets this position as the tentative center position Obc. In FIG. 3, the object Ob1 is closest to the reception position coordinate. Therefore, FIG. 3 illustrates a pattern in which the object Ob1 is the tentative center position Obc. The tentative center position Obc indicates a position that is estimated to be the center of the target moving object. The tentative center position Obc is a point that serves as a reference when calculating the target existence area Ar. As shown in FIG. 4, the processor 11 estimates the target contour Co and the target existence area Ar based on the reception size information, using the tentative center position Obc as a reference.

[0053] In S16, the processor 11 performs reclustering based on the identified object presence area Ar. FIG. 4 is a diagram showing observation points q1 to q7 indicated by the observation point cloud data. After estimating the object contour Co and the object presence area Ar, the processor 11 applies the estimated object contour Co and the object presence area Ar to the coordinate system of the observation point cloud data. The processor 11 then estimates the observation points present in the object presence area Ar as constituent points of the object cluster (in other words, the target moving object). In the example shown in FIG. 5, observation points q1 and q2 are present in the object presence area Ar. Therefore, in the example shown in FIG. 5, the processor 11 determines that the observation points q1 and q2 are constituent points of the object cluster. Note that, for simplicity of illustration, only one observation point constituting one cluster is shown here. There may be multiple observation points near the observation point q1 so as to constitute one cluster. The same applies to the other observation points q2 to q7. The observation points q1 to q7 shown in FIG. 5 may each be understood as being replaced with a cluster.

[0054] In S17, the processor 11 performs type identification processing for each cluster, as described above. However, the type identification processing for clusters whose types have already been identified through communication (i.e., target clusters) may be omitted. The processor 11 also performs state identification processing for each cluster. In S18, the processor 11 performs sensor fusion processing. In S19, the processor 11 updates the tracking data for each detected object.

[0055] The processor 11 may acquire orientation information of the target moving object together with the position information and size information using the communication unit 4. In this case, in the area identification process of S15, the target existence area Ar may be identified based on the received position information, received size information, and orientation information.

[0056] <Summary of the First Embodiment> According to the object recognition device 10 of this embodiment, the observation points in the object existence area Ar are estimated as observation points for one object (i.e., the target moving object) and the target moving object is recognized. This makes it possible to properly identify the target moving object. As a result, the state of the detected object can also be properly determined. Therefore, the possibility of erroneously determining the type or state of the target moving object can be reduced.

[0057] In this embodiment, a LiDAR is used as the distance measurement sensor 2. This is expected to enable highly accurate estimation of the position of an object.

[0058] <Modification> The object recognition device 10 may be mounted on a vehicle. When the object recognition device 10 is mounted on a vehicle, the other sensors may include on-board sensors that detect the running state of the vehicle, the environment around the vehicle, and the like.

[0059] The processor 11 may acquire information about the travel of the target moving object, such as the vehicle speed or blinker of the target moving object, from the communication device 21. The information about the travel of the target moving object may be used in the area identification process. The information about the travel of the target moving object may be used when performing the sensor fusion process or the process of tracking each object.

[0060] Although processor 11 has determined tentative center position Obc from among targets Ob1 to Ob5 being tracked, it does not have to be determined from the object being tracked. Processor 11 may also determine the center coordinates of the cluster closest to the reception position coordinate Lr among the clusters indicated by the observation point group data as tentative center position Obc. The center coordinates of the cluster may be the average position coordinates or the center of gravity position coordinates of the observation points that make up the cluster.

[0061] Second Embodiment The processor 11 may identify the center position cn of the target moving object as the state of the target moving object. The processor 11 may estimate that at least one of the observation points present in the target existence area Ar is an observation point related to the edge of the target moving object, and identify the center position cn of the target moving object based on the information on the observation point related to the edge and the received size information. Hereinafter, the observation point related to the edge will also be referred to as the edge observation point qe.

[0062] A method for identifying the center position cn in this modified example will be described below with reference to FIG. 6. The processing performed in this modified example is performed after step 17 in the flowchart of FIG. 2. FIG. 6 is a diagram showing observation points q1 to q6 indicated by observation point cloud data. In the example shown in FIG. 6, observation points q1 and q2 are present in the target existence area Ar. Processor 11 estimates observation points q1 and q2 as edge observation points qe. Processor 11 identifies the center position cn of the target moving object based on information about edge observation point qe and received size information.

[0063] Because electromagnetic waves are reflected from the surface of an object, the observation point detected by the ranging sensor 2 may indicate the edge of the object. In this configuration, the processor 11 estimates the observation point present in the target existence area Ar as the edge observation point qe. This allows the processor 11 to identify the center position cn of the target moving object using information on the edge observation point qe and the received size information.

[0064] Third Embodiment In the previous embodiment, in the area identification process, the processor 11 identified the target existence area Ar using tracking data. The target existence area Ar may be identified by other methods. In the present embodiment, the processor 11 tentatively determines the target existence area Ar based on the ranging sensor 2 or another sensor. The processor 11 corrects the tentatively determined target existence area Ar using reception position information and reception size information. The processor 11 identifies the type or state of the target moving object based on information on observation points present in the corrected target existence area Ar.

[0065] In this embodiment, the processor 11 estimates the position and size of an object within the detection range of the distance measuring sensor 2 based on the captured image acquired from the camera 3. The processor 11 extracts the edges (contours) of the objects in the captured image using a technique such as edge extraction, and identifies the position and size of each object included in the captured image. If the camera 3 is a monocular camera, the position of the object in real space may be roughly estimated from the position of the object in the image. However, if the camera is a stereo camera, the position of the object in real space may be estimated from the parallax information of the two images.

[0066] In this embodiment, the processor 11 determines a provisional area Arc based on the camera 3 as another sensor. The provisional area Arc is a provisional (tentative) target existence area Ar. Specifically, the processor 11 provisionally determines a provisional outline Coc, which is a provisional target outline Co, based on the camera. The provisional outline Coc defines the provisional area Arc. Specifically, the processor 11 identifies a target moving object in the captured image based on the edge of the object extracted using the captured image, reception position information, and reception size information. The processor 11 provisionally determines the edge corresponding to the target moving object as the provisional outline Coc. The processor 11 provisionally determines the area defined by the provisional outline Coc as the provisional area Arc.

[0067] The processor 11 corrects the provisional area Arc using the reception position information and reception size information to determine the object existence area Ar. Specifically, the processor 11 corrects the provisional contour Coc using the reception position information and reception size information to determine the object contour Co. The corrected object contour Co is also referred to as the determined contour Cof. The corrected object existence area Ar is also referred to as the determined area Arf. The determined area Arf is defined by the determined contour Cof.

[0068] As described above, the processor 11 identifies the target existence area Ar based on the received position information and received size information. On the other hand, the processor 11 estimates the remaining object area, which is the range in which a remaining object may exist, based on the position and size of the object estimated by the camera 3. The remaining object here is an object for which position information and size information cannot be obtained using the communication unit 4. The remaining object may also be referred to as a communication-incompatible object. The remaining object area is defined by edges extracted from the captured image other than the edges corresponding to the target moving object.

[0069] Processor 11 estimates that observation points present in the determined area Arf are observation points related to the target moving object, and identifies the type or state of the object. Processor 11 then groups observation points present in the remaining object area as observation points for a single object, and identifies the type or state of the object. For objects whose position information and size information have been acquired through communication but which have not been detected by camera 3, processor 11 may estimate the presence area based on the data acquired through communication and tracking data, and group the observation points.

[0070] The object recognition process performed by the processor 11 in this embodiment will be described below with reference to the flowchart shown in Fig. 7. The same step numbers are assigned to steps that are the same as those in the previous embodiment. Explanations of steps that are the same as those in the previous embodiment will be omitted.

[0071] In S25, the processor 11 performs an area identification process. The processor 11 reads out the captured image from the RAM 12. The processor 11 extracts the edges (contour lines) of objects in the captured image and identifies the position and size of the detected object. FIG. 8 is a diagram showing edges Ed1 to Ed3 extracted from the captured image. The processor 11 identifies the target moving object in the captured image based on the extracted object edges, reception position information, and reception size information. In the example shown in FIG. 8, the processor 11 identifies the edge Ed1 as the edge corresponding to the target moving object based on the position information, etc. The processor 11 identifies the remaining edges Ed2 and Ed3 as edges corresponding to the remaining object. The edges Ed2 and Ed3 define the remaining object area. The processor 11 determines the edge Ed1 to be the provisional contour Coc, and determines the inside of the provisional contour Coc to be the provisional area Arc.

[0072] In general, the processor 11 determines the determined area Arf and the determined contour Cof by correcting the provisional area Arc using the reception position information and the reception size information. Correction of the provisional area Arc using the reception position information and the reception size information may be performed, for example, using data of a second provisional contour Coe determined using the reception position information and the reception size information, as described below. For convenience, the provisional area Arc and provisional contour Coc determined based on the camera image are also referred to as the first provisional area Arc and the first provisional contour Coc.

[0073] As a preparatory process for determining the confirmed area Arf and the confirmed contour Cof, the processor 11 sets a second provisional contour Coe based on the reception position coordinates Lr and the reception size information. In Fig. 8, the second provisional contour Coe determined based on the received data is shown by a dashed line, and the first provisional contour Coc determined based on the camera image is shown by a solid line.

[0074] As an example, the determined contour Cof is set at a position midway between the second provisional contour Coe and the first provisional contour Coc. The determined contour Cof is indicated by a dashed line in the figure. The area inside the determined contour Cof corresponds to the determined area Arf. Note that if it is predicted that the target moving object's position will be estimated with high accuracy, the determined contour Cof may be set to coincide with the second provisional contour Coe. A case in which the target moving object's position is estimated with high accuracy may be when the target moving object uses the Quasi-Zenith Satellite System (QZSS) or when a localization process is performed. The localization process refers to a process of estimating the target moving object's own position on a map by comparing information on terrain / feature detected by an environment recognition sensor such as a camera with information on terrain / feature indicated in map data. The control unit 1 may acquire information indicating the reliability (in other words, accuracy) of the location information from the target moving object and change the way the determined contour Cof is determined depending on the reliability of the received location information. The method of correcting the provisional contour Coc, in other words, the specific method of determining the determined contour Cof from the information on the provisional contour Coc and the reception position information, is not limited to the above method, and various other methods may be applied.

[0075] In S26, processor 11 performs reclustering based on the confirmed area Arf. FIG. 9 is a diagram showing observation points q1 to q5 indicated by the observation point cloud data. Processor 11 applies the confirmed area Arf to the coordinate system of the observation point cloud data. FIG. 10 shows the diagram after application. Then, the observation points present in the confirmed area Arf are estimated to be constituent points of the target cluster. In the example shown in FIG. 10, observation points q1 and q2 are present in the confirmed area Arf. Therefore, in the example shown in FIG. 10, processor 11 determines that observation points q1 and q2 are constituent points of the target cluster (in other words, the target moving object).

[0076] In S27, processor 11 groups the observation points corresponding to the remaining objects. As described above, edges Ed2 and Ed3 define the remaining object region. Processor 11 applies the identified remaining object region to the coordinate system of the observation point cloud data. Figure 10 shows the result after application. Processor 11 groups the observation points that exist in the remaining object region defined by edge Ed2 as observation points for a single object. This process corresponds to the process of reclustering the observation points that exist in the remaining object region defined by the image recognition-based edges as a cluster corresponding to a single object.

[0077] In the example shown in Fig. 10, observation point q3 exists in the remaining object region defined by edge Ed2. Processor 11 groups observation point q3 as an observation point for one object. Also, in the example shown in Fig. 10, observation points q4 and q5 exist in the remaining object region defined by edge Ed3. Processor 11 groups observation points q4 and q5 as observation points for one object.

[0078] Note that, for simplicity of illustration, only one observation point constituting one cluster is shown here. There may be multiple observation points constituting one cluster near observation point q3. The same applies to the other observation points q1, q2, q4 to q7. Observation points q1 to q7 shown in FIG. 5 may be understood as being replaced with clusters.

[0079] In S28, processor 11 performs type identification processing for each cluster. For target moving objects, processor 11 performs type identification processing based on the target cluster. Note that, as in the preceding embodiment, the type identification processing for the target cluster may be omitted. For remaining objects, processor 11 performs type identification processing based on the observation points grouped in S27. In addition, processor 11 performs state identification processing for each cluster.

[0080] According to this embodiment, the processor 11 tentatively determines the target existence area Ar based on the camera 3, and corrects the tentatively determined target existence area Ar using the reception position information and reception size information. This allows the target existence area Ar to be corrected to a more accurate one. Therefore, the observation points can be grouped based on the more accurate target existence area Ar. As a result, it becomes possible to more accurately determine the type or state of the target moving object. Therefore, the possibility of misidentifying the target moving object can be reduced.

[0081] The size information obtained through communication can be more accurate than the information obtained from the distance measurement sensor 2 and other sensors. Therefore, as in this embodiment, for a target moving object that is a communicable object, the target existence area Ar is identified based on the information obtained through communication, and the observation points are grouped based on the identified target existence area Ar, thereby making it possible to group the observation points based on a more accurate target existence area Ar. This makes it possible to more appropriately determine the type or state of the target moving object.

[0082] On the other hand, processor 11 estimates the existence area of ​​an object with which communication is not possible (i.e., a remaining object) based on the position and size of the object estimated by camera 3. Then, processor 11 groups observation points that exist within the estimated existence area as observation points for one object. This makes it possible to group observation points by estimating the existence area of ​​an object with which communication is not possible using camera 3 and grouping the observation points based on the estimation result. Therefore, it becomes possible to more accurately determine the type or state of a remaining object.

[0083] <Modification> In S25, the processor 11 corrects the provisional contour Coc to a position midway between the estimated contour Coe and the provisional contour Coc, but this is not necessarily the only possible configuration. The provisional contour Coc may also be corrected so as to coincide with the estimated contour Coe.

[0084] The processor 11 provisionally determines the target existence area Ar and corrects the provisional area Arc using the reception position information and reception size information, but this configuration is not necessarily limited. The provisional area Arc does not necessarily have to be corrected. For example, if the reliability of the reception position information is low, the processor 11 may use the provisional area Arc as the confirmed area Arf as is.

[0085] When the processor 11 calculates the object contour Co, the object existence area Ar is also determined. Conversely, when the processor 11 calculates the object contour area Ar, the object contour Co is also determined. The object existence area Ar and the object contour Co have a one-to-one relationship. For this reason, these terms may be interchanged as appropriate in the description of the present disclosure. Expressions such as provisional contour and provisional area, and determined contour and determined area may also be interchanged. [Explanation of symbols]

[0086] 1...control unit, 2...ranging sensor, 3...camera, 4...communication unit, 10...object recognition device, 11...processor, 12...RAM, 13...storage, 14...I / O, 21...communication device, 100...roadside unit, 200...mobile body, Ar...target existence area.

Claims

1. An object recognition device that performs a process to recognize an object present in a predetermined recognition target area, a communication unit (4) configured to be capable of wirelessly communicating with a communication device attached to the mobile object; a distance measurement sensor (2) for detecting an observation point indicating the position where the object is present; a control unit (1) that identifies the type or state of the object based on the observation point, The control unit using the communication unit to acquire position information and size information of a target moving body that is the moving body corresponding to the communication device; executes an area identification process for identifying a target existence area (Ar) that is an area where the target moving object may exist, based on reception position information, which is the position information, and reception size information, which is the size information, acquired using the communication unit; Among the observation points, an observation point that exists within the target existence region is estimated to be an observation point related to the target moving body; An object recognition device configured to identify the type or state of the target moving object based on information about the observation point present in the target existence area.

2. the control unit, in the area identification process, provisionally determines the target existence area based on the distance measurement sensor or another sensor, and corrects the provisionally determined target existence area using the reception position information and the reception size information; The object recognition device according to claim 1 , wherein the type or state of the target moving object is identified based on information about the observation points present within the corrected target existence region.

3. The control unit Acquire an image captured by a camera (3) that includes the recognition target area in its imaging range, estimating the position and size of the object within the detection range of the distance measuring sensor based on the captured image; The target existence area is identified based on the reception position information and the reception size information; a remaining object area, which is an area where a remaining object, which is an object whose position information and size information cannot be acquired using the communication unit, may exist, is estimated based on the position and size of the object estimated by the camera; Estimating an observation point present in the target existence region as an observation point related to the target moving object, and identifying the type or state of the target moving object; The object recognition device according to claim 1 , wherein an observation point present in the remaining object region is estimated as an observation point for one object, and the type or state of the remaining object is identified.

4. The control unit provisionally determining a target contour, which is a contour of the target moving object that defines the target existence region, based on the captured image; The object recognition device according to claim 3 , wherein the region specifying process includes correcting the provisionally determined target contour using the reception position information and the reception size information.

5. the moving body is a vehicle, The control unit acquiring orientation information of the target moving object from the communication unit, the distance measurement sensor, or another sensor; The object recognition device according to claim 1 , wherein the target existence area is identified based on the orientation information in the area identification process.

6. the distance measuring sensor is a sensor that detects the observation point by transmitting electromagnetic waves of a predetermined frequency toward the recognition target area and receiving the electromagnetic waves reflected by an object, The control unit A center position of the target moving object is identified as the state of the target moving object, estimating at least one of the observation points present in the target existence region as an observation point relating to an end of the target moving body; The object recognition device according to claim 1 , wherein the center position of the target moving object is identified based on information about the observation point related to the edge and the received size information.

7. The object recognition device according to claim 1 , wherein the distance measurement sensor is a LiDAR.

Images