Recognition system

Abstract

To provide a technique capable of accurately recognizing objects in a surrounding environment.SOLUTION: A recognition system includes a distance measurement sensor 13, an image sensor 15, and an information integration unit 17. The information integration unit 17 includes a distance measurement processing unit 19 and a target state estimation unit 21. The distance measurement processing unit 19 is configured to process information estimated by the distance measurement sensor 13. The target state estimation unit 21 is configured to estimate the state of the target based on the information processed by the distance measurement processing unit 19 and the information estimated by the image sensor 15. The distance measurement processing unit 19 is configured to integrate the plurality of targets considering the plurality of targets to be the same target when there are a plurality of targets whose positions are estimated by the distance measurement sensor 13 within a predetermined range from a target boundary estimated by the image sensor 15.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This disclosure relates to a technology for recognizing objects (i.e., targets) in the surrounding environment.

Background Art

[0002] Conventionally, technologies for recognizing surrounding objects using radar or cameras are known (for example, see Patent Document 1 below). The technology described in this Patent Document 1 is a technology for determining the type of an object detected by radar using a database, and by updating the database with the result of the type determination of the camera as a teacher, the identification performance of the radar is improved.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, as a result of the inventors' detailed examination, the following problems were found in the conventional technology. When a vehicle is misjudged as a plurality of pedestrians based on a signal from a radar, the identification performance of the radar may not be improved because the misjudged data is used to update the database.

[0005] One aspect of this disclosure aims to provide a technology capable of accurately recognizing objects in the surrounding environment.

Means for Solving the Problems

[0006] (1) One aspect of this disclosure relates to a recognition system (5) for estimating the state of a target around itself. This recognition system includes a distance measuring sensor (13), an image sensor (15), and an information integration unit (17).

[0007] The distance measuring sensor is configured to obtain point cloud information corresponding to an object based on the signal of reflected waves from a probe wave that surveys the surroundings, and to estimate at least the position of the object, among its position and type, from the point cloud information.

[0008] The image sensor is configured to estimate the location of an object, as well as its boundaries, based on the signal from an image captured of the surroundings. The information integration unit is configured to perform processing to estimate the state of an object based on the information estimated by the distance measuring sensor and the information estimated by the image sensor.

[0009] This information integration unit comprises a distance measurement processing unit (19) and a target state estimation unit (21). The distance measurement processing unit is configured to process information estimated by the distance measurement sensor. The target state estimation unit is configured to estimate the state of a target based on information processed by the distance measurement unit and information estimated by the image sensor.

[0010] Furthermore, the distance measurement processing unit is configured to consider multiple targets whose positions have been estimated by the distance measurement sensor as the same target and integrate them when there are multiple targets whose positions have been estimated by the distance measurement sensor within a predetermined range from the boundary of the target estimated by the image sensor.

[0011] With this configuration, the present disclosure makes it possible to accurately recognize objects in the surrounding environment (i.e., targets). Specifically, in this disclosure, when there are multiple targets whose positions are estimated by a distance measuring sensor within a predetermined range from the boundary of a target estimated by an image sensor, the system is configured to treat the multiple targets as the same target and integrate them.

[0012] This makes it possible to accurately estimate the state of an object based on information estimated by the image sensor (i.e., information about the object's boundaries) and information estimated by the distance sensor and integrated by the distance measurement processing unit. For example, by considering an object whose position is estimated by the image sensor and an object whose multiple objects are integrated by the distance measurement processing unit as the same object, the state of that object (e.g., position, size, type, etc.) can be estimated with accuracy.

[0013] (2) Other aspects of the disclosure relate to a recognition system (5) for estimating the state of objects in its surroundings. This recognition system includes a distance measuring sensor (13), an image sensor (15), and an information integration unit (17).

[0014] The distance measuring sensor is configured to obtain point cloud information corresponding to an object based on the signal of reflected waves from a probe wave that surveys the surroundings, and to estimate at least the position of the object, among its position and type, from the point cloud information.

[0015] The image sensor is configured to estimate the location of an object, as well as its boundaries, based on the signal from an image captured of the surroundings. The information integration unit is configured to perform processing to estimate the state of an object based on the information estimated by the distance measuring sensor and the information estimated by the image sensor.

[0016] This information integration unit comprises a distance measurement processing unit (19) and a target state estimation unit (21). The distance measurement processing unit is configured to process information estimated by the distance measurement sensor. The target state estimation unit is configured to estimate the state of a target based on information processed by the distance measurement unit and information estimated by the image sensor.

[0017] Furthermore, when there are a plurality of targets whose positions are estimated by the distance measurement sensor within a predetermined range from the boundary of the target among the states of the target estimated by the target state estimation unit, the distance measurement processing unit is configured to integrate the plurality of targets by regarding the plurality of targets as the same target based on the information indicating the boundary of the target.

[0018] With such a configuration, in the present disclosure, it is possible to accurately recognize an object (i.e., a target) in the surrounding environment. Specifically, in the present disclosure, when there are a plurality of targets whose positions are estimated by the distance measurement sensor within a predetermined range from the boundary of the target among the states of the target estimated by the target state estimation unit, the distance measurement processing unit is configured to integrate the plurality of targets by regarding the plurality of targets as the same target based on the information indicating the boundary of the target.

[0019] Thereby, based on the information estimated by the target state estimation unit (i.e., the information regarding the boundary of the target) and the information estimated by the distance measurement sensor and integrated by the distance measurement processing unit, it becomes possible to accurately estimate the state of the target. For example, by regarding the target whose position is estimated by the target state estimation unit and the target obtained by integrating a plurality of targets by the distance measurement processing unit as the same target, the state of the target (e.g., position, size, type, etc.) can be accurately estimated.

[0020] In addition, the reference numerals in parentheses described in this column and the claims indicate the correspondence relationship with the specific means described in the embodiments described later as one aspect, and do not limit the technical scope of the present disclosure.

Brief Description of the Drawings

[0021] [Figure 1] It is an explanatory diagram showing a roadside device using the recognition system of the first embodiment and the surrounding configuration. [Figure 2] It is a block diagram functionally showing the recognition system of the first embodiment. [Figure 3] It is an explanatory diagram showing the principle of the processing in the recognition system of the first embodiment. [Figure 4] This is an explanatory diagram illustrating the process of integrating targets in the recognition system of the first embodiment. [Figure 5] This is a flowchart showing the processing in the recognition system of the first embodiment. [Figure 6] This is an explanatory diagram illustrating the effects of the recognition system of the first embodiment. [Figure 7] This is a flowchart showing the processing in the recognition system of the second embodiment. [Figure 8] This is a block diagram functionally illustrating the recognition system of the third embodiment. [Figure 9] This is a flowchart showing the processing in the recognition system of the third embodiment. [Figure 10] This is a block diagram functionally illustrating the recognition system of the fourth embodiment. [Figure 11] This is a flowchart showing the processing in the recognition system of the fourth embodiment. [Figure 12] This is a flowchart showing the processing in the recognition system of the fifth embodiment. [Figure 13] This is a flowchart showing the processing in the recognition system of the sixth embodiment. [Figure 14] This is a flowchart showing the processing in the recognition system of the seventh embodiment. [Modes for carrying out the invention]

[0022] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. [1. First Embodiment] In this first embodiment, we describe a recognition system applied to the recognition of targets, such as vehicles (e.g., automobiles and bicycles) and people.

[0023] This recognition system is used, for example, in roadside devices placed along roads or in vehicles such as automobiles, and here we will explain it using roadside devices as an example. [1-1. Overall Structure] As shown in Figure 1, in this first embodiment, a roadside device 1 is positioned along the road (for example, near an intersection), and this roadside device 1 is equipped with a recognition system 5 that can recognize the status of targets such as vehicles (for example, cars, bicycles) 3 and people traveling on the road.

[0024] The recognition system 5 includes, as a hardware configuration, a distance measuring device 7, an imaging device 9, and an electronic control device 11 for performing various calculations such as target estimation. The distance measuring device 7 and the imaging device 9 may have a different configuration from the recognition system 5. Also, the electronic control unit 11 may have a different configuration from the roadside device 1. For example, the electronic control unit 11 may be connected to the distance measuring device 7 and the imaging device 9 located in the roadside device 1 via a communication line (e.g., the Internet).

[0025] The rangefinder 7 is a measuring device used to measure the position of a target (for example, the distance to the target or the direction of the target) and the relative velocity to the target. It has a configuration that periodically transmits a probe wave in a predetermined direction and receives the reflected wave that is reflected off the target.

[0026] Examples of this ranging device 7 include well-known LiDAR (i.e., lidar), radar, sonar, etc. In this example, we will use LiDAR, which uses laser light as the search wave, as the ranging device 7.

[0027] The imaging device 9 is a device capable of imaging the surroundings, and examples include cameras such as CCD cameras. A monocular camera or a stereo camera can be used as the camera. The electronic control unit 11, although not shown in the diagram, is an ECU equipped with a well-known CPU, ROM, RAM, and other memory. ECU stands for Electronic Control Unit. In other words, the electronic control unit 11 is a device equipped with a well-known microcomputer (not shown). Note that the memory is not limited to the memory within the microcomputer, but also includes various external storage devices (e.g., hard disks).

[0028] The various functions performed by the ECU are realized by the CPU executing a program stored in a non-transitional tangible recording medium. In this example, memory corresponds to the non-transitional tangible recording medium that stores the program. Furthermore, when this program is executed, the method corresponding to the program is executed.

[0029] Furthermore, the memory stores not only various programs (for example, programs for estimating the state of an object), but also various data used when executing those programs.

[0030] The methods for realizing the various functions of the aforementioned ECU are not limited to software; some or all of its elements may be realized using one or more hardware components. For example, if the above functions are realized by an electronic circuit, which is hardware, that electronic circuit may be a digital circuit containing a large number of logic circuits, an analog circuit, or a combination thereof.

[0031] [1-2. Functional Configuration of the Recognition System] Next, we will describe the functional configuration of the recognition system 5 (specifically, the electronic control unit 11).

[0032] As shown in Figure 2, the recognition system 5 is a system that estimates the state of objects in its surroundings, and functionally comprises a distance measuring sensor 13, an image sensor 15, and an information integration unit 17. The distance measuring sensor 13 is configured to obtain point cloud information corresponding to an object based on the signal of the reflected wave from the exploration wave (i.e., the laser light emitted from the LiDAR, which is the distance measuring device 7) that explores the surroundings, and to estimate at least the position of the object, among its position and type, from the point cloud information. This distance measuring sensor 13 can estimate, for example, the distance to the object, the direction of the object, the type of object, the shape of the object, the size of the object, etc.

[0033] The image sensor 15 is configured to estimate the position of an object, or at least the object's position, and its boundaries, based on the signal from an image of the surroundings (i.e., an image captured by the camera, which is the imaging device 9). The image sensor 15 can, for example, estimate the distance to the object, the object's direction, the object's type, the object's shape, the object's size, and so on.

[0034] The information integration unit 17 is configured to perform processing to estimate the state of a target based on the information estimated by the distance measuring sensor 13 and the information estimated by the image sensor 15. This information integration unit 17 includes a distance measurement processing unit 19 and a target state estimation unit 21.

[0035] The distance measurement processing unit 19 is configured to process the information estimated by the distance measurement sensor 13. The target state estimation unit 21 is configured to estimate the state of a target based on the information processed by the distance measurement processing unit 19 and the information estimated by the image sensor 15. This target state estimation unit 21 can estimate, for example, the distance to the target, the direction of the target, the type of target, the shape of the target, and the size of the target.

[0036] Furthermore, the distance measurement processing unit 19 is configured to consider multiple targets whose positions have been estimated by the distance measurement sensor 13 as the same target and to integrate (i.e., merge) them if there are multiple targets whose positions have been estimated by the distance measurement sensor 13 within a predetermined range from the boundary of the target estimated by the image sensor 15.

[0037] In other words, since the multiple targets estimated by the distance measuring sensor 13 are within a predetermined range from the boundary of the target estimated by the image sensor 15, the multiple targets are treated as the target corresponding to the target estimated by the image sensor 15 (i.e., representing the same vehicle, etc.), and the process is performed to integrate the multiple targets into a single target. That is, the data of the multiple targets is treated as a single target corresponding to the target estimated by the image sensor 15.

[0038] Here, the predetermined range from the target boundary is defined as a distance set outward from the target boundary (i.e., on the opposite side from the center). This predetermined distance is a value that takes into account the error in the estimated target boundary, and can be set through experiments, etc. When converted to an actual distance, the predetermined distance can range from 0 cm to several tens of centimeters. If the predetermined distance is set to 0 cm, the target boundary and the predetermined range from the target boundary coincide.

[0039] [1-3. Overview of processing in the recognition system] Next, an overview of the process performed when the recognition system 5 estimates the state of an object will be explained based on Figures 3 and 4. Note that Figures 3 and 4 show a plan view (i.e., a bird's-eye view) of the road from above.

[0040] As shown in Figure 3, in order to estimate the state of the target, an imaging device (i.e., a camera) 9 located on the roadside device 1 periodically images a predetermined range on the road where the vehicle 3 is traveling. In other words, it images the road and the vehicle 3 traveling on the road, and periodically acquires the image (i.e., captured image). Here, the vehicle 3 is given as an example of a four-wheeled automobile.

[0041] The images captured by the camera show roads and vehicles 3 traveling on them. By applying well-known image recognition processing to these images, vehicles 3 can be recognized. For example, as is well known, vehicles 3 can be recognized by matching processing using object models (i.e., pattern matching). This pattern matching can also be used to recognize things other than automobiles, such as bicycles and people.

[0042] Therefore, it is possible to distinguish between the area of ​​the recognized vehicle 3 and the other areas. The area of ​​the vehicle (e.g., automobile) 3 can be represented by, for example, a rectangle. In Figure 3, the outline of vehicle 3 obtained in this way (i.e., the estimated rectangular outline) is shown by a dashed line. The shaded area of ​​the square at the center of the rectangular outline indicates the position of vehicle 3 estimated from past data, and the range of the rectangular outline is the area separated from the shaded area at the center of vehicle 3 by an amount corresponding to the outline of vehicle 3.

[0043] The dashed rectangular outline corresponds to the target boundary estimated by the image sensor 15. Here, for the sake of clarity, an example is shown where the target boundary and a predetermined range from the target boundary coincide (or substantially coincide). Alternatively, the predetermined range from the target boundary may be set at a slightly distant location, enclosing the estimated target boundary.

[0044] In addition, separately from this, in order to estimate the state of the target, the distance measuring device (i.e., LiDAR) 7 located on the roadside device 1 conventionally irradiates a predetermined range on the road where the vehicle 3 travels (i.e., the vehicle 3 traveling on the road) with laser light and receives the reflected light.

[0045] The detection signal from the reflected light of this LiDAR is processed by the distance measuring sensor 13, and as is well known, the position of the object to be detected (i.e., the orientation of the object, the distance to the object, and the position in a certain coordinate system) and the type of object are detected (i.e., estimated).

[0046] More specifically, since LiDAR generates a large number of reflected beams, a well-known clustering process is performed on these beams. For example, using the position and intensity of the reflected beams as elements, a process is used, such as the k-means method, to classify the reflected beams into multiple clusters corresponding to the target (more specifically, the type of target).

[0047] As is well known, the type of target can be estimated from the size, shape, location, and time-series changes of clusters. In Figure 3, three solid rectangular frames are shown within a dashed rectangle corresponding to one vehicle 3. These frames indicate the areas that have been classified by clustering and recognized as people (i.e., pedestrians). These rectangles are circumscribing the reflection points of each cluster (i.e., rectangles with their sides oriented either vertically or horizontally in Figures 3 and 4), as shown in Figure 4.

[0048] Therefore, under normal circumstances, as shown in the lower part of Figure 3, the area equivalent to one car is recognized based on the image captured by the camera (i.e., by the image sensor 15), and the presence of three people within this area equivalent to one car is recognized based on the reflected light from the LiDAR (i.e., by the distance measuring sensor 13).

[0049] However, it is unnatural for the area of ​​vehicle 3 and the area of ​​the three pedestrians to overlap. Therefore, in this first embodiment, the data of the targets corresponding to the three people located within the area of ​​one vehicle recognized by the distance measuring sensor 13 (i.e., the target data obtained from reflected light) is treated as data for vehicle 3 (i.e., data for the same target).

[0050] In other words, the data for each of the three pedestrian markers is integrated into the data for vehicle 3, which is the same marker. For example, as shown in Figure 4, first, as target data, a rectangle (i.e., a rectangle shown by a solid line) is set that circumscribes the data of each reflection point (i.e., the data of the position where the reflected light is reflected) corresponding to each pedestrian. In this case, the target is defined by the data of the vertices of the rectangle (i.e., the position data) and the data of the center of the rectangle (i.e., the centroid) (i.e., the position data).

[0051] Next, we assume a large rectangle (for example, rectangle MK shown by the dashed line in the figure) that circumscribes the three rectangles corresponding to each pedestrian, and this large rectangle MK is adopted as data indicating the range of vehicle 3. In this case, the vehicle 3 target corresponding to the large rectangle MK is defined by the data of the vertices (position data) of the large rectangle MK and the data of the center of the large rectangle MK (position data).

[0052] Then, the data for the targets corresponding to the three pedestrians is used as the data for the target corresponding to vehicle 3, and for example, the position of vehicle 3 (i.e., its position in a certain coordinate system, its distance from roadside device 1, and its direction from roadside device 1) and the relative speed of vehicle 3 are calculated.

[0053] For example, it is possible to estimate the current position of vehicle 3 (for example, its position in a certain coordinate system) from the centroid of a large rectangular area MK (for example, the center of a bird's-eye view). This allows for accurate estimation of the state of vehicle 3 using LiDAR data. In other words, the position of vehicle 3 and its relative velocity can be determined with high accuracy.

[0054] [1-4. Control Processing] Next, the control process performed when the recognition system 5 estimates the state of the target will be explained based on the flowchart in Figure 5. This process is performed at predetermined intervals.

[0055] In step 100 (hereinafter referred to as S) of Figure 5, target information is received from the distance sensor 13 and the image sensor 15. Here, the target information obtained from the distance sensor 13 is the target information after clustering the reflection point data from LiDAR. The target information obtained from the image sensor 15 is the target information recognized from the image captured by the camera, and also includes information on the estimated target boundary.

[0056] In the subsequent S110, it is determined whether there are multiple targets (for example, human targets) estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target (i.e., vehicle 3) estimated by the image sensor 15. If the determination is positive, the process proceeds to S120; if the determination is negative, the process proceeds to S130.

[0057] Here, the predetermined range from the boundary of the target is defined as, for example, the area within the outline of vehicle 3 shown by the dashed line in Figures 3 and 4. In other words, the example given is when the predetermined range is small (for example, when it practically coincides with the boundary of the target) (the same applies to other embodiments below).

[0058] In S120, since there are multiple targets estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target estimated by the image sensor 15, these multiple targets are integrated (i.e., merged), and the process proceeds to S130.

[0059] In other words, the data of multiple targets estimated by the distance measuring sensor 13 (for example, data identified as a person) corresponds to the data of the target estimated by the image sensor 15 (for example, vehicle 3). Then, the information of multiple targets is treated as the information of a single target (i.e., as the data of the large rectangular MK), as shown in Figure 4.

[0060] Therefore, the data within this large rectangular area MK can be used to determine the position and relative speed of the target (i.e., vehicle 3). In S130, a well-known integrated recognition process is performed, and the process is temporarily terminated. This integrated recognition process is a so-called fusion process, in which, based on the information obtained by the image sensor 15 and the information obtained by the distance measuring sensor 13, for example, more accurate information is selected to estimate the state of the target with high accuracy.

[0061] For example, when using LiDAR, the accuracy of measurement such as distance and relative velocity is superior to when using a camera, but it is inferior in terms of object type recognition. In other words, when using a camera, the accuracy of object type recognition is higher than when using LiDAR.

[0062] Therefore, in the integrated recognition process, the accuracy of each piece of information can be improved by adopting the information from each sensor 13 and 15, which provides highly accurate information, or by adjusting the weighting of each piece of information.

[0063] [1-5. Effects] (1) According to this first embodiment, it is possible to accurately recognize objects in the surrounding environment (i.e., targets).

[0064] Specifically, in this first embodiment, when there are multiple targets whose positions are estimated by the distance measuring sensor 13 within a predetermined range from the boundary of a target (e.g., a vehicle 3) estimated by the image sensor 15, the system is configured to consider these multiple targets as the same target (i.e., the vehicle 3) and integrate them.

[0065] This makes it possible to accurately estimate the state of an object based on the information estimated by the image sensor 15 (i.e., information regarding the boundary of the object) and the information estimated by the distance measuring sensor 13 and integrated by the distance measuring processing unit 19.

[0066] For example, by considering an object whose position is estimated by the image sensor 15 and an object whose position is integrated by the distance measurement processing unit 19 as objects of the same vehicle 3 and performing fusion processing, the state of that object (e.g., position, size, type, etc.) can be estimated with high accuracy.

[0067] (2) For example, as shown in the upper part of Figure 6, even in situations where a misrecognition occurs, such as the presence of pedestrians around vehicle 3 (i.e., in the same area as vehicle 3), in this first embodiment, as shown in the lower part of Figure 6, the data of pedestrians from the distance measuring sensor 13 is treated as data indicating vehicle 3 and processed, making it possible to accurately estimate the position and other conditions of vehicle 3. Note that in Figure 6, × (cross) indicates instantaneous data, and ○ (circle) and □ (square) indicate data that takes past data into account.

[0068] [1-6. Correspondence] Next, the relationship between this first embodiment and this disclosure will be described. The recognition system 5 corresponds to the recognition system, the distance measuring sensor 13 corresponds to the distance measuring sensor, the image sensor 15 corresponds to the image sensor, the information integration unit 17 corresponds to the information integration unit, the distance measuring processing unit 19 corresponds to the distance measuring processing unit, and the target state estimation unit 21 corresponds to the target state estimation unit.

[0069] [2. Second Embodiment] Since the basic configuration of the second embodiment is the same as that of the first embodiment, the differences from the first embodiment will be described below. Reference numerals that are the same as those in the first embodiment indicate the same components, and refer to the preceding description.

[0070] In this second embodiment, as shown in Figure 2, the distance measuring sensor 13 is configured to estimate the boundary of the target in addition to estimating the position of the target. The distance measuring processing unit 19 is configured to integrate the boundaries of multiple targets estimated by the distance measuring sensor 13. Furthermore, the target state estimation unit 21 is configured to estimate the size of the target using the boundary of the target integrated by the distance measuring processing unit 19.

[0071] The control process of this second embodiment will be described below. As shown in the flowchart in Figure 7, in S200, target information is received from the distance sensor 13 and the image sensor 15.

[0072] In the following step S210, it is determined whether there are multiple targets (for example, human targets) estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target (i.e., vehicle 3) estimated by the image sensor 15. If the determination is positive, the process proceeds to S220; if the determination is negative, the process proceeds to S230.

[0073] In S220, since there are multiple targets estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target estimated by the image sensor 15, these multiple targets are integrated (i.e., the information of the multiple targets is merged). Then, based on the integrated targets (i.e., based on the information of the integrated targets), the position of the targets, the boundaries of the targets (and therefore the size of the targets), etc., are estimated, and the process proceeds to S230.

[0074] In other words, as shown in Figure 4, information from multiple targets can be treated as information from a single target (i.e., as data for the large rectangle MK), and the boundary of the target can be determined using this data for the large rectangle MK (for example, the large rectangle MK can be used as the boundary of the target). Furthermore, the centroid (i.e., center) of this large rectangle MK can be used as the position of the target (i.e., vehicle 3), and the relative velocity of the target can also be determined.

[0075] In S230, a well-known integrated recognition process is performed, and the current process is terminated. This integrated recognition process is a so-called fusion process, in which, based on the information obtained by the image sensor 15 and the information obtained by the distance measuring sensor 13, for example, more accurate information is selected to estimate the state of the target with high accuracy.

[0076] This second embodiment provides the same effects as the first embodiment. Furthermore, this second embodiment has the advantage of high accuracy in estimating the state of the target (e.g., size).

[0077] [3. Third Embodiment] Since the basic configuration of the third embodiment is the same as that of the first embodiment, the differences from the first embodiment will be described below. Reference numerals that are the same as those in the first embodiment indicate the same components, and refer to the preceding description.

[0078] In this third embodiment, as shown in Figure 8, the recognition system 5 functionally comprises a distance measuring sensor 13, an image sensor 15, and an information integration unit 17, similar to the first embodiment, and the information integration unit 17 comprises a distance measuring processing unit 19 and a target state estimation unit 21.

[0079] In particular, in this third embodiment, the distance measurement processing unit 19 is configured to integrate multiple targets whose positions have been estimated by the distance measurement sensor 13 within a predetermined range from the boundary of a target (for example, the vehicle 3), based on information indicating the boundary of the target (for example, the vehicle 3) among the target state estimates by the target state estimation unit 21, by considering the multiple targets as the same target (for example, the vehicle 3).

[0080] The control process of this third embodiment will be described below. As shown in the flowchart in Figure 9, in S300, target information is received from the distance sensor 13 and the image sensor 15.

[0081] In the following step S310, it is determined whether there are multiple targets (for example, human targets) estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target (i.e., vehicle 3) previously estimated by the target state estimation unit 21. If the determination is positive, the process proceeds to S320; if the determination is negative, the process proceeds to S330.

[0082] In S320, since there are multiple targets estimated by the distance measuring sensor 13 within a predetermined range from the target boundary estimated by the target state estimation unit 21 in the previous step, these multiple targets are merged. Then, based on the information of the merged targets, the position and boundary of the targets are estimated, and the process proceeds to S330.

[0083] In S330, the well-known integrated recognition process is performed, and the current process is terminated. This integrated recognition process is the fusion process described above. This third embodiment provides the same effects as the first embodiment.

[0084] Furthermore, in this third embodiment, the distance measurement processing unit 19 estimates the state of the target (position, relative velocity, etc.) based on information indicating the boundary of the target estimated by the target state estimation unit 21. Therefore, for example, even if information from the image sensor 15 cannot be obtained during the initial detection, the state of the target can be estimated with high accuracy.

[0085] [4. Fourth Embodiment] Since the basic configuration of the fourth embodiment is the same as that of the first embodiment, the differences from the first embodiment will be described below. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0086] In this fourth embodiment, as shown in Figure 10, the recognition system 5 functionally comprises a distance measuring sensor 13, an image sensor 15, and an information integration unit 17, similar to the first embodiment, and the information integration unit 17 comprises a distance measuring processing unit 19 and a target state estimation unit 21.

[0087] In particular, in this fourth embodiment, the distance measurement processing unit 19 is configured to consider multiple targets whose positions have been estimated by the distance measurement sensor 13 within a predetermined range from the boundary of a target, based on information indicating the boundary of the target among the target state estimated by the target state estimation unit 21, and to integrate the multiple targets by considering them as the same target. Moreover, the distance measurement processing unit 19 is configured to consider multiple targets whose positions have been estimated by the distance measurement sensor 13 within a predetermined range from the boundary of a target estimated by the image sensor 15, and to integrate the multiple targets by considering them as the same target.

[0088] The control process of this fourth embodiment will now be described. As shown in the flowchart in Figure 11, in S400, target information is received from the distance sensor 13 and the image sensor 15.

[0089] In the following step S410, it is determined whether there are multiple targets (for example, human targets) estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target (i.e., vehicle 3) previously estimated by the target state estimation unit 21. If the determination is positive, the process proceeds to S420; if the determination is negative, the process proceeds to S430.

[0090] In S420, since there are multiple targets estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target previously estimated by the target state estimation unit 21, these multiple targets are merged. Then, based on the information of the merged targets, the position and boundary of the targets are estimated, and the process proceeds to S430.

[0091] In S430, it is determined whether there are multiple targets (for example, human targets) estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target (i.e., vehicle 3) estimated by the image sensor 15. If the determination is positive, the process proceeds to S440; if the determination is negative, the process proceeds to S460.

[0092] In S440, it is determined whether the multiple targets estimated by the distance measuring sensor 13 have not yet been integrated. If the determination is positive, the process proceeds to S450; if the determination is negative, the process proceeds to S460.

[0093] In S450, since the multiple targets estimated by the distance measuring sensor 13 have not yet been integrated, the multiple targets are integrated, and the process proceeds to S460. In S460, the well-known integrated recognition process is performed, and then this process is temporarily terminated. This integrated recognition process is the fusion process described above.

[0094] This fourth embodiment provides the same effects as the first embodiment. Furthermore, in this fourth embodiment, based on the information indicating the boundary of the target estimated by the target state estimation unit 21, if there are multiple targets whose positions have been estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target, these multiple targets are considered to be the same target and are integrated. Subsequently, if there are multiple targets whose positions have been estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target estimated by the image sensor 15, these multiple targets are considered to be the same target and are integrated.

[0095] Therefore, since the target state can be estimated not only based on the past state estimated by the target state estimation unit 21 in the previous instance, but also based on the information obtained from the image sensor 15 in the current instance, there is an advantage in that the target state can be estimated early and with high accuracy.

[0096] [5. Fifth Embodiment] Since the basic configuration of the fifth embodiment is the same as that of the first embodiment, the differences from the first embodiment will be described below. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0097] In this fifth embodiment, as shown in Figure 2 or Figure 8, the distance measuring sensor 13 is configured to estimate the velocity of an object in addition to estimating the position of the object, and to transmit the velocity information of the object to the distance measuring processing unit 19. Furthermore, when integrating multiple objects, the distance measuring processing unit 19 is configured to determine which object to integrate based on the velocity information.

[0098] The control process of this fifth embodiment will now be described. As shown in the flowchart in Figure 12, in S500, target information is received from the distance sensor 13 and the image sensor 15.

[0099] In the subsequent S510, it is determined whether there are multiple targets (for example, human targets) estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target (i.e., vehicle 3) estimated by the image sensor 15 (or target state estimation unit 21). If the determination is positive, the process proceeds to S520; if the determination is negative, the process proceeds to S530.

[0100] In S520, since there are multiple targets estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target estimated by the image sensor 15 (or target state estimation unit 21), targets with similar velocities (for example, targets whose velocity difference is within a predetermined range) are selected as the same target from among the information of these multiple targets, and the selected targets are merged. Then, based on the information of the merged targets, the position and boundary of the targets are estimated, and the process proceeds to S530.

[0101] In S530, the well-known integrated recognition process is performed, and then this process is temporarily terminated. This integrated recognition process is the fusion process described above. This fifth embodiment provides the same effects as the first embodiment.

[0102] Furthermore, in this fifth embodiment, when integrating multiple targets, targets with similar velocities are selected based on the velocity information of each target, and these targets with similar velocities are integrated. This improves the accuracy of the targets to be integrated. As a result, there is the advantage of improved accuracy in estimating the state of the targets.

[0103] [6. Sixth Embodiment] Since the basic configuration of the sixth embodiment is the same as that of the first embodiment, the differences from the first embodiment will be described below. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0104] In this sixth embodiment, as shown in Figure 2 or Figure 8, if the distance measurement processing unit 19 identifies a person as a target within the boundary of a target that the image sensor 15 or target state estimation unit 21 has determined to be an automobile or bicycle (i.e., vehicle 3), it integrates the target information estimated by the distance measurement sensor 13 that corresponds to the person identified as a target located within the boundary of the target.

[0105] The control process of this sixth embodiment will now be described. As shown in the flowchart in Figure 13, in S600, target information is received from the distance sensor 13 and the image sensor 15.

[0106] In the subsequent S610, it is determined whether there are multiple targets indicating a person, as estimated by the distance measuring sensor 13, within a predetermined range from the boundary of the target (i.e., vehicle 3) estimated by the image sensor 15 or the target state estimation unit 21. If the determination is positive, the process proceeds to S620; if the determination is negative, the process proceeds to S630.

[0107] In S620, since there are multiple human targets estimated by the distance measuring sensor 13 within a predetermined range from the boundary of the target estimated by the image sensor 15 or the target state estimation unit 21, these multiple targets are considered to be targets corresponding to the same vehicle and are merged.

[0108] In S630, the well-known integrated recognition process is performed, and then this process is temporarily terminated. This integrated recognition process is the fusion process described above. This sixth embodiment provides the same effects as the first embodiment.

[0109] [7. Seventh Embodiment] Since the seventh embodiment has the same basic configuration as the first embodiment, the following will mainly describe the differences from the first embodiment. Note that the same reference numerals as in the first embodiment indicate the same components, and refer to the preceding description.

[0110] In this seventh embodiment, as shown in Figure 2 or Figure 8, if the distance measurement processing unit 19 identifies a person as a target within the boundary of a target that the target state estimation unit 21 has determined to be an automobile or bicycle (i.e., vehicle 3), for example, using the image sensor 15, it suppresses the use of information about the person estimated by the image sensor 15 that is present within the boundary of the target. For example, it deletes the information about the person estimated by the image sensor 15.

[0111] The control process of this seventh embodiment will now be described. As shown in the flowchart in Figure 14, S700 receives target information from the distance sensor 13 and the image sensor 15.

[0112] In the subsequent step S710, the system determines whether or not there is a target indicating a person, as estimated by the image sensor 15, within a predetermined range from the boundary of the target (i.e., vehicle 3) estimated by the target state estimation unit 21. If the determination is positive, the system proceeds to S720; if the determination is negative, the system proceeds to S730.

[0113] In S720, the target state estimation unit 21 estimates that there is a human target estimated by the image sensor 15 within a predetermined range from the target boundary, so the information of that target is deleted. In S730, the well-known integrated recognition process is performed, and then this process is terminated. This integrated recognition process is the fusion process described above.

[0114] This seventh embodiment provides the same effects as the first embodiment. This seventh embodiment has the advantage of improving the accuracy of target estimation because it removes information that has been mistakenly identified as an unnecessary person. In addition to deletion, the use of such information may also be suppressed by reducing the weight of the information about people estimated by the image sensor 15.

[0115] [8. Other Embodiments] While embodiments of this disclosure have been described above, it goes without saying that this disclosure is not limited to the embodiments described above and can take various forms.

[0116] (8a) In the above embodiment, an example was given in which the recognition system is mounted on a roadside device, but the recognition system may also be mounted on various devices placed in various locations, not limited to the roadside. Furthermore, the recognition system may be mounted on a vehicle such as an automobile.

[0117] In this case, the vehicle is equipped with a rangefinder and an imaging device, and based on the signals obtained using these devices, it is possible to estimate the status of other vehicles and objects such as people in its vicinity (for example, their position, relative speed, and type).

[0118] (8b) In the above embodiment, the position of the target was shown in a planar manner using a bird's-eye view, but the position of the target, relative velocity, etc. may be grasped in three dimensions. In this case, the three-dimensional information may be projected onto a two-dimensional plane such as the road surface and processed.

[0119] For example, if the area of ​​a target (e.g., a vehicle) is understood as a three-dimensional rectangular parallelepiped, and multiple targets (e.g., people) are recognized within the area of ​​that vehicle (or within a predetermined range from the vehicle's boundary), a process to integrate the multiple targets may be performed as in the embodiments described above.

[0120] Furthermore, for example, if, in an image taken from a vehicle of the area in front of the vehicle, multiple targets (e.g., people) are recognized within the area of ​​the vehicle in front (or within a predetermined range from the vehicle's boundary), the process of integrating the multiple targets may be performed as in the embodiments described above. In this case, the area of ​​the vehicle may refer to a rectangular area or the like in an image of a plane perpendicular to the road surface. (8c) In the above embodiment, LiDAR was given as an example of a ranging device, but other examples include radar and sonar. In other words, when information on multiple reflection points of reflected waves can be obtained for a search wave, a device that detects various targets based on the signals of those reflection points can be used.

[0121] (8d) The recognition systems described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program.

[0122] Alternatively, the recognition system described herein may be implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits.

[0123] Alternatively, the recognition system described herein may be implemented by one or more dedicated computers comprising a combination of a processor and memory programmed to perform one or more functions and a processor comprising one or more hardware logic circuits.

[0124] Furthermore, computer programs may be stored on a computer-readable, non-transitional tangible recording medium as instructions executed by a computer. The methods for realizing the functions of each part included in the information notification system do not necessarily need to include software; all of its functions may be realized using one or more hardware components.

[0125] (8e) In addition to the recognition system described above, the disclosure can also be implemented in various forms, such as a system that comprises the recognition system, a program for making the computer of the information notification system function, a non-transition tangible recording medium such as a semiconductor memory on which the program is recorded, and a control method for the recognition system (e.g., a recognition method).

[0126] (8f) Multiple functions of one component in each of the above embodiments may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Furthermore, some parts of the configuration of each of the above embodiments may be omitted. Furthermore, at least some parts of the configuration of each of the above embodiments may be added to or replaced with the configuration of other embodiments. [Explanation of Symbols]

[0127] 1...Roadside device, 3...Vehicle, 5...Recognition system, 13...Distance sensor, 15...Image sensor, 17...Information integration unit, 19...Distance measurement processing unit, 21...Target state estimation unit

Claims

1. A recognition system (5) that estimates the state of objects in its surroundings, A distance measuring sensor (13) is configured to obtain point cloud information corresponding to the target based on the signal of the reflected wave from the exploration wave that explores the surrounding area, and to estimate the position of the target, among the position and type of the target, from the point cloud information, An image sensor (15) is configured to estimate the position of the target and at least the type of the target, based on the signal of the image captured of the surrounding area, and to estimate the boundary of the target. An information integration unit (17) is configured to perform processing to estimate the state of the target based on the information estimated by the distance measuring sensor and the information estimated by the image sensor, Equipped with, The aforementioned information integration unit, A distance measurement processing unit (19) configured to process information estimated by the distance measurement sensor, A target state estimation unit (21) is configured to estimate the state of the target based on the information processed by the distance measurement processing unit and the information estimated by the image sensor, It has, The distance measurement processing unit, When there are multiple targets whose positions are estimated by the distance measuring sensor within a predetermined range from the boundary of the target estimated by the image sensor, the system is configured to treat the multiple targets as the same target and integrate them. Recognition system.

2. A recognition system (5) that estimates the state of objects in its surroundings, A distance measuring sensor (13) is configured to obtain point cloud information corresponding to the target based on the signal of the reflected wave from the exploration wave that explores the surrounding area, and to estimate the position of the target, among the position and type of the target, from the point cloud information, An image sensor (15) is configured to estimate the position of the target and at least the type of the target, based on the signal of the image captured of the surrounding area, and to estimate the boundary of the target. An information integration unit (17) is configured to perform processing to estimate the state of the target based on the information estimated by the distance measuring sensor and the information estimated by the image sensor, Equipped with, The aforementioned information integration unit, A distance measurement processing unit (19) configured to process information estimated by the distance measurement sensor, A target state estimation unit (21) is configured to estimate the state of the target based on the information processed by the distance measurement processing unit and the information estimated by the image sensor, It has, The distance measurement processing unit, Based on the information indicating the boundary of the target, which is estimated by the target state estimation unit, if there are multiple targets whose positions have been estimated by the distance measuring sensor within a predetermined range from the boundary of the target, the multiple targets are treated as the same target and integrated. Recognition system.

3. A recognition system according to claim 1 or claim 2, The distance measuring sensor is configured to estimate the boundary of the target in addition to estimating the position of the target. The distance measurement processing unit is configured to integrate the boundaries of the multiple targets estimated by the distance measurement sensor, The target state estimation unit is configured to estimate the size of the target using the target boundary integrated by the distance measurement processing unit. Recognition system.

4. The recognition system according to claim 2, The distance measurement processing unit, Based on the information indicating the boundary of the target, which is estimated by the target state estimation unit, if there are multiple targets whose positions have been estimated by the distance measuring sensor within a predetermined range from the boundary of the target, the multiple targets are treated as the same target and integrated. When there are multiple targets whose positions are estimated by the distance measuring sensor within a predetermined range from the boundary of the target estimated by the image sensor, the system is configured to treat the multiple targets as the same target and integrate them. Recognition system.

5. A recognition system according to claim 1 or claim 2, The distance measuring sensor is configured to estimate the velocity of the target in addition to estimating the position of the target, and to transmit the velocity information of the target to the distance measuring processing unit. The distance measurement processing unit, When integrating the aforementioned multiple targets, the system is configured to determine which target to integrate based on the speed information. Recognition system.

6. A recognition system according to claim 1 or claim 2, The distance measurement processing unit, If the image sensor or the target state estimation unit determines that a person is a car or bicycle, and estimates the presence of a person as the target within the boundary of the target, A system configured to integrate the targets estimated by the distance measuring sensor, which are located within the boundary of the aforementioned targets. Recognition system.

7. A recognition system according to claim 1 or claim 2, The distance measurement processing unit, If the target state estimation unit determines that a person is a car or bicycle, and then estimates the presence of a person as the target within the boundary of that target, The system is configured to suppress the use of information about the person estimated by the image sensor that is located within the boundary of the target. Recognition system.

8. A recognition system according to claim 1 or claim 2, The distance measuring sensor is Based on the point cloud information corresponding to the target obtained by clustering the reflected wave signals, the system is configured to estimate at least the position of the target, among its position and type. Recognition system.

Images