Object recognition device, recognition system, object recognition method, and object recognition program

Abstract

In this object recognition device for recognizing, by using an observation sensor for which an observation visual field has been fixed, an object in the observation visual field, a processor is configured so as to: read, from a memory, map data (Dm) in which mapping has been performed using point cloud information obtained by observing the inside of the observation visual field in the past by using the observation sensor; acquire observation data (Ds) that has been built using the point cloud information obtained by observing the inside of the observation visual field by using the observation sensor; monitor a distribution difference (ΔD) in point cloud information between the observation data (Ds) and the map data (Dm); and output, to the map data (Dm) and as an object that has not been registered, recognition data (Dc) representing an unregistered object recognized in association with the distribution difference (ΔD).

Description

Object recognition device, recognition system, object recognition method, object recognition program Cross-reference to related applications

[0001] This application is based on Japanese Patent Application No. 2024-158516 filed in Japan on September 12, 2024, and the contents of the base application are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to a recognition technique for recognizing an object using an observation sensor.

[0003] The disclosed technique of Patent Document 1 uses a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) with an observation field set as an observation sensor to recognize an object within the observation field. Therefore, in order to improve the recognition accuracy of the object, the observation data obtained by LiDAR is matched with the latest map data.

[0004] Japanese Patent No. 7444136

[0005] In the disclosed technique of Patent Document 1, both the vehicle on which the observation sensor is mounted and the object to be recognized are moving objects. Therefore, for the purpose of ensuring high recognition accuracy, the observation data and the map data are subjected to three-dimensional map matching processing, enabling the extraction of observation points corresponding to moving objects that do not exist in the map data. However, when such a disclosed technique of Patent Document 1 is applied to object recognition using an observation sensor with a fixed observation field, there is a concern that the load of the recognition process may become excessive with respect to the required accuracy.

[0006] An object of the present disclosure is to provide an object recognition device that achieves both high recognition accuracy and low processing load using an observation sensor with a fixed observation field, and a recognition system configured to include the object recognition device. Another object of the present disclosure is to provide an object recognition method that achieves both high recognition accuracy and low processing load using an observation sensor with a fixed observation field. Still another object of the present disclosure is to provide an object recognition program that achieves both high recognition accuracy and low processing load using an observation sensor with a fixed observation field.

[0007] The following describes the technical means of solving the problem described in this disclosure.

[0008] A first aspect of this disclosure is an object recognition device comprising a processor and a storage medium for recognizing objects within an observation field using an observation sensor with a fixed observation field, wherein the processor is configured to read map data from the storage medium, in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquire observation data constructed from point cloud information observed within the observation field by the observation sensor; monitor the difference in distribution between the point cloud information of the observation data and the map data; and output recognition data representing an unregistered object recognized in correspondence with the difference in distribution as an object not registered in the map data.

[0009] A recognition system according to a second aspect of this disclosure comprises an observation sensor with a fixed observation field of view, and an object recognition device according to a first aspect, which includes a processor and a storage medium for recognizing objects using the observation sensor.

[0010] A third aspect of this disclosure is an object recognition method performed jointly by a processor and a storage medium for recognizing an object within an observation field using an observation sensor with a fixed observation field, comprising: reading map data from the storage medium in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquiring observation data constructed from point cloud information observed within the observation field by the observation sensor; monitoring the difference in distribution between the point cloud information of the observation data and the map data; and outputting recognition data representing an unregistered object recognized in correspondence with the distribution difference as an object not registered in the map data.

[0011] A fourth aspect of this disclosure is an object recognition program stored in a storage medium for recognizing objects within an observation field using an observation sensor with a fixed observation field, and including instructions for causing a processor to perform object recognition, the program including instructions for: reading map data from the storage medium in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquiring observation data constructed from point cloud information observed within the observation field by the observation sensor; monitoring the difference in distribution between the point cloud information of the observation data and the map data; and outputting recognition data representing an unregistered object recognized in correspondence with the distribution difference as an object not registered in the map data.

[0012] In these first to fourth embodiments, observation data constructed from point cloud information observed by an observation sensor within a fixed observation field of view is acquired. Then, by reading map data, which maps the fixed observation field of view using point cloud information previously observed by the observation sensor, from the storage medium, the difference in the distribution of point cloud information between the observation data and the map data can be monitored with a low processing load. As a result, in the process from monitoring the distribution difference to recognizing the corresponding unregistered object in the observation data and outputting the recognition data, it is possible to ensure high recognition accuracy while achieving such a low processing load.

[0013] This is a block diagram showing the physical configuration of the recognition system according to the first embodiment. This is a block diagram showing the functional configuration of the recognition system according to the first embodiment. This is a schematic diagram showing the field of view of the recognition system according to the first embodiment. This is a schematic diagram showing the field of view of the recognition system according to the first embodiment. This is a flowchart showing the recognition flow according to the first embodiment. This is a schematic diagram for explaining flowchart showing the recognition flow according to the second embodiment. This is a block diagram showing the functional configuration of the recognition system according to the third embodiment. This is a flowchart showing the recognition flow according to the third embodiment. This is a flowchart showing the update subroutine according to the third embodiment. This is a flowchart showing the recognition flow according to a modified example combining the second and third embodiments.

[0014] Hereinafter, several embodiments of this disclosure will be described with reference to the drawings. In each embodiment, the same reference numerals will be used for corresponding components, and redundant explanations may be omitted. Furthermore, if only a part of the configuration is described in each embodiment, the configuration of other embodiments described earlier may be applied to the other parts of that configuration. Moreover, not only the combinations of configurations explicitly stated in the description of each embodiment, but also the configurations of multiple embodiments can be partially combined even if not explicitly stated, as long as there are no particular problems with the combination.

[0015] (First Embodiment) As shown in Figures 1 and 2, the recognition system 1 of the first embodiment is configured to include an observation sensor 2 and an object recognition device 3. As shown in Figures 3 and 4, the recognition system 1 is a system for recognizing an object 4 using a fixed observation sensor 2 with an optically scanned observation field Vs. The observation sensor 2 is installed on a static structure such as a building, road structure, or infrastructure equipment. In particular, the observation sensor 2 is fixed in position on a static structure attached to the road, i.e., the road, in the travel path 10 on which the dynamic object 4D to be recognized, such as a vehicle, travels.

[0016] Observation sensor 2 generates point cloud information by observing the observation field Vs in three dimensions. Observation sensor 2 is either a LiDAR capable of optical scanning within the observation field Vs, or a radar sensor capable of radio wave scanning within the observation field Vs. The observation sensor 2 described below is a beam scanner type LiDAR, but other types of LiDARs, such as flash type LiDARs, or radar sensors are also acceptable.

[0017] As shown in Figures 1 and 2, the observation sensor 2 comprises a light-emitting unit 21, a scanning unit 22, and a light-receiving unit 23. The light-emitting unit 21 is mainly composed of multiple light-emitting elements that emit directional laser light in the infrared region, such as laser diodes. The light-emitting unit 21 generates a pulsed beam of illumination light directed into the observation field of view Vs through the intermittent emission of each light-emitting element.

[0018] The scanning unit 22 mainly consists of a scanning mirror 220 and a scanning drive source 221. The scanning mirror 220 is formed in the shape of a rotating plate with a reflective film deposited on one side of a base material and is pivotally supported so as to be able to swing and rotate. The scanning mirror 220 reflects the light emitted from the light-emitting unit 21 and projects the emitted light into the observation field of view Vs through the light-transmitting window of the observation sensor 2. As a result, as shown in Figures 3 and 4, the scanning direction Rs corresponding to the rotation angle of the scanning mirror 220 is scanned temporally and spatially by the emitted light within the observation field of view Vs.

[0019] Specifically, in the observation sensor 2, the orientation of the sensor 2 is fixed such that the central optical axis (hereinafter simply referred to as the optical axis) As, which is assumed to lie on the bisectors of the horizontal and vertical fields of view in the observation field of view Vs, is inclined at an angle less than perpendicular to the road surface 100 of the road 10. In particular, the observation sensor 2 is positioned such that the horizontal field of view of the observation field of view Vs is defined by the rotational change of the scanning direction Rs due to the rotational drive of the scanning mirror 220. Alternatively, the observation sensor 2 may be positioned such that the vertical field of view of the observation field of view Vs is defined by the rotational change of the scanning direction Rs due to the rotational drive of the scanning mirror 220.

[0020] In this observation sensor 2, as shown in Figure 4, the observation field Vs for one observation cycle is determined by the change in the forward direction of the rotation angle of the scanning mirror 220 from the starting direction Rss to the ending direction Rse. During the period in one observation cycle when the rotation angle of the scanning mirror 220 changes in the reverse direction from the ending direction Rse to the starting direction Rss, the generation of illumination light by the light-emitting unit 21 is interrupted, and observation within the observation field Vs by scanning is effectively stopped.

[0021] As shown in Figures 3 and 4, the observation target space in which the observation field of view Vs is set is assumed to have a spatial coordinate system [X, Y, Z], also generally called the world coordinate system, represented by a three-dimensional orthogonal coordinate system. In particular, in the spatial coordinate system [X, Y, Z] assumed in the object recognition device 3, the XY plane is approximately defined on the road surface 100 of the road 10, which can be considered a static object 4S among the objects 4 present in the observation field of view Vs. Furthermore, in this spatial coordinate system [X, Y, Z], the zero point coordinates of the observation sensor 2 are set to be offset in the direction of the Z axis from the origin coordinate on the Y axis. As a result, for the observation sensor 2, whose optical axis As is inclined with respect to the XY plane and XZ plane of the spatial coordinate system [X, Y, Z], a sensor coordinate system [x, y, z] with respect to the optical axis As is assumed to be on the y axis, represented by the three-dimensional orthogonal coordinate system shown in Figure 3.

[0022] The scanning drive source 221 shown in Figures 1 and 2 is mainly composed of an electric motor capable of oscillating and rotating the scanning mirror 220. The scanning drive source 221 may also be equipped with a reduction gear capable of amplifying the output of the electric motor and transmitting it to the scanning mirror 220. The scanning drive source 221 drives the scanning mirror 220 to oscillate and rotate according to the control from the object recognition device 3 within a driving angle range corresponding to the distance between the starting azimuth Rss and ending azimuth Rse of the observation field of view Vs shown in Figure 4.

[0023] The light-receiving unit 23 shown in Figures 1 and 2 is composed of a combination of an integrated circuit and a highly sensitive light-receiving element in which multiple light-receiving pixels are constructed using SPAD (Single Photon Avalanche Diode). The light-receiving unit 23 receives reflected light from the light emitted from the object 4 located at the scanning direction Rs within the observation field of view Vs, after re-reflection according to the rotation angle of the scanning mirror 220, and receives the light for each light-receiving pixel constituting the light-receiving element. In this way, the light-receiving unit 23 generates a light-receiving signal for each light-receiving pixel at the scanning direction Rs for each set angular period repeated within one observation cycle, corresponding to the observation position of the object 4 (i.e., the scanning position in LiDAR) and the reflection intensity from the object 4.

[0024] Therefore, in one observation cycle, the light receiving unit 23 generates and outputs observation data Ds representing point cloud information for each of the multiple observation points (i.e., each of the multiple scanning points) observed by scanning the object 4, based on the received signal generated at each set angular period of the scanning direction Rs across the observation field of view Vs. At this time, the point cloud information is constructed by linking at least one of the following for each observation point: position information, which represents the three-dimensional position coordinates of each observation point as the point cloud position, and intensity information, which represents the received light intensity relative to the reflection intensity from each observation point as the point cloud intensity.

[0025] Here, the positional information constituting the observation data Ds may directly represent the positional coordinates of the observation point in the spatial coordinate system [X, Y, Z] or the sensor coordinate system [x, y, z]. Alternatively, the positional information constituting the observation data Ds may represent the distance and angle of the observation point in the polar coordinate system, which can be converted to the positional coordinates of the observation point in the spatial coordinate system [X, Y, Z] or the sensor coordinate system [x, y, z].

[0026] On the other hand, the intensity information constituting the observation data Ds only needs to represent the received light intensity for each receiving pixel in relation to the reflected light from each observation point. For example, the intensity information constituting the observation data Ds may be luminance information obtained by converting the maximum received light intensity for each set angular period of the scanning direction Rs and for each receiving pixel into a two-dimensional luminance value. Furthermore, the intensity information constituting the observation data Ds may include luminance information representing the background light intensity (i.e., ambient light intensity) as the received light intensity at the moment the illumination light stops.

[0027] The object recognition device 3 shown in Figures 1 and 2 is installed on a static structure that is common to or different from the observation sensor 2. The object recognition device 3 is configured by combining a control unit 30 with a presentation unit 33 and an input unit 34. The control unit 30 is connected to the presentation unit 33 and the input unit 34 via at least one of the following: a LAN (Local Area Network) line, a wire harness, an internal bus, and a wireless communication line.

[0028] The presentation unit 33 presents necessary information to the user, who is the operator of the recognition system 1. The presentation unit 33 has, for example, a liquid crystal panel or an organic EL panel, which is capable of displaying the necessary information as an image. The presentation unit 33 may also have, for example, a speaker, which is capable of producing sound related to the image display. The input unit 34 has at least one of the following, for example, a mouse, a trackball, and a keyboard, which is capable of receiving input from the user. The presentation unit 33 and the input unit 34 may be configured to jointly provide a GUI (Graphical User Interface) to the user according to the control from the control unit 30.

[0029] As shown in Figure 1, the dedicated computer constituting the control unit 30 is equipped with at least one memory 31 and one processor 32. The memory 31 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, magnetic medium, and optical medium, which non-temporarily stores programs and data that can be read by the computer.

[0030] The memory 31 stores map data Dm in a format readable by the processor 32. The map data Dm is constructed from past observation data Ds acquired when the observation field Vs was observed by the observation sensor 2 during past observation cycles. As a result, the map data Dm is stored in the memory 31 to represent point cloud information, which is a mapping of at least one of the position information representing the position coordinates of the observation points and the intensity information representing the light intensity received from the observation points, linked to each observation point. In particular, the map data Dm includes point cloud information representing the static object 4S of the object 4. On the other hand, point cloud information representing the dynamic object 4D of the object 4 is not substantially registered in the map data Dm.

[0031] The processor 32 includes at least one type as a core, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU. The processor 32 executes multiple instructions included in the object recognition program stored in the memory 31 in order to recognize the object 4 using the observation sensor 2. As a result, the control unit 30 of the object recognition device 3 constructs multiple functional blocks for recognizing the object 4 using the observation sensor 2. As shown in Figure 2, the multiple functional blocks constructed include a read block 300, an observation block 310, a monitoring block 320, and an output block 330.

[0032] Through the combined efforts of these blocks 300, 310, 320, and 330, the object recognition method for recognizing object 4 using the observation sensor 2 is executed according to the recognition flow shown in Figure 5. This recognition flow is executed in response to a start input from the user to the input unit 34 while the recognition system 1 is running. In the recognition flow, each "S" represents a step executed by multiple instructions included in the object recognition program stored in memory 31.

[0033] In S10, the read block 300 reads map data Dm from the memory 31, which is a map of the observation field Vs using point cloud information previously observed by the observation sensor 2. In S20, which is executed before, after, or in parallel with S10 (Figure 5 shows an example executed after S10), the observation block 310 acquires the latest observation data Ds, which is constructed from point cloud information observed by the observation sensor 2 within the observation field Vs during at least one observation cycle. In S20, it is preferable that the latest observation data Ds be acquired by each of the observation cycles during an observation period in which multiple observation cycles occur consecutively. Furthermore, in S20, point cloud information for the observation data Ds to be used in subsequent steps may be generated by averaging at least one of the point cloud position and point cloud intensity across each of these observation cycles.

[0034] In S30, which follows S10 and S20, the monitoring block 320 monitors the distribution difference ΔD between the point cloud information between the map data Dm read out in S10 and the observation data Ds acquired in S20. At this time, the distribution difference ΔD between the observation data Ds and the map data Dm is monitored for each grid Gs obtained by dividing the observation field of view Vs into multiple grids. For the sake of easier understanding of the explanation below, the grid Gs in Figures 6 to 17 described later are abbreviated to fewer rows than actually exist (especially in Figures 6 to 9, one row in the Y-axis direction).

[0035] If the point cloud information is at least the point cloud position, the grid Gs may be defined as a three-dimensional grid with edges along each coordinate axis of the spatial coordinate system [X, Y, Z], as illustrated in Figures 6 to 9. Here, the three-dimensional grid Gs, also commonly called a voxel, may be divided into substantially equal sizes along each coordinate axis within the observation field Vs, as shown in Figure 6. The three-dimensional grid Gs may be divided into smaller sizes along each coordinate axis, i.e., finer divisions, as shown in Figure 7, the closer it is to the zero point coordinates of the observation sensor 2 (the position coordinates of the black circle labeled 2 in Figure 7) within the observation field Vs.

[0036] The three-dimensional grid Gs may be divided into smaller and smaller sections in each coordinate axis direction, as shown in Figure 8, the closer it is to the road surface 100 (see also Figures 3 and 4) within the observation field of view Vs. In this case, for grids Gs where the height from the road surface 100 in the Z axis direction of the spatial coordinate system [X, Y, Z] is in the range of, for example, 0 to 0.5 m, it is desirable to define the minimum division size.

[0037] The three-dimensional grid Gs may be divided into sizes such that the number of points in each grid Gs falls within the acceptable division range, especially when considering the map data Dm, as shown in Figure 9. In this case, the acceptable division range should be defined as a range below or less than a set threshold for the number of points, so that the monitoring accuracy for monitoring the distribution difference ΔD meets the required accuracy. Alternatively, the grid Gs may be divided so that each side is equally divided.

[0038] If the point cloud information includes at least the point cloud position, a grid Gs may be defined in a two-dimensional grid shape with edges along each coordinate axis, under the assumption of a Cartesian coordinate system [X, Y] equivalent to a system obtained by projecting the position coordinates in the Z-axis direction in a spatial coordinate system [X, Y, Z] onto the XY plane, as illustrated in Figures 10 and 11. The size of the two-dimensional grid Gs may be assumed to conform to any of Figures 6, 7, or 9, and Figures 10 and 11 in particular correspond to the size divisions conforming to Figures 6 and 9, respectively.

[0039] If the point cloud information includes at least the point cloud position, the grid Gs may be defined as a three-dimensional polar coordinate grid, as illustrated in Figure 12, under the assumption of a polar coordinate system [L, θ, ψ] defined by the distance L and angles θ, ψ with the zero coordinates of the observation sensor 2 in the sensor coordinate system [x, y, z] as the origin. In such a three-dimensional polar coordinate grid Gs, the size may be made smaller in the distance direction and each angular direction, i.e., finer, the closer the grid is to the observation sensor 2 within the observation field of view Vs.

[0040] If the point cloud information includes at least the point cloud position, as illustrated in Figure 13, the grid may be defined in the form of a two-dimensional polar coordinate grid, under the assumption of a polar coordinate system [L, θ] equivalent to a system obtained by projecting the polar coordinates in the polar coordinate system [L, θ, ψ] onto the xy plane of the spatial coordinate system [x, y, z], or the XY plane of the spatial coordinate system [X, Y, Z]. In such a two-dimensional polar coordinate grid Gs, the size may be made smaller in the distance direction and angular direction, i.e., finer, as the grid is closer to the observation sensor 2 within the observation field of view Vs.

[0041] For each data set Dm and Ds, point cloud information corresponding to the received signal of each receiving pixel is assigned to multiple data pixels Ps assumed to be in two dimensions, vertically and horizontally, at each set angular period of the scanning direction Rs. Therefore, regardless of whether the point cloud information includes point cloud position or point cloud intensity, as illustrated in Figures 14 and 15, a grid Gs may be defined as an array in which a set number of data pixels Ps in each vertical and horizontal direction of each data set Dm and Ds form a group, and these grids Gs are further arranged in each vertical and horizontal direction. In such a grid Gs that forms the group range of data pixels Ps, the set number of data pixels Ps is equivalent to the size in each vertical and horizontal direction. Therefore, the size of the grid Gs that forms the group range of data pixels Ps may be assumed to conform to any of Figures 6 to 9, and in particular, Figures 14 and 15 correspond to the size divisions conforming to Figures 6 and 8, respectively.

[0042] In S30, the monitoring block 320 monitors, as a distribution difference ΔD for each grid Gs, a Mahalanobis distance ΔDm that represents a statistical distribution correlated with point cloud information (hereinafter simply referred to as point cloud information), which is at least one of the point cloud positions and point cloud intensities that construct both data Dm and Ds. At this time, in the statistical analysis of the map data Dm, as schematically shown two-dimensionally in FIG. 16, the mean vector μ and the variance-covariance matrix Σ of the normal distribution regarding the point cloud information of the same data Dm are extracted. Therefore, in the statistical analysis of the observation data Ds, as shown in Equation 1 below, regarding the observation vector ρ composed of the point cloud information of the same data Ds, the Mahalanobis distance ΔDm representing the distribution difference ΔD from the mean vector μ of the map data Dm is extracted for each grid Gs. Note that, instead of the Mahalanobis distance ΔDm, the Euclidean distance ΔDe representing the distribution difference ΔD from the mean vector μ regarding the observation vector ρ may be extracted for each grid Gs as shown in Equation 2 below.

[0043] As shown in FIG. 5, in S40 following S30, the output block 330 determines whether or not a grid Gs whose distribution difference ΔD monitored in S30 is outside the distribution allowable range is detected from all the grids Gs within the observation field Vs. At this time, being outside the distribution allowable range may be defined as a range exceeding or equal to a determination threshold value regarding the distribution difference ΔD such that the determination accuracy of detecting an unregistered object 4U (see FIG. 4), which is an object 4 not registered in the map data Dm, in the observation data Ds satisfies the required accuracy. Therefore, the determination threshold value for determining the inner and outer boundaries of the distribution allowable range may be a common value that does not depend on the position of the grid Gs. Alternatively, the determination threshold value may be a variable value representing an increasing distribution difference ΔD for grids Gs closer to the observation sensor 2, for example.

[0044] In S40, when the point cloud information of each data Dm and Ds includes both the point cloud position and the point cloud intensity, the distribution difference ΔD regarding the point cloud position and the distribution difference ΔD regarding the point cloud intensity may be respectively executed based on individual distribution tolerance ranges. In this case, if at least one of the distribution difference ΔD regarding the point cloud position and the distribution difference ΔD regarding the point cloud intensity is the grid Gs outside the distribution tolerance range, a determination that it has been detected may be given. Alternatively, the determination that it has been detected may be given only for the grid Gs where both the distribution difference ΔD regarding the point cloud position and the distribution difference ΔD regarding the point cloud intensity are outside the distribution tolerance range.

[0045] In S40 shown in FIG. 5, when the distribution difference ΔD of all the grids Gs is within the distribution tolerance range, that is, when a negative determination is made, the current execution of the recognition flow ends. On the other hand, in S40, when a positive determination is made because a grid Gs with a minimum distribution difference ΔD outside the distribution tolerance range is detected in response to the presence of the unregistered object 4U, the recognition flow shifts to S50. At this time, without obtaining the distribution difference ΔD of all the grids Gs, the distribution difference ΔD may be calculated by narrowing down to the grids Gs close, for example, within a range of 1 m with respect to the observation data Ds.

[0046] In S50, the output block XXXX clusters the point cloud information in the observation data Ds obtained in S20 for the grid Gs in which the distribution difference ΔD outside the distribution tolerance range has been detected in S40. At this time, in the clustering process, for example, by inputting the point cloud information of each observation point into a rule-based model or a machine learning model, etc., the unregistered object 4U clustered corresponding to the distribution difference ΔD is recognized.

[0047] Note: There is an unclear "XXXX" in the original text of . It should be corrected to the correct content before translation to ensure the accuracy of the translation. Here, it is temporarily translated as "output block XXXX".In S60, which follows S50, the output block 330 performs tracking processing on the point cloud information of clusters representing unregistered objects 4U recognized in S50, using the observation data Ds acquired in S20. At this time, the tracking processing separates and recognizes the unregistered objects 4U into static objects 4S and dynamic objects 4D by inputting the point cloud information of clusters into a tracking estimation filter, such as a Kalman filter, for each observation data Ds of each observation cycle that is continuous within the observation period of S20. Here, dynamic objects 4D are defined as objects 4 whose movement speed is equal to or exceeds a speed threshold, for example, 1 km / h, while static objects 4S are defined as objects 4 whose movement speed is less than or equal to the said speed threshold.

[0048] In S70, which follows S60, the output block 330 generates and outputs recognition data Dc to represent the recognized unregistered object 4U. At this time, the recognition data Dc generated after the execution of S60 can represent the unregistered object 4U by separating it into a static object 4S and a dynamic object 4D.

[0049] The output of recognition data Dc in S70 is preferably stored in memory 31 for use after the execution of the object recognition program to utilize the recognition information of the unregistered object 4U. In addition to storage in memory 31, the output form of recognition data Dc may be at least one of the following: for example, data transmission to an external computer or external server, and display output from the presentation unit 33. Once the execution of S70 is completed, this execution of the recognition flow is finished.

[0050] (Effects) The effects of the first embodiment described above are explained below.

[0051] In the first embodiment, observation data Ds is obtained from point cloud information observed by the observation sensor 2 within a fixed observation field Vs. Then, map data Dm, which is mapped by point cloud information previously observed by the observation sensor 2 within the fixed observation field Vs, is read from memory 31, and the distribution difference ΔD between the point cloud information of the observation data Ds and the map data Dm can be monitored with a low processing load. As a result, in the process from monitoring the distribution difference ΔD to recognizing the corresponding unregistered object 4U in the observation data Ds and outputting recognition data Dc, it is possible to ensure high recognition accuracy even under such a low processing load.

[0052] According to the first embodiment, by dividing the observation field of view Vs into multiple grids Gs, the distribution difference ΔD between the observation data Ds and the map data Dm can be efficiently monitored. Therefore, monitoring by grid Gs is effective in achieving low processing load while maintaining high recognition accuracy.

[0053] According to the first embodiment, the observation field Vs may be divided into grids Gs of smaller size the closer they are to the observation sensor 2. In this case, unregistered objects 4U at close range within the observation field Vs can be easily recognized by monitoring the distribution difference ΔD due to slight differences in the point cloud distribution in the small-sized grids Gs. Therefore, recognition by monitoring each of these variable-sized grids Gs can improve accuracy while reducing the processing load across the entire observation field.

[0054] According to the first embodiment, the observation field of view Vs may be divided into grids Gs of smaller size the closer they are to the road surface 100. In this case, unregistered objects 4U on the road surface 100 within the observation field of view Vs can be easily recognized by monitoring the distribution difference ΔD due to slight differences in the point cloud distribution in the small-sized grids Gs. Therefore, recognition by monitoring each of these variable-sized grids Gs makes it possible to ensure high recognition accuracy for unregistered objects 4U, which are of high importance as targets for recognition due to concerns about potential road obstruction, while suppressing the overall processing load on the observation field of view Vs.

[0055] According to the first embodiment, the grid Gs may be divided into sizes such that the number of point clouds for each grid Gs in the map data Dm falls within the acceptable division range. In this case, since the number of point clouds within the acceptable division range suitable for monitoring the distribution difference ΔD can be assigned to each grid Gs, it becomes possible to optimize both the processing load and the recognition accuracy for each grid Gs.

[0056] (Second Embodiment) The second embodiment is a modification of the first embodiment. As shown in Figure 17, in the recognition flow of the second embodiment, S2030, which follows S10 and S20, is executed before S60 among S30 to S60. In S2030, the monitoring block 320 distributes each grid Gs within the observation field of view Vs into registered grid Gsr and unregistered grid Gsu. At this time, a registered grid Gsr is defined as a grid Gs in which point cloud information representing a static object 4S among the objects 4 is registered in the map data Dm read out by S10. On the other hand, an unregistered grid Gsu is defined as a grid Gs in which point cloud information representing a static object 4S among the objects 4 is not registered in the map data Dm read out by S10.

[0057] Therefore, for the registered grid Gsr assigned by S2030, S30, S40, and S50, which branch off from S2030 in the recognition flow of the second embodiment, are executed sequentially. On the other hand, in S50 when branching from S2030, the output block 330 skips S30 and S40 for the unregistered grid Gsu assigned by S2030 and performs clustering processing of point cloud information similar to the first embodiment.

[0058] In this second embodiment of the recognition flow, the output block 330 in S60, which follows S50, performs tracking processing on clusters representing unregistered objects 4U recognized in S50 in the observed data Ds. As a result, in S70, which follows S60, recognition data Dc is output that represents at least the former of the unregistered objects 4U recognized from the registered grid Gsr in correspondence with the distribution difference ΔD and the unregistered objects 4U recognized from the unregistered grid Gsu in correspondence with the point cloud information of the observed data Ds. The recognition data Dc output at this time can represent static objects 4S and dynamic objects 4D separately, regardless of whether the unregistered objects 4U were recognized from the registered grid Gsr or the unregistered grid Gsu.

[0059] As explained above, according to the second embodiment, for registered grid Gsr in map data Dm where object 4 is registered, unregistered objects 4U corresponding to the monitored distribution difference ΔD can be recognized with both high recognition accuracy and low processing load. On the other hand, for unregistered grid Gsu where object 4 is not registered, the point cloud information present in observation data Ds itself represents the unregistered objects 4U. Therefore, in unregistered grid Gsu, even if the monitoring of the distribution difference ΔD is skipped to reduce the processing load, the unregistered objects 4U corresponding to the point cloud information in observation data Ds can be accurately recognized. Thus, selecting processing according to the registration status of object 4 on map data Dm is effective in achieving both high recognition accuracy and low processing load.

[0060] (Third Embodiment) The third embodiment is a modification of the first embodiment. As shown in Figure 18, in the third embodiment, an update block 3340 is added to the functional block for recognizing an object 4 using the observation sensor 2.

[0061] Therefore, as shown in Figure 19, in the recognition flow of the third embodiment, S3080, which follows S70, is executed. In S3080, the update block 3340 performs an update subroutine to update the map data Dm. As shown in Figure 20, in the update subroutine's S3080, the update block 3340 determines whether or not the static object 4S was recognized as an unregistered object 4U in S60. If the result is negative, the current execution of the recognition flow ends along with the update subroutine. On the other hand, if the result is positive, the update subroutine proceeds to S3081.

[0062] In S3081, the update block 3340 determines whether the number of points in the grid Gs recognized by S60 for the static object 4S in the observation data Ds acquired in S20 is outside the update tolerance range. The update tolerance range is defined as a range below or less than a judgment threshold for the number of points in the grid Gs recognized for the static object 4S, so that the judgment accuracy for determining whether or not to update the map data Dm based on the number of points in the grid Gs for the static object 4S meets the required accuracy. Therefore, the judgment threshold that determines the boundary between the inside and outside of the update tolerance range may be a common value that does not depend on the position of the grid Gs. Alternatively, the judgment threshold may be a variable value that represents an increasing number of points for grid Gs closer to the observation sensor 2, for example.

[0063] In S3081, if the number of points in the grid Gs is deemed to be sufficiently large and within the update tolerance range, the update subroutine proceeds to S3082, and the update block 3340 performs the update process for the static object 4S recognized by the grid Gs that was deemed to be invalid. At this time, the update process is performed by registering the point cloud information representing the static object 4S in the map data Dm. After that, the map data Dm is stored in memory 31.

[0064] In S3082, the position coordinates of the point cloud information representing the static object 4S, which is not included in the map data Dm, are extracted for each observation data Ds of each observation cycle that is continuous within the observation period of S20. In this case, the static object 4S recognized as an unregistered object 4U by the tracking process in S60 will be registered for the common position coordinates among the observation data of each observation cycle as a result of the tracking process. At this time, the common position coordinates may be registered if an object is recognized as a static object 4S for a certain period of time or longer, for example, 10 seconds or more. Also, if the static object 4S is no longer recognized in the grid Gs of the common position coordinates that was registered once by the update process in S3082, it may be determined that the static object 4S has moved, and the grid Gs of the common position coordinates may be changed to unregistered. In this way, the object can be recognized even at coordinates where the object was previously placed temporarily and has since moved. With the completion of the execution of S3082, the current execution of the recognition flow ends along with the update subroutine.

[0065] On the other hand, for static objects 4S recognized in grid Gs that have been determined to have a positive result in S3081 because the number of points in the point cloud is too low to be within the update tolerance range, the update process by registering the point cloud information is skipped, and the current execution of the recognition flow ends along with the update subroutine. Therefore, if the number of points in the grid Gs recognized for static objects 4S is all outside the update tolerance range, the execution of S3082 itself is skipped.

[0066] As described above, the third embodiment updates the map data Dm by registering a static object 4S recognized as an unregistered object 4U corresponding to a distribution difference ΔD, and stores the update result in the memory 31. With this, even if the distribution difference ΔD between the updated map data Dm and the observation data Ds can be monitored with low load, the corresponding unregistered object 4U can be recognized with high accuracy. Therefore, updating the map data Dm is effective in recognizing objects 4 while achieving both high recognition accuracy and low processing load.

[0067] According to the third embodiment, the map data Dm is updated to register the recognized static object 4S to the common position coordinates of multiple observation data Ds acquired through successive observation cycles. As a result, for the latest map data Dm in which the static object 4S is registered to the common position coordinates, where the risk of misrecognition has been reduced, high recognition accuracy for the corresponding unregistered object 4U can be guaranteed even if the distribution difference ΔD between it and the observation data Ds can be monitored with low load. Therefore, updating the map data Dm using successive observation cycles makes it possible to improve the reliability of recognizing object 4 while achieving both high recognition accuracy and low processing load.

[0068] According to the third embodiment, there is a concern about the risk of misrecognition of static objects 4S, which are recognized in correspondence with the distribution difference ΔD between the observed data Ds and grid Gs, where the number of point clouds in the observed data Ds is so small that it falls outside the update tolerance range. However, for grid Gs with a small number of point clouds that falls outside the update tolerance range in the observed data Ds, the update of the map data Dm by registering point cloud information representing static objects 4S is skipped, thus making it possible to suppress the decrease in accuracy due to misrecognition. (Other embodiments) Although several embodiments have been described so far, this disclosure is not to be interpreted as being limited to those embodiments, and can be applied to various embodiments and combinations without departing from the gist of this disclosure.

[0069] In the modified embodiments of the first to third embodiments, the dedicated computer constituting the object recognition device 3 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following: ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Such a digital circuit may also have a memory that stores a program.

[0070] In the modified versions of the first to third embodiments, the latest observation data Ds may be acquired in a single observation cycle in S20. In the modified versions of the first to third embodiments to which such single-cycle data acquisition is applied, the tracking process in S60 may be performed based on sensing information from other sensors. Alternatively, in the modified version of the first embodiment to which single-cycle data acquisition is applied, recognition data Dc of an unregistered object 4U that is not separated into static object 4S and dynamic object 4D due to the skipping of S60 itself may be output in S70.

[0071] In the modified versions of the first to third embodiments, the distribution difference ΔD monitored for each grid Gs in S30 may be the difference between representative values ​​such as the mean, variance, maximum, or minimum, which represent the statistical distribution of the point cloud information for each data Dm and Ds. In the modified version of the second embodiment, as shown in Figure 21, S3080 of the third embodiment may be executed following S70. In the modified version of the third embodiment, S3080 may be executed as a separate flow by a different program from S10 to S70. Furthermore, the processing of S3081 may be skipped, and S3082 may be executed after S3080.

[0072] In addition to the embodiments described so far, the object recognition device 3 of the above-described embodiments and modifications may also be implemented in the form of a semiconductor device (for example, a semiconductor chip) that includes at least one processor 32 and one memory 31.

[0073] (Addendum) This specification discloses several technical ideas and several combinations thereof, as listed below. The symbols in parentheses in this addendum indicate the correspondence with the specific means described in the embodiments detailed above, and do not limit the technical scope of this disclosure.

[0074] (Technical Concept 1) An object recognition device comprising a processor (32) and a storage medium (31) for recognizing an object (4) within a fixed observation field (Vs) using an observation sensor (2), wherein the processor is configured to: read map data (Dm) from the storage medium, in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquire observation data (Ds) constructed from point cloud information observed within the observation field by the observation sensor; monitor the difference in distribution (ΔD) between the point cloud information of the observation data and the map data; and output recognition data (Dc) representing an unregistered object (4U) that is recognized in correspondence with the difference in distribution as an object not registered in the map data.

[0075] (Technical Concept 2) The object recognition device according to Technical Concept 1, wherein monitoring the distribution difference includes monitoring the distribution difference between the observation data and the map data for each grid (Gs) obtained by dividing the observation field into multiple parts.

[0076] (Technical Concept 3) An object recognition device according to Technical Concept 2, wherein the observation field of view is divided into grids of smaller size the closer the grid is to the observation sensor.

[0077] (Technical idea 4) An object recognition device according to technical idea 2, wherein the observation field of view is divided into grids of smaller size the closer the grid is to the road surface (100).

[0078] (Technical idea 5) An object recognition device according to technical idea 2, wherein the grid in the map data is divided into sizes such that the number of points for each grid falls within the allowable division range.

[0079] (Technical Idea 6) The object recognition device according to any one of Technical Ideas 2 to 5, wherein monitoring the distribution difference includes monitoring the distribution difference with respect to a registered grid (Gsr) which is the grid in which the object is registered in the map data, and skipping monitoring the distribution difference with respect to an unregistered grid (Gsu) which is the grid in which the object is not registered, and outputting the recognition data includes outputting the recognition data such that it represents the unregistered object recognized from the registered grid in correspondence with the distribution difference, and the unregistered object recognized from the unregistered grid in correspondence with the point cloud information of the observation data.

[0080] (Technical Idea 7) An object recognition device according to any one of Technical Ideas 1 to 6, wherein the processor is further configured to update the map data in the storage medium so as to register static objects (4S) recognized as unregistered objects corresponding to the distribution difference.

[0081] (Technical Concept 8) The object recognition device according to Technical Concept 7, wherein updating the map data includes updating the map data to register the recognized static object with respect to the common position coordinates of a plurality of observation data acquired by successive observation cycles.

[0082] (Technical Idea 9) The object recognition device according to Technical Idea 7 or 8, wherein monitoring the distribution difference includes monitoring the distribution difference between the observation data and the map data for each grid (Gs) obtained by dividing the observation field into multiple parts, and updating the map data includes skipping the update of the map data by registration with respect to static objects recognized in the grid where the number of point clouds in the observation data is so small that it falls outside the update tolerance range, in accordance with the distribution difference between the map data and the observation data.

[0083] (Technical Concept 10) A recognition system comprising an observation sensor (2) with a fixed observation field of view (Vs), and an object recognition device (3) according to any one of Technical Concepts 1 to 9, which includes a processor (32) and a storage medium (31) for recognizing an object (4) using the observation sensor.

[0084] Furthermore, the technical concepts 1 to 9 described above may also be understood within the respective technical concepts of the methods and programs.

Claims

1. An object recognition device comprising a processor (32) and a storage medium (31) for recognizing an object (4) within a fixed observation field (Vs) using an observation sensor (2), wherein the processor is configured to: read map data (Dm) from the storage medium, in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquire observation data (Ds) constructed from point cloud information observed within the observation field by the observation sensor; monitor the distribution difference (ΔD) between the point cloud information of the observation data and the map data; and output recognition data (Dc) representing an unregistered object (4U) recognized in correspondence with the distribution difference as an object not registered in the map data.

2. The object recognition device according to claim 1, wherein monitoring the distribution difference includes monitoring the distribution difference between the observation data and the map data for each grid (Gs) obtained by dividing the observation field into multiple parts.

3. The object recognition device according to claim 2, wherein the observation field of view is divided into grids of smaller size the closer the grid is to the observation sensor.

4. The object recognition device according to claim 2, wherein the observation field of view is divided into grids of smaller size the closer the grid is to the road surface (100).

5. The object recognition device according to claim 2, wherein the grid in the map data is divided into sizes such that the number of points for each grid falls within the allowable division range.

6. Monitoring the distribution difference includes monitoring the distribution difference with respect to a registered grid (Gsr) which is the grid in which the object is registered in the map data, and skipping monitoring the distribution difference with respect to an unregistered grid (Gsu) which is the grid in which the object is not registered, and outputting the recognition data includes outputting the recognition data such that it represents the unregistered object recognized from the registered grid in correspondence with the distribution difference, and the unregistered object recognized from the unregistered grid in correspondence with the point cloud information of the observation data.

7. The object recognition device according to claim 1, wherein the processor is further configured to update the map data in the storage medium so as to register static objects (4S) recognized as unregistered objects corresponding to the distribution difference.

8. The object recognition device according to claim 7, wherein updating the map data includes updating the map data to register the recognized static object with respect to the common position coordinates of a plurality of observation data acquired by successive observation cycles.

9. The object recognition device according to claim 7, wherein monitoring the distribution difference includes monitoring the distribution difference between the observation data and the map data for each grid (Gs) obtained by dividing the observation field into multiple parts, and updating the map data includes skipping the update of the map data by registration with respect to static objects recognized in the grid where the number of point clouds in the observation data is so small that it falls outside the update tolerance range, in accordance with the distribution difference between the observation data and the map data.

10. A recognition system comprising: an observation sensor (2) with a fixed observation field of view (Vs); and an object recognition device (3) according to any one of claims 1 to 9, which includes a processor (32) and a storage medium (31) for recognizing an object (4) using the observation sensor.

11. An object recognition method performed jointly by a processor (32) and a storage medium (31) to recognize an object (4) within a fixed observation field (Vs) using an observation sensor (2) with a fixed observation field, comprising: reading map data (Dm) from the storage medium, in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquiring observation data (Ds) constructed from point cloud information observed within the observation field by the observation sensor; monitoring the distribution difference (ΔD) between the point cloud information of the observation data and the map data; and outputting recognition data (Dc) representing an unregistered object (4U) recognized in correspondence with the distribution difference as an object not registered in the map data.

12. An object recognition program that includes instructions for a processor (32) to perform object recognition, which is stored in a storage medium (31) for recognizing an object (4) within a fixed observation field (Vs) using an observation sensor (2) with a fixed observation field (Vs), the program comprising instructions for reading map data (Dm) from the storage medium, in which the observation field is mapped by point cloud information previously observed by the observation sensor; acquiring observation data (Ds) constructed from point cloud information observed within the observation field by the observation sensor; monitoring the distribution difference (ΔD) between the point cloud information of the observation data and the map data; and outputting recognition data (Dc) representing an unregistered object (4U) that is recognized in correspondence with the distribution difference as an object not registered in the map data.

Images