Object recognition system and object recognition method

Abstract

The present invention relates to an object recognition system and an object recognition method. A data processing device performs abnormality determination processing, i) in the abnormality determination processing, predicting a relative movement distance indicating a relative movement distance of a tracking object relative to a moving body within a range of a data set for representing point group data from the current time to a predetermined time, ii) the relative movement distance includes a vertical distance and a horizontal distance, iii) when the vertical distance or the horizontal distance is greater than a predetermined distance, calculating an estimated intensity of reflected light, iv) when it is determined that the difference between the estimated intensity of reflected light and the actual intensity of reflected light exceeds a predetermined intensity, determining that an abnormality has occurred in a predetermined partition.

Description

Technical Field

[0001] The present invention relates to a system and method for identifying an object using a laser radar (LIDAR: Laser Imaging Detection And Ranging). Background Art

[0002] International Publication No. 2020 / 170679 discloses a sensor system, which includes: a housing for a vehicle lamp; a laser radar housed in a space formed inside the housing; a lamp cleaner to remove dirt attached to the housing; and a control unit to control the driving of the cleaner for the vehicle lamp. In this previous system, the vehicle lamps are arranged at the four corners of the vehicle. The laser radar outputs laser pulses around the vehicle to detect the reflected light of the laser pulse reflected at the reflection point. The control unit determines whether there is dirt such as rain, snow, mud, etc. attached to the housing.

[0003] When dirt is attached to the outer cover, the intensity of the reflected light detected by the laser radar decreases. In the determination by the control unit, the intensity of the reflected light of the laser pulse reflected by the dry road surface around the vehicle is obtained. Then, when the intensity of the reflected light is below a predetermined threshold, it is determined that dirt is attached to the outer cover. When it is determined that dirt is attached to the outer cover, the control unit drives the cleaner.

[0004] However, in the conventional system using reflected light reflected from a dry road surface, only dirt attached to the lower part of the housing can be detected. That is, in the conventional system, dirt attached to other than the lower part of the housing cannot be detected. The range of the laser pulse output from the laser radar includes the up, down, left, and right directions of the vehicle. Therefore, in order to suppress the decrease in the detection accuracy of the object detected by the laser radar, it is desirable to develop a technology for detecting dirt attached to the housing in the range through which the laser pulse and the reflected light pass. Summary of the invention

[0005] An object of the present invention is to provide a technology capable of detecting an abnormality occurring in a range through which a laser pulse from a main body of a laser radar passes in a cover portion covering the main body of the laser radar.

[0006] The first invention is an object recognition system for recognizing objects around a moving body, and has the following features.

[0007] The object recognition system includes a main body of a laser radar, a cover, and a data processing device. The main body outputs laser pulses around the moving object, and acquires data of the reflection point by detecting reflected light of the laser pulse reflected at the reflection point around the moving object. The cover protects the main body. The data processing device performs object recognition processing based on point group data obtained by processing the data of the reflection point.

[0008] The mask surface includes a scanning range indicating a range through which the laser pulse and the reflected light pass.

[0009] The data processing device further performs an abnormality determination process for determining an abnormality in a predetermined section of the scanning range.

[0010] The data processing device predicts, based on the history of the point group data, a relative movement distance representing a distance that the tracking object identified by the object recognition processing moves relative to the moving body within a range of a data set for representing the point group data during the period from the current time to the elapse of a predetermined time in the abnormality determination process. The relative movement distance includes a vertical distance representing a component in a direction perpendicular to the output center axis of the laser pulse and parallel to the vertical direction, and a horizontal distance representing a component in a direction perpendicular to the output center axis and parallel to the horizontal direction.

[0011] In the abnormality determination process, the data processing device calculates an estimated intensity of the reflected light based on the history of the point cloud data constituting the tracking target when the vertical distance or the horizontal distance is greater than or equal to a predetermined distance. The estimated intensity is the intensity of the reflected light constituting the point cloud data of the tracking target predicted to be within the range of the data set after the predetermined time has passed and in a corresponding partition representing a partition corresponding to the predetermined partition.

[0012] The data processing device determines, in the abnormality determination process, whether the difference between the estimated intensity and the actual intensity of the reflected light exceeds a predetermined intensity, wherein the actual intensity is the intensity of the reflected light that actually constitutes the point cloud data of the tracking object in the corresponding partition within the range of the data set after the predetermined time has passed.

[0013] In the abnormality determination process, the data processing device determines that an abnormality has occurred in the predetermined section when it is determined that the difference between the estimated intensity and the actual intensity exceeds the predetermined intensity.

[0014] The second invention further has the following features in the first invention.

[0015] The data processing device further performs, in the abnormality determination process, a process of setting the corresponding section based on the history of the point group data constituting the tracking target.

[0016] In the process of setting the corresponding partition, the data processing device sets the corresponding partition as a partition inside a partition of point cloud data predicted to constitute an outer edge portion of the tracking target within the range of the data set after the predetermined time has elapsed.

[0017] The third invention further has the following features in the first invention or the second invention.

[0018] The data processing device also performs in the abnormality determination process: determining a specific partition within the range of the data set at the current moment, wherein the specific partition represents a partition predicted to be consistent with the corresponding partition within the range of the data set after the specified time has passed; and selecting arbitrary data within the specific partition and evaluating the intensity of the reflected light constituting the arbitrary data.

[0019] The data processing device calculates, in the process of evaluating the intensity of the reflected light, an average value of the intensities of the reflected light that constitute all the data in the specific partition, and determines whether the difference between the average value and the intensity of the reflected light that constitutes the arbitrary selected data exceeds a specified deviation; if it is determined that the difference between the average value and the intensity of the reflected light that constitutes the arbitrary selected data exceeds the specified deviation, data different from the arbitrary selected data is selected.

[0020] The fourth invention is an object recognition method having the following features.

[0021] The object recognition method is performed by a data processing device.

[0022] The data processing device performs object recognition processing based on data of reflection points acquired by detecting reflected light of a laser pulse reflected at reflection points around the moving body.

[0023] The data processing device performs abnormality determination processing, wherein the abnormality determination processing determines an abnormality in a specified partition of a scanning range representing a range through which the laser pulse and the reflected light pass in a mask surface, and the mask surface protects a main body of a laser radar that outputs the laser pulse, detects the reflected light, and obtains data on the reflection point.

[0024] The data processing device predicts, based on the history of the point group data, a relative movement distance representing a distance that the tracking object identified by the object recognition processing moves relative to the moving body within a range of a data set for representing the point group data during the period from the current time to the elapse of a predetermined time in the abnormality determination process. The relative movement distance includes a vertical distance representing a component in a direction perpendicular to the output center axis of the laser pulse and parallel to the vertical direction, and a horizontal distance representing a component in a direction perpendicular to the output center axis and parallel to the horizontal direction.

[0025] In the abnormality determination process, the data processing device calculates an estimated intensity of the reflected light based on the history of the point cloud data constituting the tracking target when the vertical distance or the horizontal distance is greater than or equal to a predetermined distance. The estimated intensity is the intensity of the reflected light constituting the point cloud data of the tracking target predicted to be within the range of the data set after the predetermined time has passed and in a corresponding partition representing a partition corresponding to the predetermined partition.

[0026] The data processing device determines, in the abnormality determination process, whether the difference between the estimated intensity and the actual intensity of the reflected light exceeds a predetermined intensity, wherein the actual intensity is the intensity of the reflected light that actually constitutes the point cloud data of the tracking object in the corresponding partition within the range of the data set after the predetermined time has passed.

[0027] In the abnormality determination process, the data processing device determines that an abnormality has occurred in the predetermined section when it is determined that the difference between the estimated intensity and the actual intensity exceeds the predetermined intensity.

[0028] According to the first invention or the fourth invention, an abnormality determination process for determining an abnormality in a predetermined sub-area of ​​the scanning range is performed by a data processing device. According to the abnormality determination process, an estimated intensity of the reflected light is calculated based on the history of the point group data obtained by the data processing of the reflection point. In addition, according to the abnormality determination process, when it is determined that the difference between the estimated intensity of the reflected light and the actual intensity of the reflected light exceeds the predetermined intensity, it is determined that an abnormality has occurred in the predetermined sub-area. Therefore, an abnormality occurring in the scanning range can be detected.

[0029] The partition of the point group data predicted to constitute the outer edge of the tracking object may include data of an object different from the tracking object. In this regard, according to the second invention, the corresponding partition can be set as a partition closer to the inside than the partition of the point group data predicted to constitute the outer edge of the tracking object. Therefore, the accuracy of calculating the estimated intensity of the reflected light can be improved, thereby improving the accuracy of detecting abnormalities occurring within the scanning range.

[0030] When arbitrary data in a specific partition is selected, the intensity of the reflected light constituting the selected arbitrary data may deviate significantly from the intensity of the surrounding reflected light. In this regard, according to the third invention, the average value of the intensity of the reflected light constituting all the data in the specific partition is calculated. In addition, when it is determined that the difference between the average value and the intensity of the reflected light constituting the selected arbitrary data exceeds a specified deviation, other data different from the arbitrary data is selected. Therefore, arbitrary data in a specific partition can be appropriately selected, thereby improving the accuracy of detecting abnormalities occurring within the scanning range. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Hereinafter, features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, wherein like reference numerals denote like elements, and wherein:

[0032] Figure 1 This figure illustrates an example of object recognition using a laser radar.

[0033] Figure 2 This is a schematic diagram showing an example of the configuration of a 3D laser radar.

[0034] Figure 3 The figure schematically shows the range of a data set for expressing point group data and corresponding partitions defined within the data set range as partitions corresponding to the range through which the laser pulse and the reflected light pass.

[0035] Figure 4 This is a diagram for explaining an example of a method for predicting the future movement direction and movement distance of a tracked object within a data set.

[0036] Figure 5 This is a diagram for explaining an example of a method for estimating future Z position data.

[0037] Figure 6 This is a block diagram showing a configuration example of an object recognition system according to an embodiment.

[0038] Figure 7 This is a block diagram showing an example of the functional configuration of a data processing device.

[0039] Figure 8 FIG. 1 is a diagram showing an example of the future movement direction and movement distance of the tracking target.

[0040] Fig. 9 This is a flowchart showing the flow of abnormality determination processing executed by a data processing device (processor). DETAILED DESCRIPTION

[0041] Hereinafter, an object recognition system according to an embodiment of the present invention will be described with reference to the accompanying drawings. It should be noted that the object recognition method according to the embodiment is implemented by computer processing performed in the object recognition system according to the embodiment. In addition, in each figure, the same or corresponding parts are marked with the same reference numerals, and their descriptions are simplified or omitted.

[0042] 1. Overview of implementation methods

[0043] 1-1. Object recognition using LiDAR

[0044] Figure 1 This figure illustrates an example of object recognition using laser radar. Figure 1 A moving body MB is depicted in FIG. Figure 1 The mobile body MB shown is a vehicle. However, the mobile body MB may also be a robot, a marine mobile body, or a flying body. As robots, logistics robots and work robots are given as examples. As marine mobile bodies, ships and submarines are given as examples. As flying bodies, airplanes, drones, etc. are given as examples.

[0045] The moving body MB is equipped with a 3D laser radar 10. The 3D laser radar 10 outputs laser pulses around the moving body MB. When the laser pulses are reflected at the reflection points RP around the moving body MB, the reflected light of the laser pulses returns to the 3D laser radar 10. The 3D laser radar 10 obtains data of the reflection points RP by detecting the reflected light.

[0046] exist Figure 1 The road surface RS on which the moving body MB is traveling is depicted in FIG. When the reflected light of the laser pulse reflected at the reflection point RP on the road surface RS is detected by the 3D laser radar 10, the data of the reflection point RP is obtained. Figure 1 , an object OB1 existing in front of the moving body MB is depicted. Figure 1 The object OB1 shown is a vehicle. The object OB1 may be moving or stationary. When the reflected light of the laser pulse reflected at the reflection point RP on the object OB1 is detected by the 3D laser radar 10, the data of the reflection point RP is obtained.

[0047] The data of the reflection point RP include, for example, the output angle of the laser pulse (horizontal angle θ and vertical angle ) and the data of the flight time (TOF: Time Of Flight) of the laser pulse at the output angle. When the data of the reflection point RP is processed, the point group data LID is obtained. The individual data constituting the point group data LID include 3D position data (i.e., X, Y, and Z position data) fixed in the sensor coordinate system of the 3D laser radar 10 and the data of the intensity IR of the reflected light.

[0048] In addition, the mobile body MB is equipped with a data processing device 20. The data processing device 20 performs "object recognition processing" to recognize objects around the mobile body MB based on the point group data LID. In the object recognition processing, the object to be tracked is recognized. As the tracking object in the case where the mobile body MB is a vehicle, other mobile bodies, road signs, road surfaces, roadside objects, and dropped objects are exemplified. Object recognition is performed by clustering processing. Clustering processing is a process of grouping similar point group data with similar reflection characteristics into multiple cluster clusters based on the point group data LID. The object recognition method including clustering processing is not particularly limited, and known techniques can be applied.

[0049] 1-2. Abnormality determination and processing

[0050] Figure 2 3D laser radar 10 is a schematic diagram showing a configuration example. Figure 2 In the example shown, the 3D laser radar 10 includes a main body 11 and a cover 12. The main body 11 outputs laser pulses, detects reflected light of the laser pulses, and processes data of the reflection point RP. The cover 12 is provided for the purpose of protecting the main body 11. The cover 12 includes a cover surface covering the outer periphery of the main body 11. Figure 2 In FIG. 1 , as a part of the mask surface, a range (hereinafter also referred to as a “scanning range”) SR through which the laser pulse and the reflected light pass is depicted.

[0051] The cover 12 is exposed to the outside air, so dirt such as rain, snow, and mud may adhere to the cover surface. In addition, the cover surface may be damaged by contact with flying objects such as pebbles. If dirt adheres to the scanning range SR or damage occurs in the scanning range SR, the accuracy of the data of the reflection point RP will decrease, and the accuracy of the object recognition based on the point group data LID will decrease. Therefore, in the embodiment, an "abnormality determination process" is performed to determine the abnormality of the scanning range SR.

[0052] In the abnormality determination process, it is determined whether an abnormality has occurred in a specified partition PR within the scanning range SR. The specified partition PR is defined based on the unit partition when the scanning range SR is divided into N. The unit partition can be composed of one partition or a set of more than two partitions. For example, the position of the center of the specified partition PR is represented by the coordinates (xi, yj, zk) on the coordinate system common to the above-mentioned sensor coordinate system. The Z axis in this case is the output center axis of the laser pulse and is parallel to the horizontal direction. In addition, the Y axis is orthogonal to the Z axis and parallel to the vertical direction. The X axis is orthogonal to the Z axis and the Y axis.

[0053] The range of the output of the laser pulse is consistent with the scanning range SR. In addition, the reflected light of the laser pulse passes through the scanning range SR. Therefore, it can be said that the shape of the range DR of the data set used to express the point group data LID is similar to the shape of the scanning range SR. Therefore, the data set range DR is divided into N in the same way as the scanning range SR. In this way, a partition (hereinafter also referred to as a "corresponding partition") PRC corresponding to the specified partition PR in the scanning range SR can be defined within the data set range DR.

[0054] Figure 3 : is a diagram schematically showing the data set range DR and the corresponding partition PRC. As already explained, the shape of the data set range DR is similar to the shape of the scanning range SR. In addition, the data set range DR is divided into N. Therefore, when the position of the center of the prescribed partition PR is represented by the coordinates (xi, yj, zk), the position of the center of the corresponding partition PRC is represented by the coordinates (xi, yj, zk).

[0055] In addition, Figure 3 In the Figure 1 The point group data LID_OB1 (X, Y, Z, IR) is obtained by processing the data of the reflection point RP of the object OB1 shown in . The point group data LID_OB1 (X, Y, Z, IR) is composed of individual data LID (X, Y, Z, IR) including the 3D position data (i.e., X, Y and Z position data) in the above-mentioned sensor coordinate system and the data of the reflected light intensity IR. The X and Y position data respectively included in each data LID (X, Y, Z, IR) represent the data of the X coordinate and Y coordinate of the data LID (X, Y, Z, IR) in the data set range DR. On the other hand, the Z position data respectively included in each data LID (X, Y, Z, IR) represent the data of the distance DS from the moving body MB (main body 11) to the reflection point RP.

[0056] Here, the reflected light intensity IR is described. The reflected light intensity IR is calculated based on the received power Pr of the laser radar according to the following equation (1), for example.

[0057]

[0058] In the first term on the right side of equation (1), Pt is the transmission power, Gt is the coefficient related to the directivity (extension) of the transmission, and R is the distance from the light source to the reflection point RP. Tt is the transmittance of the transmitting part, Tr is the transmittance of the receiving part, and L is the coefficient related to spatial attenuation. The second term on the right side of equation (1) represents the noise N.

[0059] As can be seen from equation (1), the reflected light intensity IR is inversely proportional to the fourth power of the distance R. Therefore, if the variables other than the distance R in the first term on the right side of equation (1) are considered constant and the second term is assumed to be negligible, the reflected light intensity IR can be expressed based on the distance R (that is, the distance DS).

[0060] In the abnormality determination process, the future moving direction and moving distance of the tracking object within the data set range DR are predicted based on the history of the point group data LID constituting the tracking object. In this specification, "future moving direction" is defined as the direction in which the tracking object moves relative to the moving body MB within the data set range DR from the current time to the specified time. "Future moving distance" refers to the distance that the tracking object moves in the future moving direction within the data set range DR from the current time to the specified time.

[0061] Reference Figure 4 An example of a method for predicting the future movement direction and movement distance of a tracked object within the data set range DR is described below. Figure 4 , point group data LID constituting the object OB2 to be tracked is depicted in a simplified manner. Figure 4 The object OB2 shown is a vehicle that is stopped on the shoulder of the road in front of the right side of the lane in which the moving body MB is traveling.

[0062] In addition, Figure 4 The data set range DR from time t1 to time t3 is shown in the time series. It should be noted that Figure 4 The description is based on the following contents. First, the current time is time t2, and the tracking of the object OB2 has already started at time t1. Second, during the period from time t1 to t2, the moving body MB is moving straight on the lane at a constant speed and slowly approaches the object OB2. That is, during the period from time t1 to t2, the relative speed Vrz of the moving body MB in the Z-axis direction relative to the object OB2 is constant. Third, time t3 is a future time, and the time difference from time t2 to time t3 is equal to the above-mentioned prescribed time.

[0063] The prediction of the future moving direction and moving distance of the object OB2 is performed based on the assumption that the movement of the object OB2 during the period from time t1 to time t2 will continue after time t2. In the prediction of the moving direction, first, the data (X1, Y1, Z1) of the reference position of the object OB2 (for example, the center position of the object OB2) is determined from the point group data LID_OB2 (X, Y, Z, IR) constituting the object OB2 at time t1. This determination process is also performed on the point group data LID_OB2 (X, Y, Z, IR) at time t2. In this way, the data (X2, Y2, Z2) of the reference position is also determined.

[0064] The moving direction of the object OB2 during the period from time t1 to time t2 is equal to the orientation of the vector VE having the coordinates (X1, Y1, Z1) as the starting point and the coordinates (X2, Y2, Z2) as the end point. Therefore, the future moving direction of the object OB2 (that is, the moving direction of the object OB2 within the data set range DR during the period from time t2 to time t3) is predicted to be the same direction as the orientation of the vector VE.

[0065] The moving distance of the object OB2 during the period from time t1 to time t2 is predicted based on the magnitude |VE| of the vector VE. Specifically, the future moving distance of the object OB2 (that is, the moving distance of the object OB2 within the data set range DR during the period from time t2 to time t3) is predicted using the following formula (2).

[0066] Future travel distance = |VE| * (t3-t2) / (t2-t1)...(2)

[0067] In the abnormality determination process, data of future reflected light intensity IR corresponding to the section PRC is estimated based on the predicted future moving direction and moving distance. In the following description, the future reflected light intensity IR is also referred to as "estimated value IRE".

[0068] Reference Figure 5 An example of a method for estimating the estimated value IRE is described below. Figure 5 The premise and Figure 4 In addition, the use of Figure 4 The prediction method described above is also applicable to the prediction of the future movement direction and movement distance of the object OB2 described next.

[0069] In estimating the estimated value IRE, first, a corresponding partition PRC is set within the data set range DR at the current time (that is, time t2). The corresponding partition PRC is set, for example, to a unit partition predicted to acquire the point cloud data LID constituting the tracking object at a future time. The unit partition is predicted based on the history of the position of the point cloud data LID constituting the tracking object and the future movement direction of the tracking object.

[0070] exist Figure 5 At the time t1 and t2 shown, the point group data LID_OB2 (X, Y, Z, IR) constituting the object OB2 to be tracked is located in the right region of the data set range DR. In addition, the future movement direction of the object OB2 is expected to be the direction of the vector VE. Thus, it can be seen that the unit partition predicted to acquire the point group data LID_OB2 (X, Y, Z, IR) is located in the right region and is located closer to the direction of the vector VE than the point group data LID_OB2 (X, Y, Z, IR) at the current time.

[0071] If the corresponding partition PRC is set, the coordinates (Xp, Yq, Zr) of the position after leaving the vector VE by the size |VE| in the opposite direction are determined based on the coordinates (Xi, Yj, Zk) of the center of the corresponding partition PRC. Figure 4 Specifically, the coordinates (Xp, Yq, Zr) correspond to the end point of the inverse vector IVE of the vector VE starting from the coordinates (Xi, Yj, Zk).

[0072] The coordinates (Xp, Yq, Zr) are determined within the data set range DR at the current time (that is, time t2). If the coordinates (Xp, Yq, Zr) are determined, the unit partition including the coordinates (Xp, Yq, Zr) is determined. Hereinafter, the determined unit partition is also referred to as a "specific partition" PRS. The specific partition PRS is a unit partition that is predicted to be consistent with the corresponding partition PRC within the data set range DR at a future time. The determination of the specific partition PRS is also performed within the data set range DR at the current time.

[0073] If the specific partition PRS is determined, the Z position data and the data of the reflected light intensity IR constituting the arbitrary data LID (X, Y, Z, IR) in the specific partition PRS are selected. As the arbitrary data LID (X, Y, Z, IR), the data LID (Xp, Yq, Zr, IR) located at the coordinates (Xp, Yq, Zr) in the data set range DR is exemplified. In the case where the data LID (Xp, Yq, Zr, IR) does not exist, the data LID (X, Y, Z, IR) existing in the data set range DR at the shortest distance from the coordinates (Xp, Yq, Zr) can be selected.

[0074] like Figure 4 As described in the description of , the prediction of the future moving direction and moving distance of the object OB2 is based on the assumption that the movement of the object OB2 within the data set range DR during the period from time t1 to time t2 will continue after time t2. Based on this assumption, the movement will continue at time t3. In this way, it is expected that the Z position data of any data LID (X, Y, Z, IR) constituting a specific partition PRS is highly correlated with the data of the reflected light intensity IR of the data LID (X, Y, Z, IR) contained in the corresponding partition PRC within the data set range DR at the future time t3.

[0075] The distance DSz between the moving body MB and the object OB2 in the Z-axis direction changes by Vrz*(t3-t2) during the period from time t2 to time t3. Therefore, the estimated value IRE(t3) at time t3 is calculated using the following equation (3).

[0076] IRE(t3)=IRA(t2)*DSz(t2) 4 / [DSz(t2)-Vrz*(t3-t2)] 4 ……(3)

[0077] In the above formula (3), IRA(t2) is the value of the reflected light intensity IR data of any data LID(X, Y, Z, IR) in the specific partition PRS at the time t2. DSz(t2) is the value of the Z position data of any data LID(X, Y, Z, IR) in the specific partition PRS at the time t2.

[0078] In the abnormality determination process, the estimated value IRE(t3) and the actual value IRA(t3) are compared. The actual value IRA(t3) is the actual value at time t3 of the data of the reflected light intensity IR of any data LID(X, Y, Z, IR) constituting the specific partition PRS. Then, when the difference |IRE(t3)-IRA(t3)| between the estimated value IRE(t3) and the actual value IRA(t3) exceeds the specified intensity IRth, it is determined that an abnormality has occurred in the specified partition PR.

[0079] In summary, according to the embodiment, an abnormality determination process is performed to determine whether an abnormality has occurred in a specified partition PR of the cover portion 12. In the abnormality determination process, the future moving direction and moving distance of the tracked object within the data set range DR are predicted based on the history of the point group data LID constituting the tracked object. In addition, in the abnormality determination process, the data of the future reflected light intensity IR of the corresponding partition PRC (i.e., the estimated value IRE) is estimated based on the future moving direction and moving distance. Then, when the difference |IRE-IRA| between the estimated value IRE and the actual reflected light intensity IR data of the corresponding partition PRC (i.e., the actual value IRA) exceeds the specified intensity IRth, it is determined that an abnormality has occurred in the specified partition PR.

[0080] Thus, according to the embodiment, an abnormality determination process is performed. According to the abnormality determination process, it is possible to detect that an abnormality has occurred in the predetermined section PR. Therefore, it is possible to sense that the accuracy of the data of the reflection point RP is reduced, and further it is possible to sense that the accuracy of the object recognition based on the point group data LID is reduced.

[0081] Hereinafter, the object recognition system according to the embodiment will be described in detail.

[0082] 2. Object Recognition System

[0083] 2-1. Example of configuration

[0084] Figure 6 : is a block diagram showing an example of the structure of an object recognition system according to an embodiment. Figure 6 In the example shown, the object recognition system 100 includes a 3D laser radar 10, a data processing device 20, a millimeter wave radar 30, a camera 40, a GNSS (Global Navigation Satellite System) receiver 50, a map database 60, an HMI (Human Machine Interface) unit 70 and a laser radar cleaner 80.

[0085] 3D LiDAR 10 has Figure 2The main body 11 and the cover 12 are shown in the figure. The main body 11 includes, for example, a light emitting unit that outputs laser pulses, an optical system such as an optical deflector, a lens, etc. that scans the laser pulses in the horizontal direction and the vertical direction, and a light receiving unit that detects the reflected light of the laser pulses. The light emitting unit is, for example, a laser diode. The peak wavelength of the laser pulse output from the light emitting unit is not particularly limited. The optical deflector is, for example, a MEMS (Micro Electro Mechanical Systems) mirror or a polygon mirror. The light receiving unit is, for example, a photodiode.

[0086] The 3D laser radar 10 may also have a structure for acquiring point group data LID by scanning laser pulses without using an optical deflector. Examples of such a structure include a phase array method and a flash method. The 3D laser radar 10 may also have a structure for acquiring point group data LID by mechanically rotating a light emitting unit and a light receiving unit.

[0087] The data processing device 20 is composed of a microcomputer having at least one processor 21 and at least one memory 22. The processor 21 includes a CPU (Central Processing Unit). The memory 22 is a volatile memory such as a DDR (Double Data Rate) memory, which is used to expand the program used by the processor 21 and temporarily store various data. The various data include the point group data LID acquired by the 3D laser radar 10. The functional configuration example of the data processing device 20 will be described later.

[0088] The millimeter wave radar 30 outputs millimeter wave electromagnetic waves to the surroundings of the moving body MB. When the electromagnetic waves are reflected by objects around the moving body MB, the electromagnetic waves return to the millimeter wave radar 30. The millimeter wave radar 30 detects the reflected electromagnetic waves and generates radar data MIL. As radar data MIL, data of the distance from the moving body MB (millimeter wave radar 30) to the object, data of the direction of the object relative to the moving body MB, and data of the relative speed of the object relative to the moving body MB are shown as examples. The radar data MIL is stored in the memory 22.

[0089] The camera 40 captures the surroundings of the moving body MB to acquire image data IMG. The image data IMG is stored in the memory 22. The camera 40 may also process the image data IMG to recognize objects around the moving body MB. As the image data IMG in this case, data representing the attributes of the recognized object, data representing the distance from the moving body MB to the object, data representing the direction of the object relative to the moving body MB, and data representing the position of the object are exemplified.

[0090] The GNSS receiver 50 receives signals from three or more artificial satellites. The GNSS receiver 50 generates position and posture data POS of the mobile body MB based on the received signals. The position and posture data POS is stored in the memory 22 .

[0091] The map data MAP is stored in the map database 60. Examples of the map data MAP include road position data, road shape data (e.g., curve and straight line types), intersection and structure position data. The map data MAP also includes traffic control data. The map database 60 is formed in a predetermined storage device (e.g., hard disk, flash memory) mounted on the mobile body MB. The map database 60 may also be formed in a computer of a facility that can communicate with the mobile body MB.

[0092] The HMI unit 70 exchanges various information with the driver of the mobile body MB. The HMI unit 70 includes, for example, an input device (e.g., an operation button, a touch panel), a display device, and a sound output device. The HMI unit 70 sends information input from the driver via the input device to the data processing device 20. The HMI unit 70 provides information to the driver based on a control signal from the data processing device 20. The information provided to the driver includes attention reminder information. The attention reminder information includes information indicating that an abnormality has occurred in the cover portion 12.

[0093] The laser radar cleaner 80 removes dirt attached to the cover 12. The laser radar cleaner 80 is provided near the cover 12. The laser radar cleaner 80 is, for example, an ejector that ejects cleaning fluid or air toward the cover 12. When the laser radar cleaner 80 is an ejector, two or more ejectors may be provided. In this case, the two or more ejectors are respectively provided corresponding to the responsible ranges set within the scanning range SR. The laser radar cleaner 80 is driven, for example, based on a control signal INS from the data processing device 20.

[0094] 2-2. Functional configuration example of data processing device

[0095] Figure 7 2 is a block diagram showing an example of the functional configuration of the data processing device 20. Figure 7 In the example shown, the data processing device 20 includes a data acquisition unit 23, an object recognition unit 24, a movement prediction unit 25, an estimated value calculation unit 26, and an abnormality determination control unit 27. These functions are realized by the processor 21 processing a predetermined program stored in the memory 22.

[0096] The data acquisition unit 23 acquires data required for various processes performed by the data processing device 20. Examples of various processes include object recognition processing, tracking processing, and abnormality determination processing. Examples of data required for various processes include the above-mentioned point group data LID, radar data MIL, image data IMG, position and posture data POS, and map data MAP.

[0097] The object recognition unit 24 performs object recognition processing based on the point group data LID acquired by the data acquisition unit 23. In the object recognition processing, the object to be tracked is recognized. The recognition of the object is performed by clustering processing. The object recognition method including clustering processing is not particularly limited, and a known technique can be applied. The object recognition unit 24 sends the data of the tracked object to the movement prediction unit 25.

[0098] In the object recognition process, the point group data LID may also be integrated (fusion) with other data. As a first example of other data, position and posture data POS and map data MAP are cited. By integrating these data with the point group data LID, stationary objects around the mobile body MB are identified as tracking objects. As a second example of other data, radar data MIL or image data IMG are exemplified. By integrating radar data MIL or image data IMG with the point group data LID, mobile objects around the mobile body MB are identified as tracking objects.

[0099] The movement prediction unit 25 predicts the future movement direction and movement distance of the tracking target based on the history of the point cloud data LID constituting the tracking target. Figure 8 This is a diagram showing an example of the future moving direction and moving distance. Figure 8 The coordinates (X1, Y1, Z1) and (X2, Y2, Z2) are shown in Figure 4 The coordinates of the reference position in the point group data LID_OB2 (X, Y, Z, IR) described in .

[0100] The movement prediction unit 25 calculates the direction of the vector VE starting from the coordinates (X1, Y1, Z1) and ending at the coordinates (X2, Y2, Z2). The direction of the vector VE is the future movement direction. The movement prediction unit 25 sends data indicating the direction of the vector VE to the estimated value calculation unit 26. In addition, the movement prediction unit 25 calculates the size |VE| of the vector VE. The size |VE| is equal to the distance DSR from the coordinates (X1, Y1, Z1) to the coordinates (X2, Y2, Z2) within the data set range DR.

[0101] The movement prediction unit 25 also calculates the X-axis component and the Y-axis component of the distance DSR. As already explained, the Y-axis is orthogonal to the Z-axis (the output center axis of the laser pulse) and is parallel to the vertical direction. Therefore, hereinafter, the component in the Y-axis direction is also referred to as the "vertical distance DSRy". In addition, the X-axis is orthogonal to the Z-axis and the Y-axis and is parallel to the horizontal direction. Therefore, hereinafter, the component in the X-axis direction is also referred to as the "horizontal distance DSRx". The movement prediction unit 25 sends the data of the vertical distance DSRy and the horizontal distance DSRx to the estimated value calculation unit 26.

[0102] The estimated value calculation unit 26 calculates the estimated value IRE of the corresponding partition PRC at the future time. The estimated value calculation unit 26 first sets the corresponding partition PRC within the data set range DR at the current time. The corresponding partition PRC is set to a unit partition that is predicted to acquire the point cloud data LID constituting the tracking object at the future time. The unit partition is predicted based on the history of the position of the point cloud data LID constituting the tracking object and the future movement direction of the tracking object.

[0103] Ideally, the corresponding partition PRC is set to a partition other than the unit partition predicted to acquire the point cloud data LID constituting the outer edge of the tracking object. The reason is that the point cloud data LID constituting the outer edge may include 3D position data of an object different from the tracking object. Therefore, ideally, the corresponding partition PRC is set to a unit partition predicted to be located inside the position of the outer edge.

[0104] After the corresponding partition PRC is set, the estimated value calculation unit 26 determines the coordinates (Xi, Yj, Zk) of the center position of the corresponding partition PRC. Then, the estimated value calculation unit 26 determines the coordinates (Xp, Yq, Zr) based on the coordinates (Xi, Yj, Zk) and the data of the future moving direction and moving distance of the tracking object. As described above, the coordinates (Xp, Yq, Zr) are the coordinates of the position after the coordinates (Xi, Yj, Zk) are separated from the direction opposite to the direction of the vector VE by the magnitude |VE|.

[0105] After the coordinates (Xp, Yq, Zr) are determined, the estimated value calculation unit 26 determines the unit partition (that is, the specific partition) including the coordinates (Xp, Yq, Zr). As already described, the specific partition is a unit partition that is predicted to be consistent with the corresponding partition PRC within the data set range DR at a future time. The determination of the coordinates (Xp, Yq, Zr) and the specific partition PRS is also as described above when it is performed within the data set range DR at the current time.

[0106] After the specific partition PRS is determined, the estimated value calculation unit 26 selects the Z position data and the reflected light intensity IR data constituting the arbitrary data LID (X, Y, Z, IR) in the specific partition PRS. The example of the arbitrary data LID (X, Y, Z, IR) is as described above.

[0107] Ideally, when selecting the data of reflected light intensity IR, the estimated value calculation unit 26 evaluates the selected data of reflected light intensity IR. The reason for this is that the selected data of reflected light intensity IR may correspond to the data of a singular point that deviates greatly from the data of reflected light intensity IR around it. Whether the data of reflected light intensity IR corresponds to the data of the singular point is evaluated based on a comparison with the average value AIR of the data of reflected light intensity IR of all the data LID (X, Y, Z, IR) constituting the specific partition PRS. When the difference between the selected data of reflected light intensity IR and the average value AIR exceeds the specified deviation AIRth, it is determined that the data corresponds to the data of the singular point. In this case, the data of reflected light intensity IR is discarded, and the data of reflected light intensity IR of other data LID (X, Y, Z, IR) constituting the specific partition PRS is selected.

[0108] After the Z position data and the data of the reflected light intensity IR are selected, the estimated value calculation unit 26 calculates the estimated value IRE. The estimated value IRE is calculated based on the value of the selected reflected light intensity IR data at the current time (that is, the actual value IRA), the value of the selected Z position data at the current time (that is, the distance DSz between the moving body MB and the tracking target), and the change amount of the Z position data from the current time to the future time (that is, the change amount ΔDSz of the distance DSz). The calculation formula of the estimated value IRE is expressed by the following formula (4) obtained by modifying the above formula (3).

[0109] IRE=IRA*DSz 4 / [DSz-Vrz*ΔDSz] 4 ……(4)

[0110] The estimated value calculation unit 26 sends data of the estimated value IRE to the abnormality determination control unit 27 .

[0111] The abnormality determination control unit 27 determines whether an abnormality has occurred in a predetermined section PR. This determination is performed based on a comparison between the estimated value IRE and the actual value IRA acquired with the passage of time. The actual value IRA corresponds to the data of the reflected light intensity IR constituting the data LID (X, Y, Z, IR) in the specific section PRS selected by the estimated value calculation unit 26.

[0112] When the difference |IRE-IRA| between the estimated value IRE and the actual value IRA exceeds the specified intensity IRth, the abnormality determination control unit 27 determines that an abnormality has occurred in the specified partition PR. On the contrary, it is determined that the specified partition PR is normal. The abnormality determination control unit 27 counts the number of times NJ that an abnormality has occurred in the specified partition PR. Then, when it is determined that the determination number NJ exceeds the specified number Nth, the abnormality determination control unit 27 outputs a control signal INS to the laser radar cleaner 80.

[0113] Alternatively, after the control signal INS is output to the laser radar cleaner 80, if it is determined again that the number of determinations NJ exceeds the prescribed number of times Nth, the abnormality determination control unit 27 outputs the control signal INS to the HMI unit 70. In the case where the cause of the abnormality of the prescribed partition PR is dirt, the cause is eliminated by driving the laser radar cleaner 80 based on the control signal INS. On the other hand, in the case where the cause of the abnormality is damage, the cause may not be eliminated even by driving the laser radar cleaner 80. In this regard, the control signal INS is output to the HMI unit 70 to provide the driver with a warning message. This can prompt the driver to repair the damage, etc.

[0114] 2-3. Example of abnormality determination processing

[0115] Fig. 9 This is a flowchart showing the flow of abnormality determination processing executed by the data processing device 20 (processor 21 ). Fig. 9 The routine shown is repeatedly executed in a predetermined control cycle.

[0116] exist Fig. 9 In the illustrated routine, first, a tracking target is determined (step S11). The tracking target is determined from objects recognized by an object recognition process that is performed separately.

[0117] Following the processing of step S11, it is determined whether the data range TR of the point group data LID constituting the tracking object exceeds the prescribed range TRth (step S12). For example, the data range TR is calculated by connecting the unit partitions where the point group data LID constituting the outer edge of the tracking object is located within the data set range DR. When the data range TR is less than the prescribed range TRth, it is determined that the tracking object determined in the processing of step S11 is not suitable for the abnormality determination processing. In this case, the abnormality determination processing ends.

[0118] If the result of the determination in step S12 is positive, the horizontal distance DSRx and the vertical distance DSRy are calculated (step S13). The horizontal distance DSRx and the vertical distance DSRy are calculated based on the future moving direction and moving distance of the tracking object. Figure 8 As described in .

[0119] Following the processing of step S13, it is determined whether the horizontal distance DSRx or the vertical distance DSRy exceeds the specified distance DSRth (step S14). If the horizontal distance DSRx or the vertical distance DSRy is short, the same unit partition may be compared at the current time and the future time. In this case, since the data range of the corresponding partition PRC and the specific partition PRS overlap, the data of the reflected light intensity IR with different reflection characteristics may be compared. In this way, the accuracy of abnormality determination will decrease.

[0120] Therefore, when the horizontal distance DSRx or the vertical distance DSRy is less than the predetermined distance DSRth, it is determined that the tracking target specified in the process of step S11 is not suitable for the abnormality determination process. In this case, the abnormality determination process ends.

[0121] If the determination result of step S14 is affirmative, the estimated value IRE is calculated (step S15). The estimated value IRE is calculated based on the above-mentioned equation (4).

[0122] Following the processing of step S15, it is determined whether the difference |IRE-IRA| exceeds the specified intensity IRth (step S16). If the determination result of step S16 is positive, the number of determinations NJ is incremented (NJ=NJ+1) (step S17). The increment of the number of determinations NJ is performed for each specified partition PR. If the determination result of step S16 is negative, the number of determinations NJ is reset and the abnormality determination processing ends.

[0123] Following the processing of step S17, it is determined whether the number of determinations NJ exceeds the specified number of times NJth (step S18). If the determination result of step S18 is positive, the control signal INS is output to the laser radar cleaner 80 (step S19). After the output of the control signal INS, the number of determinations NJ is reset, and the abnormality determination processing ends.

[0124] 3. Effect

[0125] According to the above-described embodiment, an abnormality determination process is performed. According to the abnormality determination process, it is possible to detect that an abnormality has occurred in a predetermined section PR based on the history of the point group data LID constituting the tracking object. Therefore, it is possible to sense a decrease in the accuracy of the data of the reflection point RP, and further, it is possible to detect a decrease in the accuracy of the object recognition based on the point group data LID.

Claims

1. An object recognition system for recognizing objects around a moving object, the object recognition system comprising: The main body of the laser radar outputs laser pulses to the surroundings of the moving body, and acquires data of the reflection points by detecting reflected light of the laser pulses reflected at the reflection points in the surroundings; A cover surface, protecting the main body; as well as a data processing device for performing object recognition processing based on point group data obtained by processing the data of the reflection points; The mask surface includes a scanning range indicating a range through which the laser pulse and the reflected light pass. The data processing device further performs an abnormality determination process for determining an abnormality in a predetermined section of the scanning range. In the abnormality determination process, the data processing device: Based on the history of the point group data, predicting a relative movement distance representing a distance that a tracking object identified by the object recognition processing moves relative to the moving body within a range of a data set used to represent the point group data during a period from the current time to a predetermined time, wherein the relative movement distance includes a vertical distance and a horizontal distance, the vertical distance representing a component in a direction orthogonal to an output center axis of the laser pulse and parallel to the vertical direction, and the horizontal distance representing a component in a direction orthogonal to the output center axis and parallel to the horizontal direction; When the vertical distance or the horizontal distance is equal to or greater than a predetermined distance, an estimated intensity of the reflected light is calculated based on the history of the point group data constituting the tracking target, wherein the estimated intensity is the intensity of the reflected light constituting the point group data of the tracking target predicted to be within the range of the data set after the predetermined time has passed and in a corresponding partition representing a partition corresponding to the predetermined partition; determining whether a difference between the estimated intensity and an actual intensity of the reflected light exceeds a prescribed intensity, wherein the actual intensity is an intensity of the reflected light that is within the range of the data set after the prescribed time has elapsed and that actually constitutes point cloud data of the tracking object in the corresponding partition; and When it is determined that the difference between the estimated intensity and the actual intensity exceeds the predetermined intensity, it is determined that an abnormality has occurred in the predetermined section.

2. The object recognition system according to claim 1, characterized in that: The data processing device further performs processing for setting the corresponding partition based on the history of the point group data constituting the tracking object in the abnormality determination processing. In the process of setting the corresponding partition, the data processing device sets the corresponding partition as a partition inside a partition of point cloud data predicted to constitute an outer edge portion of the tracking target within the range of the data set after the predetermined time has elapsed.

3. The object recognition system according to claim 1 or 2, characterized in that: The data processing device further performs, during the abnormality determination process: Determining a specific partition within the range of the data set at the current moment, wherein the specific partition refers to a partition predicted to be consistent with the corresponding partition within the range of the data set after the specified time has passed; and a process of selecting arbitrary data in the specific partition and evaluating the intensity of the reflected light constituting the arbitrary data, In the process of evaluating the intensity of the reflected light, the data processing device: Calculate the average value of the intensity of the reflected light respectively constituting all the data in the specific partition; determining whether a difference between the average value and the intensity of the reflected light constituting the selected arbitrary data exceeds a predetermined deviation; and When it is determined that the difference between the average value and the intensity of the reflected light constituting the selected arbitrary data exceeds the predetermined deviation, data different from the selected arbitrary data is selected.

4. An object recognition method for identifying objects around a moving object, wherein the object recognition method is characterized in that: A data processing device that performs object recognition processing based on point group data obtained by processing the reflection point data performs abnormality determination processing, wherein: The data of the reflection point is acquired by detecting reflected light of a laser pulse reflected at a reflection point around the moving body, the abnormality determination process determines an abnormality of a prescribed partition of a scanning range in a mask surface representing a range through which the laser pulse and the reflected light pass, and the mask surface protects a main body of a laser radar that outputs the laser pulse, detects the reflected light, and acquires the data of the reflection point. In the abnormality determination process, the data processing device: Based on the history of the point group data, predicting a relative movement distance representing a distance that a tracking object identified by the object recognition processing moves relative to the moving body within a range of a data set used to represent the point group data during a period from the current time to a predetermined time, wherein the relative movement distance includes a vertical distance and a horizontal distance, the vertical distance representing a component in a direction orthogonal to an output center axis of the laser pulse and parallel to the vertical direction, and the horizontal distance representing a component in a direction orthogonal to the output center axis and parallel to the horizontal direction; When the vertical distance or the horizontal distance is equal to or greater than a predetermined distance, an estimated intensity of the reflected light is calculated based on the history of the point group data constituting the tracking target, wherein the estimated intensity is the intensity of the reflected light constituting the point group data of the tracking target predicted to be within the range of the data set after the predetermined time has passed and in a corresponding partition representing a partition corresponding to the predetermined partition; determining whether a difference between the estimated intensity and an actual intensity of the reflected light exceeds a prescribed intensity, wherein the actual intensity is an intensity of the reflected light that is within the range of the data set after the prescribed time has elapsed and that actually constitutes point cloud data of the tracking object in the corresponding partition; and When it is determined that the difference between the estimated intensity and the actual intensity exceeds the predetermined intensity, it is determined that an abnormality has occurred in the predetermined section.

Images