Apparatus, method and program for processing information

Abstract

To provide an apparatus, method and program for processing information allowing for swiftly detecting an outlier of group-of-points data.SOLUTION: An apparatus for processing information 100 comprises an acquiring section 131, a superposing section 132 and a determining section 133. The acquiring section 131 acquires image data generated by photographing with a 3D camera and group-of-points data generated by a LiDAR and including a plurality of points at least partially overlapped with an angle of view of image data. The superposing section 132 generates a superposition image having group-of-points data superposed on the image data. The determining section 133 identifies, out of the group-of-points data, two points having values of abscissa on the superposition image both having predetermined values and determines, if one of values of ordinate on the superposition image is a first value and the other is a second value and a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value, one of the two points as a false transmission point.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and an information processing program, and more particularly, to image processing applied to an image captured by a camera provided in a moving object such as an automobile. [Background technology]

[0002] One technology proposed in relation to autonomous driving is one that improves the accuracy of object detection by combining multiple sensors equipped in a vehicle.

[0003] For example, a technique is known in which inappropriate points (outliers) are removed from measured point cloud data by using multiple LiDAR (Light Detection and Ranging) sensors that use lasers.

[0004] As a technology for removing such outliers, Patent Document 1 describes an information processing device that includes a first acquisition unit that acquires first point cloud data indicating the distance between a sensor and an object to be measured at a first measurement point along a route, a second acquisition unit that acquires second point cloud data indicating the distance between the sensor and the object to be measured at a second measurement point different from the first measurement point, a division unit that divides point cloud data including the first point cloud data and the second point cloud data into one or more segments, a distribution calculation unit that calculates a distribution for the point cloud data within the segment, and a removal unit that removes outliers from the point cloud data within the segment based on the distribution. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2022 / 153426 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technique disclosed in Patent Document 1 has a problem in that it takes a long time to detect outliers because it requires data acquired at different times.

[0007] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and an information processing program capable of quickly detecting outliers in point cloud data. [Means for solving the problem]

[0008] An information processing device according to an embodiment of the present technology includes an acquisition unit, a superimposition unit, and a determination unit. The acquisition unit acquires image data captured by a 3D camera and point cloud data generated by a LiDAR, the point cloud data including a plurality of points that at least partially overlap with the angle of view of the image data. The superimposing unit generates a superimposed image by superimposing the point cloud data on the image data. The determination unit identifies two points among the point cloud data superimposed on the image data, both of which have a predetermined value in the horizontal axis coordinate value corresponding to the azimuth angle direction in the LiDAR's irradiation on the superimposed image, and when one of the vertical axis coordinate values ​​corresponding to the elevation / depression angle direction in the LiDAR's irradiation on the superimposed image is a first value and the other is a second value different from the first value, and when a difference between the first height information of the 3D camera corresponding to the first value and the second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value, determines that one of the two points is a false transmission point.

[0009] According to this embodiment, when the point cloud data is superimposed on the image data, the horizontal axis coordinate values ​​corresponding to the azimuth angle direction in the LiDAR irradiation of two points in the point cloud data are both predetermined values, one of the vertical axis coordinate values ​​corresponding to the elevation / depression angle direction in the LiDAR irradiation of two points is a first value and the other is a second value different from the first value, and when the difference between the first height information of the 3D camera corresponding to the first value and the second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value, one of the two points is determined to be a false transmission point. In other words, the determination of a false transmission point, which is an outlier in the point cloud data, can be made without using sensing data at different times, in other words, a false transmission point can be quickly (real-time) determined from one image data.

[0010] The image forming apparatus may further include a generating unit that removes the false transparent points and generates the superimposed image by superimposing the point cloud data excluding the removed false transparent points on the image data.

[0011] The two points may be adjacent on the ordinate coordinate.

[0012] When a difference between the first height information and the second height information is equal to or greater than a predetermined value, the determination unit may determine that, of the two points, the one having a smaller value of the vertical axis coordinate is a false transmission point.

[0013] The information processing device may be mounted in a vehicle.

[0014] A judgment information processing method according to one embodiment of the present technology includes: The computer Acquire image data captured by a 3D camera and point cloud data generated by a LiDAR that at least partially overlaps with the angle of view of the image data and includes a plurality of points; generating a superimposed image by superimposing the point cloud data on the image data; Among the point cloud data superimposed on the image data, two points are identified whose horizontal axis coordinate values ​​corresponding to the azimuth angle direction in the LiDAR's irradiation on the superimposed image both have a predetermined value, and one of the vertical axis coordinate values ​​corresponding to the elevation / depression angle direction in the LiDAR's irradiation on the superimposed image is a first value and the other is a second value different from the first value, and if the difference between the first height information of the 3D camera corresponding to the first value and the second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value, one of the two points is determined to be a false transmission point.

[0015] An information processing program according to an embodiment of the present technology includes: The computer Acquiring image data captured by a 3D camera and point cloud data generated by a LiDAR, the point cloud data including a plurality of points at least partially overlapping with the angle of view of the image data; generating a superimposed image by superimposing the point cloud data on the image data; identifying two points, both of which have a predetermined value in abscissa coordinate values ​​corresponding to an azimuth angle direction in the irradiation of the LiDAR on the superimposed image, among the point cloud data superimposed on the image data, and determining that one of the two points is a false penetration point when one of the ordinate coordinate values ​​corresponding to an elevation / depression angle direction in the irradiation of the LiDAR on the superimposed image is a first value and the other is a second value different from the first value, and a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value; Execute the command. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a first diagram for explaining an overview of information processing according to an embodiment of the present technology. [Diagram 2] FIG. 2 is a second diagram for explaining an overview of information processing according to the present embodiment of the present technology. [Diagram 3]FIG. 3 is a third diagram for explaining an overview of information processing according to the present embodiment of the present technology. [Figure 4] 1 is a diagram for explaining a configuration of an information processing device according to an embodiment of the present technology. [Diagram 5] 1A and 1B are superimposed images in which point cloud data is superimposed on image data according to an embodiment of the present technology, where (A) is the superimposed image before the point cloud is removed, and (B) is the superimposed image after the point cloud is removed. [Figure 6] FIG. 13 is a diagram showing the amount of change between points in point cloud data. [Figure 7] 1 is a flowchart illustrating a process flow according to an embodiment of the present technology. [Figure 8] 1 is a block diagram showing an example of the configuration of a vehicle control system that is an example of a mobility device control system to which the present technology is applied. [Figure 9] 9 is a diagram showing an example of a sensing area by a camera, a radar, a LiDAR, an ultrasonic sensor, and the like of the external recognition sensor in FIG. 8. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0018] <Present embodiment> (Outline of Information Processing) FIG. 1 is a first diagram for explaining an overview of information processing according to the present embodiment of the present technology. The information processing in this embodiment is executed by an information processing device 100 mounted on a vehicle 1 shown in FIG. 1. In each diagram, the X-axis, Y-axis, and Z-axis indicate three mutually orthogonal axial directions of a camera image (described later), and the U-axis and V-axis indicate two mutually orthogonal axial directions when point cloud data by LiDAR 150 is drawn in an elevation. The X-axis direction is the horizontal direction in the camera image, the Y-axis direction is the height direction in the camera image, and the Z-axis direction is the depth direction in the camera image. The U-axis direction is the horizontal direction when the elevation is drawn, and the V-axis direction is the height direction when the elevation is drawn.

[0019] The information processing device 100 executes image processing according to the embodiment (acquisition of point cloud data using LiDAR, superimposition of the point cloud data on an image, etc.) by operating various functional units described later.

[0020] The vehicle 1 is, for example, a four-wheeled vehicle equipped with technology related to automatic driving. For example, the vehicle 1 uses an object detection function performed by an information processing device 100 mounted on the vehicle 1 to automatically park in a predetermined parking space, control behavior to avoid objects, and select an appropriate route.

[0021] In object detection performed by an autonomously driving moving body such as the vehicle 1, various sensors detect the external situation, and based on the detected information, detection is performed as to whether or not an object is actually present.

[0022] Examples of sensors include LiDAR, which uses a laser to measure the position of an object and the distance to the object, an image sensor mounted on a camera, and a millimeter wave radar (radar sensor) that uses the reflection of radio waves such as millimeter waves. One example of object detection is a method in which an image captured by a camera (image sensor) is used as input, and a detector performs image recognition to determine whether or not there is anything in the image that can be determined as an object. Another example is a method in which point cloud data obtained by LiDAR (depth information obtained by information reflected from an object, etc.) is used as input to a detector to perform object detection. In recent years, with the dramatic improvement in image recognition accuracy, image recognition and object detection using cameras (image sensors) are mainly used.

[0023] Furthermore, multiple sensors are also used in object detection. For example, by using LiDAR and a camera together (sensor fusion), feature information obtained from multiple sensors can be input to a neural network detector, generating a detector that outputs an object detection result. Detection using multiple sensors has the advantage of being able to detect objects more accurately than detection using a single sensor.

[0024] Also, a detector or the like can be trained based on information obtained by sensing using a sensor. In one example of training, point cloud data obtained from a LiDAR is superimposed on an image captured by a camera, and the distance to each object is accurately measured using depth information of each object included in the image. Then, by using the object captured in the image and the depth information as ground truth data, the relationship between each object in the image and the distance can be trained to learn a highly accurate distance estimation model.

[0025] However, in such a method, there are cases where point cloud data cannot be handled appropriately due to different installation positions of multiple sensors, road conditions, and the like. For example, the above-mentioned LiDAR and camera are installed in different locations on the vehicle 1. In such a situation, there is a possibility that a discrepancy occurs between an object illuminated by the LiDAR and an object captured by the camera. In this case, when the point cloud data is rendered on an image, an image is generated in which a point cloud that is farther away than the object captured by the camera is mistakenly recognized as hitting the object. If learning is performed using such an image as correct answer data, there is a risk that high-precision learning cannot be performed.

[0026] Hereinafter, information processing for a situation in which the above-mentioned erroneously recognized image is generated will be described with reference to Fig. 1. First, the difference between the irradiation range of the LiDAR 150 equipped in the vehicle 1 and the imaging range of the (3D) camera 160 will be described with reference to Fig. 1. As shown in Fig. 1, the vehicle 1 is equipped with the LiDAR 150 and the camera 160 for imaging the front. Although not shown in Fig. 1, the vehicle 1 may be equipped with even more LiDARs 150 and cameras 160.

[0027] While traveling, the vehicle 1 continuously performs detection using the LiDAR 150 and the camera 160. Fig. 1 shows point A1 indicating the point where the laser emitted by the LiDAR 150 hits the forward wall 300, and point B1 indicating that the laser hits above point A1 (in the negative direction of point A1 in the Y-axis direction) on the forward wall 300. In Fig. 1, the point where the Y coordinate value is 0 is the left end (sky side) of the image captured by the camera 160 (i.e., the Y coordinate value increases as it approaches the ground).

[0028] The camera 160 captures an image of an imaging range including a wall 300 ahead of the vehicle 1. For example, the camera 160 captures an image including a point A1 in the imaging range by imaging. At this time, the image captured by the camera 160 does not include point B1 because the wall 300 is included in the imaging range and blocks the object ahead. On the other hand, when data corresponding to point B1 is superimposed on the image captured by the camera 160 (camera image), a point B1' is included such that point B1 exists on the extension line. In other words, point B1' is a point (outlier or false penetration point) that is deemed to be a point where the laser irradiated by the LiDAR 150 hits the wall 300 in the superimposed image when point cloud data is superimposed on the camera image, even though the laser irradiated by the LiDAR 150 does not actually hit the wall 300.

[0029] At this time, point A1 is a point indicating that the laser emitted from LiDAR 150 actually hits wall 300, and the depression angle (dip angle), which is the irradiation information of the laser, is obtained from the line segment (Line A) connecting LiDAR 150 and point A1. On the other hand, point B1', which is a false penetration point, is actually a point based on the laser emitted from LiDAR 150 at the depression angle at which point B1 is obtained, so even though point B1' is located higher on the vertical axis than point A1 on the image, it is a point obtained based on the laser emitted at a deeper depression angle than point A1.

[0030] In this way, even though the laser did not actually hit point B1', when the point B1' is superimposed on the camera image, it is observed as if the laser had hit point B1'. Therefore, if the camera image and the depth information contained in point B1' are to be used as learning data, a discrepancy occurs between the distance to wall 300 and the depth information contained in point B1', and the reliability of the point B1' as correct answer data becomes low.

[0031] Therefore, it is considered that the information processing device 100 specifies the point B1' that is a false transparent point (occlusion point) by the following method, and executes a process of removing the specified false transparent point.

[0032] When irradiating a laser, the LiDAR 150 can acquire irradiation information including "elevation" indicating the height angle of the irradiation, i.e., elevation / depression angle information, and "azimuth" indicating the horizontal angle with respect to the vehicle 1, i.e., azimuth angle information. The LiDAR 150 can also acquire identification information (irradiation ID) for each irradiation as irradiation information. The information processing device 100 acquires this irradiation information together with the point cloud data. That is, the information processing device 100 can specify the elevation / depression angle information and azimuth angle information when the laser is irradiated, for each piece of point cloud data obtained from the LiDAR 150, based on the irradiation ID. The information processing device 100 can also specify the height (Y axis) information and the horizontal position (X axis) in the image captured by the camera 160.

[0033] Based on such information, the information processing device 100 identifies the false transparent point. First, the information processing device 100 selects one point to be processed from the point cloud data superimposed on the image captured by the camera 160. For example, the information processing device 100 selects point A1. Next, the information processing device 100 selects another point on the same or approximately the same X-axis. Here, the information processing device 100 selects point B1', which is another point on the same or approximately the same X-axis.

[0034] Then, the information processing device 100 compares the illumination information based on the illumination IDs of the two points. Specifically, the information processing device 100 compares the elevation and depression angle information of the two points. Then, when there is a contradiction in the elevation and depression angles even though the points are on the same X-axis, the information processing device 100 specifies the point where the contradiction occurs as a false transparent point. Specifically, when point B1', which should be superimposed at a higher position on the Y-axis of the camera image than point A1 because the point B1' is superimposed at a lower position than point A1 in the camera image because the point B1' is irradiated at a shallower depression angle, the information processing device 100 specifies point B1' as a false transparent point. Alternatively, when the illumination information of the point cloud data is specified by "(elevation, azimuth)", the point A1 is drawn lower than point B1' in the camera image, while the "elevation" value of point A1 is larger than the "elevation" value of point B1', the information processing device 100 specifies point B1' as a false transparent point. Furthermore, if the "elevation" value of point A1 is smaller than the "elevation" value of point B1', the information processing device 100 determines that neither point A1 nor point B1' is a false transparent point, compares the illumination information with that of the next point to be processed, and sequentially identifies false transparent points.

[0035] The information processing device 100 deletes the identified false transparent points and does not superimpose them on the camera image. The information processing device 100 can identify all false transparent points included in the camera image by performing the relevant processing on all point cloud data of the camera image.

[0036] The information processing device 100 may specify a false transmission point using not only elevation and depression angle information but also azimuth angle information. That is, depending on the relationship between the installation positions of the LiDAR 150 and the camera 160, not only the laser ahead of the object may be detected as a false transmission point due to the difference in the installation positions based on the elevation direction as described above, but also the laser may be irradiated to the front of the object so as to go around from the side of the object, and thus detected as a false transmission point.

[0037] In this case, the information processing device 100 removes the point B1' and generates an image by superimposing the remaining point cloud data on the camera image. This allows the information processing device 100 to obtain an image on which only point cloud data having accurate depth information corresponding to objects on the image is superimposed.

[0038] However, with the above-described information processing method, it becomes difficult to appropriately process information in the case of Fig. 2. Fig. 2 is a second diagram for explaining an overview of information processing according to the present embodiment of the present technology.

[0039] 2 is a diagram showing a downhill road D1 in front of a traveling vehicle 1. A laser LA emitted from the LiDAR 150 hits a point A on the road D1. Similarly, a laser LB emitted from the LiDAR 150 hits a point B on the road D1, and a laser LC emitted from the LiDAR 150 hits a point C on the road D1.

[0040] 2, in the (U, V) coordinates (pixel coordinates) when multiple points A, B, and C of the point cloud data of LiDAR 150 are plotted (projected) in an elevation, the V coordinates are arranged in ascending order as indicated by the arrow Lv (i.e., the V coordinate value decreases in the order of point A, point B, and point C). However, in the (X, Y, Z (not shown)) coordinates (camera coordinates) of the imaging range captured by camera 160, the Y coordinates are arranged in descending order as indicated by the arrow Cy (i.e., the Y coordinate value decreases in the order of point C, point B, and point A).

[0041] In the case of Fig. 2, when the above-mentioned information processing is performed, when point C is compared with point B, point B has a deeper depression angle than point C, but when points B and C are superimposed on a camera image (image data), point C will be superimposed at a lower position in the Y-axis direction of the camera image than point B. In other words, in the above-mentioned information processing method, point C will be removed as a false transparent point due to the occurrence of a contradiction.

[0042] However, the road D1 shown in Fig. 2 is a slope (downhill), and point C is not actually a false penetration point. Therefore, in the above-mentioned information processing method, there is a risk that a point that is not a false penetration point will be removed as a false penetration point.

[0043] In the case of Fig. 3, point C' is removed in the same manner as in Fig. 2. Fig. 3 shows a downward road D2 in front of a traveling vehicle 1, and a puddle P exists in the middle of the road D2. A laser beam LA' irradiated from the LiDAR 150 hits a point A' on the road D2. A laser beam LB' irradiated from the LiDAR 150 hits a point B' on the road D2, and a laser beam LC' irradiated from the LiDAR 150 hits the puddle P on the road D2.

[0044] Here, as shown in FIG. 3, point C' is a point (false transmission point) where the laser LC' emitted from the LiDAR 150 enters the puddle P and appears to hit point C' due to refraction.

[0045] In the case of Fig. 3, the above-mentioned information processing method removes point C' in the same way as Fig. 2. In other words, in the case of the above-mentioned processing, each point in Fig. 2 and Fig. 3 is regarded as a false penetration point because it is a slope (downhill), and it was difficult to judge whether it was actually a false penetration point or not.

[0046] Therefore, the information processing device 100 according to this embodiment executes the following process. That is, the information processing device 100 acquires image data generated by capturing an image with the (3D) camera 160, and point cloud data G generated by the LiDAR 150 and including a plurality of points that at least partially overlap with the angle of view of the image data, generates a superimposed image T1 by superimposing the point cloud data G on the image data, and identifies two points among the point cloud data G superimposed on the image data, the horizontal axis (X' coordinate) coordinate values ​​corresponding to the azimuth angle direction (U axis direction) in the irradiation of the LiDAR 150 on the superimposed image T1 both having a predetermined value P, and the vertical axis (Y' coordinate) coordinate values ​​corresponding to the elevation / depression angle direction (V axis direction) in the irradiation of the LiDAR 150 on the superimposed image T1 are one first value and the other a second value different from the first value, and if the difference between the first height information of the (3D) camera 160 corresponding to the first value and the second height information of the (3D) camera 160 corresponding to the second value is equal to or greater than a predetermined value, one of the two points is determined to be a false penetration point. Then, the information processing device 100 removes the false transparent points, and generates a superimposed image T2 by superimposing the point cloud data G' excluding the removed false transparent points on the image data.

[0047] The configuration of the information processing device 100 will be described with reference to Figs. 4 and 5. Fig. 4 is a diagram for explaining the configuration of the information processing device 100 according to this embodiment of the present technology. Fig. 5 shows a superimposed image in which point cloud data is superimposed on image data, (A) is a superimposed image T1 before point cloud deletion, and (B) is a superimposed image T2 after point cloud deletion. As shown in Fig. 4, the information processing device 100 has a communication unit 110, a storage unit 120, a control unit 130, and a detection unit 140. Note that the configuration shown in Fig. 4 is a functional configuration, and the hardware configuration may be different from this. Also, the functions of the information processing device 100 may be distributed and implemented in multiple physically separated devices.

[0048] The communication unit 110 is realized by, for example, a network interface controller or a network interface card (NIC). The communication unit 110 may be a USB interface configured by a USB (Universal Serial Bus) host controller, a USB port, or the like. The communication unit 110 may be a wired interface or a wireless interface. For example, the communication unit 110 may be a wireless communication interface of a wireless LAN system or a cellular communication system. The communication unit 110 functions as a communication means or a transmission means of the information processing device 100. For example, the communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information between an external device such as a cloud server and another information processing terminal via the network N. The network N is realized by, for example, a wireless communication standard or system such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), LPWA (Low Power Wide Area), ELTRES (registered trademark), or the like.

[0049] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various data. For example, the storage unit 120 stores irradiation information when the LiDAR 150 irradiates a laser, image data captured by the camera 160, and the like. The storage unit 120 may store a learning device (object detection model) trained for object detection, image data used for learning, and the like. The storage unit 120 may also store map data for performing autonomous driving. Note that, in the present disclosure, an example is shown in which the storage unit 120 is mounted on the information processing device 100 (i.e., the vehicle 1), but the data stored in the storage unit 120 may be stored on an external device such as a cloud server.

[0050] The detection unit 140 detects various types of information related to the vehicle 1 and the information processing device 100. Specifically, the detection unit 140 detects the environment around the vehicle 1, location information of the vehicle 1, information related to other devices connected to the information processing device 100 mounted on the vehicle 1, and the like. The detection unit 140 may be interpreted as a sensor that detects various types of information.

[0051] For example, the detection unit 140 has a LiDAR 150 and a camera 160 as sensors. The LiDAR 150 is a sensor that reads the three-dimensional structure of the surrounding environment of the vehicle 1, and the irradiation range of the LiDAR 150 at least partially overlaps with the angle of view of image data described later. Specifically, the LiDAR 150 detects the distance to an object and the relative speed by irradiating a laser beam such as an infrared laser to the surrounding object and measuring the time it takes for the laser beam to be reflected and returned.

[0052] The camera 160 is a sensor having a function of capturing an image of the surroundings of the vehicle 1, and captures an image so that the imaging range of the camera 160 is included within the irradiation range of the laser by the LiDAR 150. In this embodiment, the camera 160 is a 3D (three-dimensional) camera including two lenses, but may be in any form, such as a stereo camera, a monocular camera, or a lensless camera. The camera 160 is not limited to a visible light camera such as an RGB camera, and may be a camera with a depth sensor equipped with a ToF (Time of Flight) sensor. The camera 160 may also be equipped with an AI-equipped image sensor capable of detecting and recognizing objects.

[0053] The detection unit 140 may have various sensors other than the LiDAR 150 and the camera 160. For example, the detection unit 140 may include a distance measurement system using a millimeter wave radar. The detection unit 140 may include a depth sensor for acquiring height information. The detection unit 140 may be a sonar that searches the surrounding environment by sound waves. The detection unit 140 may also include a microphone that collects sounds around the vehicle 1, an illuminance sensor that detects the illuminance around the vehicle 1, a humidity sensor that detects the humidity around the vehicle 1, a geomagnetic sensor that detects the magnetic field at the location of the vehicle 1, and the like.

[0054] Although not shown in FIG. 4, the information processing device 100 may include a display unit that displays various information. The display unit is a mechanism for outputting various information, such as a liquid crystal display. For example, the display unit may display an image captured by the detection unit 140, or an object detected by the information processing device 100 in the image. The display unit may also function as a processing unit for accepting various operations from a user or the like who uses the information processing device 100. For example, the display unit may accept input of various information via key operations, a touch panel, or the like.

[0055] The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an image processing program according to the present embodiment) stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is a controller, and may be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0056] 4, the control unit 130 has an acquisition unit 131, a superimposition unit 132, a determination unit 133, and a generation unit 134. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform information processing described later.

[0057] The acquisition unit 131 acquires various information and includes a first acquisition unit 131A and a second acquisition unit 131B. The first acquisition unit 131A acquires point cloud data indicating that a surrounding object has been detected from the LiDAR 150, which is a sensor using a laser.

[0058] The first acquisition unit 131A also acquires point cloud data G, which is generated by the LiDAR 150 and includes a plurality of points that at least partially overlap with the angle of view of the camera image (image data) captured by the camera 160 (3D camera), as well as laser irradiation information when the point cloud data is obtained. For example, the first acquisition unit 131A acquires a plurality of points (see, for example, points A, B, and C in FIG. 2) as point cloud data, and elevation / depression angle information and azimuth angle information at which the laser is irradiated as irradiation information. In other words, the first acquisition unit 131A acquires a numerical value indicating the elevation direction and a numerical value indicating the horizontal direction when the laser is irradiated as irradiation information.

[0059] The second acquisition unit 131B acquires a three-dimensional image, which is the camera coordinate system whose imaging range includes the laser irradiation range of the LiDAR 150. Specifically, the second acquisition unit 131B acquires image data generated by capturing an image of the surroundings of the vehicle 1 with the camera 160.

[0060] The superimposing unit 132 generates a superimposed image T1 by superimposing the point cloud data G on the image data (see FIG. 5(A)). The superimposing unit 132 calibrates the point cloud data G from the LiDAR 150, and converts the coordinate system of the LiDAR 150, which is installed at a different position, into the coordinate system of the camera 160. Then, the converted coordinate system of the LiDAR 150 is superimposed on the image data. Here, when the point cloud data G is superimposed on the image data, the azimuth angle direction (U-axis direction) in the irradiation of the LiDAR 150 corresponds to the horizontal axis (X'-axis) coordinate of the superimposed image T1, and the elevation / depression angle direction (V-axis direction) in the irradiation of the LiDAR 150 corresponds to the vertical axis (Y'-axis) coordinate of the superimposed image T1.

[0061] The determination unit 133 identifies two points among the point cloud data G superimposed on the image data by the superimposition unit 132, both of which have a predetermined value for the horizontal axis (X' axis) coordinate value corresponding to the azimuth angle direction (U axis direction) in the irradiation of the LiDAR 150 on the superimposed image T1, and one of the vertical axis (Y') coordinate values ​​corresponding to the elevation / depression angle direction (V axis direction) in the irradiation of the LiDAR 150 on the superimposed image T1 is a first value and the other is a second value different from the first value, and if the difference between the first height information of the (3D) camera 160 corresponding to the first value and the second height information of the (3D) camera 160 corresponding to the second value is equal to or greater than a predetermined value, the determination unit 133 determines that one of the two points is a false transparent point.

[0062] Specifically, when the point cloud data G is superimposed, the determination unit 133 extracts coordinates (X', Y') = (P, Q1) of a first point S1 (one of the multiple points constituting the point cloud data G) corresponding to a first feature point of the superimposed image T1, and coordinates (X', Y') = (P, Q2) of a second point S2 (one of the multiple points constituting the point cloud data G) adjacent in the vertical axis (Y' axis) coordinate and corresponding to a second feature point of the superimposed image T1 when the point cloud data G is superimposed. Then, the determination unit 133 calculates the difference between the values ​​of the height information of the (3D) camera 160 corresponding to the Y' coordinates of the two extracted points, and if the difference is equal to or greater than a predetermined value R, determines that one of the two points is a false transparent point.

[0063] Here, adjacent refers to the relationship between a certain point (for example, the first point S1) in the point cloud data G and a point (for example, the second point S2) having the next largest or smallest value in the Y' coordinate of the certain point.

[0064] Here, the height information of the (3D) camera 160 is the Y coordinate value of the (3D) camera 160. That is, when the difference between the first height information (the Y coordinate value of the (3D) camera 160) corresponding to the first value (the Y' coordinate value) Q1 of the first point S1 and the second height information (the Y coordinate value of the (3D) camera 160) corresponding to the second value (the Y' coordinate value) Q2 of the second point S2 is equal to or greater than a predetermined value R, the determination unit 133 determines that one of the two points is a false transparent point.

[0065] Here, the origin of the superimposed image is based on the upper left of the superimposed image as shown in Fig. 5(A) and (B). That is, more specifically, the determination unit 133 determines the second point S2 as a false transparent point when the difference between the first height information corresponding to the Y' coordinate value of the first point S1 on the ground side (where the Y' coordinate value is larger in Fig. 5(A)) in the point cloud data and the second height information corresponding to the Y' coordinate value of the second point S2 having the second smallest Y' coordinate value after the first point S1, although the X' coordinate of the coordinate of the first point S1 is the same, is a predetermined value R or more. That is, in this embodiment, when the difference between the first height information of the first point S1 and the second height information of the second point S2 is a predetermined value R or more, the determination unit 133 determines the one having the smaller vertical axis (Y' axis) coordinate value between the first point S1 and the second point S2 as a false transparent point. Of course, the method of determining the reference for the Y' coordinate value being 0 is not limited to this, and the line where the Y' coordinate value is 0 may be determined based on the bottom right or bottom left of the camera image.

[0066] The determination unit 133 determines the value of the X'-coordinate of the superimposed image T1 and scans the Y'-coordinate (for example, scans along the Y'-axis direction within the vertically long black frame in FIG. 5(A) for each value of the X'-coordinate) to determine the above-mentioned pseudo-transparency point. Specifically, the determination unit 133 scans in order from a large value to a small value of the Y'-coordinate value of the superimposed image T1. That is, in the case of the superimposed image T, the points of the point cloud data G are scanned in order from the bottom of the superimposed image T1, and when the first point S1 is specified and then the second point S2 is specified, the above-mentioned determination process is performed. Here, in the present embodiment, the determination unit 133 scans the Y-coordinate for a plurality of pixels (for example, 10 pixels) of the superimposed image T1 at once. Thereby, the pseudo-transparency point can be determined quickly. Of course, it is not limited to this, and the Y-coordinate may be scanned for each pixel to determine the pseudo-transparency point.

[0067] In the case of FIG. 2, the determination unit 133 compares the Y-coordinate (height information) corresponding to the camera image of point C in the point cloud data with the coordinate (height information) corresponding to the camera image of point B. At this time, the value of the horizontal axis (X'-axis) coordinate in the superimposed image is the same. In this case, the difference H2 (<R) between the values of the height information of point B and point C. In the present embodiment, since the difference H2 is less than the predetermined value R, point C is not determined as a pseudo-transparency point. Similarly, when comparing point B and point A, the difference H1 (<R) between the values of their height information is also less than the predetermined value R in the present embodiment, so point B is not determined as a pseudo-transparency point.

[0068] In the case of FIG. 3, the determination unit 133 compares the Y-coordinate (height information) corresponding to the camera image of point B' in the point cloud data with the Y-coordinate (height information) corresponding to the camera image of point A' in the same manner as in FIG. 2. At this time, similar to FIG. 2, the difference H1' (<R) between the values of the height information is less than the predetermined value R in the present embodiment, so point B' is not determined as a pseudo-transparency point. However, when comparing the Y-coordinate (height information) corresponding to the camera image of point C' in the point cloud data with the Y-coordinate (height information) corresponding to the camera image of point B', the difference H2' (>R) between the values of the height information is greater than or equal to the predetermined value R in the present embodiment, so point C' is determined as a pseudo-transparency point. Note that the Y'-coordinate value of point C' is smaller than that of point B'.

[0069] In this embodiment, the predetermined value R increases toward the top or bottom of the image as shown in FIG. 6. FIG. 6 is a diagram showing the amount of change H between points in the point cloud data. In other words, the larger the absolute value of the Y coordinate value, the larger the predetermined value R becomes. Here, the upper part of the image in this embodiment is the sky side, and the lower part of the image in this embodiment is the ground side. In this embodiment, the predetermined value changes linearly. Furthermore, the determination unit 133 determines that multiple points in a range W that exceeds the predetermined value R shown in FIG. 6 are false transparent points.

[0070] 5(A) and (B), the generating unit 134 removes the false transparent points determined by the determining unit 133 (see the lower right frames in FIGS. 5(A) and (B)), and generates a superimposed image T2 in which point cloud data G' excluding the removed false transparent points is superimposed. In the case of FIG. 3, an image is generated in which points A' and B' are superimposed with point C' removed.

[0071] (Flowchart of information processing) The above-mentioned information processing will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of processing according to the embodiment.

[0072] 7, the first acquisition unit 131A of the information processing device 100 acquires point cloud data from the LiDAR 150 (step S101). The second acquisition unit 131B of the information processing device 100 acquires a camera image (image data) captured by the (3D) camera 160 (step S102). Here, as described above, the point cloud data has an angle of view that at least partially overlaps with the angle of view of the camera image.

[0073] Next, the superimposing unit 132 of the information processing device 100 generates a superimposed image T1 by superimposing the acquired point cloud data G on the camera image (see FIG. 5(A)) (step S103). Then, the information processing device 100 extracts two points to be processed (a first point S1 and a second point S2 in this embodiment) from the superimposed point cloud data G (step S104).

[0074] Then, the information processing device 100 executes the above-mentioned determination process for the relationship between the two extracted points, and determines whether there is a difference between the two extracted points that is equal to or greater than a predetermined value R (step S105). If there is a difference that is equal to or greater than the predetermined value R (step S105; Yes), the information processing device 100 executes the above-mentioned determination process and deletes the false transparent points (step S106).

[0075] On the other hand, if there is no difference between the two extracted points that is equal to or greater than the predetermined value (step S105; No), the information processing device 100 judges whether or not the processing of all point cloud data has been completed at that point (step S107). If there is point cloud data remaining to be processed (step S107; No), the information processing device 100 extracts the next two points and repeats the process of determining false transparent points.

[0076] On the other hand, when the processing of all the point cloud data has been completed (Step S107; Yes), the information processing device 100 generates a superimposed image T2 by superimposing the remaining point cloud data G' after deleting the false transparent points (Step S108).

[0077] As described above, the information processing device 100 determines false transmission points in the point cloud data. This allows the information processing device 100 to use only points that are actually irradiated on the target object for subsequent processing, and therefore allows the point cloud data obtained by sensing to be appropriately used.

[0078] The information processing device 100 further includes a generating unit 133 that removes the identified false transparent points and generates a superimposed image by superimposing the point cloud data excluding the removed false transparent points. This allows the information processing device 100 to generate a superimposed image by superimposing the point cloud data after removing the false transparent points. Therefore, the information processing device 100 can use such an image as correct answer data in the detection process and the learning process, and can execute a process with higher accuracy.

[0079] Furthermore, the information processing device 100 can determine false transmission points from the point cloud data in one frame (one image data), which makes it possible to quickly (in real time) determine false transmission points.

[0080] <Variation 1> In the above-described embodiment, an example has been shown in which the vehicle 1, which is an autonomous vehicle equipped with the LiDAR 150 and the camera 160, executes the image processing according to the embodiment. However, the image processing according to the embodiment may be executed by various moving bodies, not limited to autonomous vehicles (so-called automobiles).

[0081] For example, the moving object that executes the image processing according to the embodiment may be a small vehicle such as a motorcycle or three-wheeled vehicle, a large vehicle such as a bus or truck, or an autonomous moving object such as a robot or drone. Furthermore, the image processing device 100 is not necessarily integrated with a moving object such as the vehicle 1, and may be a cloud server that acquires information from the moving object via a network and performs image processing based on the acquired information.

[0082] <Other variations> The process according to each embodiment described above may be implemented in various different forms other than the above embodiments. For example, in the above embodiments, the road is a downhill road, but this is not limited to this, and it is possible to determine a false penetration point even on an uphill road or a road with a mixture of uphill and downhill roads.

[0083] <<1. Example of vehicle control system configuration>> FIG. 8 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0084] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to automated driving of the vehicle 1. This automated driving includes automated driving of levels 1 to 5, and remote driving and remote assistance of the vehicle 1 by a remote driver.

[0085] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0086] The vehicle control ECU 21, the communication unit 22, the map information storage unit 23, the position information acquisition unit 24, the external recognition sensor 25, the in-vehicle sensor 26, the vehicle sensor 27, the storage unit 28, the driving automation control unit 29, the DMS 30, the HMI 31, and the vehicle control unit 32 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 41 is configured by an in-vehicle communication network or a bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 41 may be used according to the type of data to be transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-volume data. Note that each unit of the vehicle control system 11 may be directly connected to each other using wireless communication intended for communication at a relatively short distance, such as near-field communication (NFC (Near Field Communication)) or Bluetooth (registered trademark), without going through the communication network 41.

[0087] In the following description, when each unit of the vehicle control system 11 communicates with each other via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, it will be simply described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0088] The vehicle control ECU 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0089] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can communicate using a plurality of communication methods.

[0090] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method that the communication unit 22 uses with the external network is not particularly limited as long as it is a wireless communication method that allows digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0091] Also, for example, the communication unit 22 can communicate with a terminal existing near the vehicle by using a P2P (Peer To Peer) technique. The terminal existing near the vehicle can be, for example, a terminal attached to a moving body moving at a relatively low speed such as a pedestrian or a bicycle, a terminal installed at a fixed position in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. The V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside device, vehicle-to-home communication with a home, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.

[0092] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (Over The Air). The communication unit 22 can further receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. For example, the communication unit 22 can transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1, a recognition result by the recognition unit 73, and the like. For example, the communication unit 22 also performs communication corresponding to a vehicle emergency call system such as e-Call.

[0093] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0094] The communication with the inside of the vehicle that can be performed by the communication unit 22 will be briefly described. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can wirelessly communicate with each device in the vehicle using a communication method that allows digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wireless communication, such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). Not limited to this, the communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal not shown. The communication unit 22 can communicate with each device in the vehicle using a communication method that allows digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or MHL (Mobile High-definition Link).

[0095] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 41. Assumed in-vehicle devices include, for example, a mobile device or a wearable device carried by a user in the vehicle, such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0096] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that has lower precision than a high-precision map and covers a wide area, and the like.

[0097] The high-precision map is, for example, a dynamic map, a point cloud map, a vector map, etc. The dynamic map is, for example, a map consisting of four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. The point cloud map is a map composed of a point cloud (point cloud data). The vector map is, for example, a map adapted to driving automation by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.

[0098] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 1 as a map for matching with a local map described later based on sensing results from the camera 51, the radar 52, the LiDAR 53, etc., and stored in the map information storage unit 23. In addition, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred meters square regarding the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0099] The position information acquisition unit 24 receives GNSS signals from Global Navigation Satellite System (GNSS) satellites and acquires position information of the vehicle 1. The acquired position information is supplied to the driving automation control unit 29. Note that the method used by the position information acquisition unit 24 is not limited to using GNSS signals, and for example, the position information may be acquired using a beacon.

[0100] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0101] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Not limited to this, the external recognition sensor 25 may include one or more types of sensors among the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of the cameras 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 is not particularly limited as long as it is a number that can be practically installed in the vehicle 1. In addition, the type of the sensor included in the external recognition sensor 25 is not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0102] The imaging method of the camera 51 is not particularly limited. For example, cameras of various imaging methods, such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera, which are imaging methods capable of distance measurement, can be applied to the camera 51 as necessary. However, the camera 51 may be one simply for acquiring a photographed image, regardless of distance measurement.

[0103] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as the weather, climate, and brightness, and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0104] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0105] The in-vehicle sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they are of the types and number that can be realistically installed in the vehicle 1.

[0106] For example, the in-vehicle sensor 26 may include one or more types of sensors selected from a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, and a biosensor. The camera included in the in-vehicle sensor 26 may be a camera using various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera. The camera included in the in-vehicle sensor 26 may be a camera simply for acquiring captured images, regardless of distance measurement. The biosensor included in the in-vehicle sensor 26 is provided, for example, on a seat, a steering wheel, or the like, and detects various types of bioinformation of the user.

[0107] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are types and numbers that can be realistically installed in the vehicle 1.

[0108] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of operation of an accelerator pedal, and a brake sensor that detects the amount of operation of a brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or the motor, an air pressure sensor that detects the air pressure of the tires, a slip ratio sensor that detects the slip ratio of the tires, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining charge and temperature of the battery, and an impact sensor that detects an impact from the outside.

[0109] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage unit 28 stores various programs and data used by each part of the vehicle control system 11. For example, the storage unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and stores information on the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensor 26.

[0110] The driving automation control unit 29 controls the driving automation function of the vehicle 1. For example, the driving automation control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0111] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0112] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0113] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the above-mentioned point cloud map. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size, and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and the probability of its existence. The local map is also used, for example, in detection processing and recognition processing of the external situation of the vehicle 1 by the recognition unit 73.

[0114] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

[0115] The sensor fusion unit 72 performs a sensor fusion process to obtain information by combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include compounding, integration, fusion, and association.

[0116] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1.

[0117] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, and the like.

[0118] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0119] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points. This allows the presence or absence, size, shape, and position of objects around the vehicle 1 to be detected.

[0120] For example, the recognition unit 73 performs tracking to follow the movement of the clusters of the point cloud classified by clustering, thereby detecting the movement of the objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0121] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like, based on image data supplied from the camera 51. Furthermore, the recognition unit 73 may recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0122] For example, the recognition unit 73 can perform recognition processing of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the self-position by the self-position estimation unit 71, and the recognition result of the objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the positions and states of traffic lights, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is possible.

[0123] For example, the recognition unit 73 can perform a recognition process of the environment around the vehicle 1. The surrounding environment to be recognized by the recognition unit 73 may include weather, temperature, humidity, brightness, and road surface conditions.

[0124] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0125] The route planning includes global path planning and local path planning. Global path planning includes a process of planning a rough route from a start to a goal. Local path planning is also called trajectory planning, and includes a process of generating a trajectory that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 on the planned route.

[0126] Path following is a process for planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The action planning unit 62 can calculate a target speed and a target angular velocity of the vehicle 1, for example, based on the result of the path following process.

[0127] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0128] For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in the vehicle control unit 32 described later, to perform lateral vehicle motion control and longitudinal vehicle motion control so that the vehicle 1 moves along a trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs control for the purpose of driving automation such as driver assistance functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, host vehicle collision warning, and host vehicle lane departure warning, and driving without the operation of the driver or a remote driver.

[0129] The DMS 30 performs authentication processing of the driver and recognition processing of the driver's state based on the sensor data from the in-vehicle sensor 26 and input data input to the HMI 31 described later. Examples of the driver's state to be recognized include physical condition, alertness, concentration, fatigue, line of sight, drunkenness, driving operation, posture, and the like.

[0130] The DMS 30 may perform authentication processing of a user other than the driver and recognition processing of the state of the user. For example, the DMS 30 may perform recognition processing of the state inside the vehicle based on sensor data from the in-vehicle sensor 26. The state inside the vehicle to be recognized may be, for example, temperature, humidity, brightness, odor, and the like.

[0131] The HMI 31 inputs various data and instructions, and presents various data to the user.

[0132] The input of data by the HMI 31 will be briefly described. The HMI 31 includes an input device for a person to input data. The HMI 31 generates an input signal based on data, instructions, etc. input by the input device, and supplies the input signal to each part of the vehicle control system 11. The HMI 31 includes, as input devices, operators such as a touch panel, a button, a switch, and a lever. Without being limited to this, the HMI 31 may further include an input device that allows information to be input by a method other than manual operation, such as voice or gesture. Furthermore, the HMI 31 may use, as an input device, an externally connected device such as a remote control device using infrared rays or radio waves, or a mobile device or wearable device compatible with the operation of the vehicle control system 11.

[0133] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the user or the outside of the vehicle. The HMI 31 also performs output control to control the output, output contents, output timing, output method, etc. of each generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image indicating the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information given to the user's sense of touch by, for example, force, vibration, movement, etc.

[0134] Examples of the output device to which the HMI 31 outputs visual information include a display device that displays an image to present visual information, and a projector device that projects an image to present visual information. The display device may be a device that displays visual information within the user's field of vision, such as a head-up display, a transmissive display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device having a normal display. The HMI 31 may also use display devices provided in a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, and the like provided in the vehicle 1 as output devices that output visual information.

[0135] As an output device to which the HMI 31 outputs auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.

[0136] For example, a haptic element using haptic technology can be applied as an output device for outputting tactile information from the HMI 31. The haptic element is provided on a part that the user touches, such as a steering wheel or a seat.

[0137] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0138] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0139] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0140] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, etc. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, an actuator for driving the drive system, etc.

[0141] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0142] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0143] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0144] Fig. 9 is a diagram showing an example of a sensing area by the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 of the external recognition sensor 25 in Fig. 8. Note that Fig. 9 shows a schematic view of the vehicle 1 as seen from above, with the left end side being the front end of the vehicle 1 and the right end side being the rear end of the vehicle 1.

[0145] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. The sensing area 101F covers the periphery of the front end of the vehicle 1 by the multiple ultrasonic sensors 54. The sensing area 101B covers the periphery of the rear end of the vehicle 1 by the multiple ultrasonic sensors 54.

[0146] The sensing results in the sensing area 101F and the sensing area 101B are used for, for example, parking assistance for the vehicle 1.

[0147] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther in the rear of the vehicle 1 than sensing area 101B. Sensing area 102L covers the rear periphery of the left side of the vehicle 1. Sensing area 102R covers the rear periphery of the right side of the vehicle 1.

[0148] The sensing results in the sensing area 102F are used, for example, to detect vehicles, pedestrians, and the like that exist in front of the vehicle 1. The sensing results in the sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in the sensing area 102L and the sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0149] Sensing area 103F to sensing area 103B show examples of sensing areas by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0150] The sensing results in the sensing area 103F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in the sensing area 103B can be used for, for example, parking support and surround view systems. The sensing results in the sensing areas 103L and 103R can be used for, for example, surround view systems.

[0151] Sensing area 104 illustrates an example of the sensing area of ​​the LiDAR 53. Sensing area 104 covers a position farther forward than sensing area 103F in front of the vehicle 1. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0152] The sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles, for example.

[0153] Sensing area 105 shows an example of the sensing area of ​​the long-range radar 52 . The sensing area 105 covers a position farther in front of the vehicle 1 than the sensing area 104. On the other hand, the sensing area 105 has a narrower range than the sensing area 104 in the left-right direction.

[0154] The sensing results in the sensing area 105 are used for, for example, ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

[0155] The sensing areas of the cameras 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. 9. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the above-mentioned examples. The number of sensors may be one or more.

[0156] In the above-described on-vehicle control system 11, the information processing device 100 of the present disclosure may be applied to the radar 52 of the external recognition sensor 25. In the information processing device 100 of the present disclosure, the presence or absence of a false penetration point may be determined, and if it is determined that there is a false penetration point, a notice may be sent to the user of the vehicle 1 that the information processing device 100 may not operate properly. Alternatively, if it is determined that there is a false penetration point, the on-vehicle control system 11 may restrict some or all of the driving automation functions that operate based on the sensing result of the information processing device 100. The present disclosure may have the following configurations.

[0157] (1) An acquisition unit that acquires image data captured by a 3D camera and point cloud data generated by a LiDAR, which at least partially overlaps with the angle of view of the image data and includes a plurality of points; a superimposing unit that generates a superimposed image by superimposing the point cloud data on the image data; a determination unit that identifies two points, both of which have a predetermined value in abscissa coordinate values ​​corresponding to an azimuth angle direction in the irradiation of the LiDAR on the superimposed image, from among the point cloud data superimposed on the image data, and determines that one of the two points is a false transmission point when one of the ordinate coordinate values ​​corresponding to an elevation / depression angle direction in the irradiation of the LiDAR on the superimposed image is a first value and the other is a second value different from the first value, and a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value; An information processing device comprising: (2) The method further includes a generating unit that removes the false transparent points and generates the superimposed image by superimposing the point cloud data excluding the removed false transparent points on the image data. The information processing device according to (1) above. (3) The two points are adjacent in the vertical axis coordinate. The information processing device according to (1) or (2) above. (4) When a difference between the first height information and the second height information is equal to or greater than a predetermined value, the determining unit determines that the point having the smaller vertical axis coordinate value among the two points is a false transmission point. An information processing device according to any one of (1) to (3) above. (5) The information processing device is mounted on a vehicle. An information processing device according to any one of (1) to (4) above. (6) The computer Acquire image data captured by a 3D camera and point cloud data generated by a LiDAR that at least partially overlaps with the angle of view of the image data and includes a plurality of points; generating a superimposed image by superimposing the point cloud data on the image data; Among the point cloud data superimposed on the image data, two points are identified whose horizontal axis coordinate values ​​corresponding to the azimuth angle direction in the irradiation of the LiDAR on the superimposed image both have a predetermined value, and one of the vertical axis coordinate values ​​corresponding to the elevation / depression angle direction in the irradiation of the LiDAR on the superimposed image is a first value and the other is a second value different from the first value, and when a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value, one of the two points is determined to be a false penetration point. Information processing methods. (7) The computer Acquiring image data captured by a 3D camera and point cloud data generated by a LiDAR, the point cloud data including a plurality of points that at least partially overlap with the angle of view of the image data; generating a superimposed image by superimposing the point cloud data on the image data; identifying two points, both of which have a predetermined value in abscissa coordinate values ​​corresponding to an azimuth angle direction in the irradiation of the LiDAR on the superimposed image, among the point cloud data superimposed on the image data, and determining that one of the two points is a false penetration point when one of the ordinate coordinate values ​​corresponding to an elevation / depression angle direction in the irradiation of the LiDAR on the superimposed image is a first value and the other is a second value different from the first value, and a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value; An information processing program that executes the above.

[0158] Although the embodiments and modified examples of the present technology have been described above, the present technology is not limited to the above-described embodiments, and it goes without saying that various modifications can be made within the scope of the gist of the present technology. [Explanation of symbols]

[0159] 1. Vehicle 100...Information processing device 110…Communications Department 120...Storage section 130...Control unit 131…Acquisition part 132…Judgment section 133…Generation part 140…Detection unit 150…LiDAR 160…Camera

Claims

1. An acquisition unit that acquires image data generated by capturing an image with a 3D camera and point cloud data generated by a LiDAR that at least partially overlaps with the angle of view of the image data and includes a plurality of points; a superimposing unit that generates a superimposed image by superimposing the point cloud data on the image data; a determination unit which identifies two points among the point cloud data superimposed on the image data, both of which have a predetermined value in abscissa coordinate values ​​corresponding to the azimuth angle direction in the LiDAR irradiation on the superimposed image, and which determines that one of the two points is a false transmission point when one of the ordinate coordinate values ​​corresponding to the elevation / depression angle direction in the LiDAR irradiation on the superimposed image is a first value and the other is a second value different from the first value, and a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value; An information processing device comprising:

2. 2. The information processing device according to claim 1, The method further includes a generating unit that removes the false transparent points and generates the superimposed image by superimposing the point cloud data excluding the removed false transparent points on the image data. Information processing device.

3. 2. The information processing device according to claim 1, The two points are adjacent in the vertical axis coordinate. Information processing device.

4. 2. The information processing device according to claim 1, When a difference between the first height information and the second height information is equal to or greater than a predetermined value, the determining unit determines that, of the two points, the point having a smaller value of the vertical axis coordinate is a false transmission point. Information processing device.

5. 2. The information processing device according to claim 1, The information processing device is mounted on a vehicle. Information processing device.

6. The computer Acquire image data captured by a 3D camera and point cloud data generated by a LiDAR, which at least partially overlaps with the angle of view of the image data and includes a plurality of points; generating a superimposed image by superimposing the point cloud data on the image data; Among the point cloud data superimposed on the image data, two points are identified whose horizontal axis coordinate values ​​corresponding to the azimuth angle direction in the irradiation of the LiDAR on the superimposed image both have a predetermined value, and one of the vertical axis coordinate values ​​corresponding to the elevation / depression angle direction in the irradiation of the LiDAR on the superimposed image is a first value and the other is a second value different from the first value, and if a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value, one of the two points is determined to be a false penetration point. Information processing methods.

7. The computer Acquiring image data captured by a 3D camera and point cloud data generated by a LiDAR, the point cloud data including a plurality of points and at least partially overlapping with the angle of view of the image data; generating a superimposed image by superimposing the point cloud data on the image data; identifying two points among the point cloud data superimposed on the image data, both of which have a predetermined value in abscissa coordinate values ​​corresponding to the azimuth angle direction in the LiDAR irradiation on the superimposed image, and determining that one of the two points is a false penetration point when one of the ordinate coordinate values ​​corresponding to the elevation / depression angle direction in the LiDAR irradiation on the superimposed image is a first value and the other is a second value different from the first value, and a difference between first height information of the 3D camera corresponding to the first value and second height information of the 3D camera corresponding to the second value is equal to or greater than a predetermined value; An information processing program that executes the above.

Images