Information Processing Method and Information Processing Apparatus

Abstract

On the basis of a captured image obtained by imaging the surroundings of a vehicle using an imaging unit, and point cloud data relating to ranging points around the vehicle, generated by a sensor, this information processing method and information processing device assign attribute information to each ranging point, and calculate a distance from the imaging unit to the ranging point and a direction of the ranging point as seen from the imaging unit, on the basis of the point cloud data. Furthermore, for each ranging point, a pixel value is calculated on the basis of the distance to the ranging point and the attribute information, and a two-dimensional image is generated in which a pixel corresponding to the direction of the ranging point has the pixel value.

Description

Technical Field

[0001] The present invention relates to an information processing method and an information processing apparatus.

Background Art

[0002] There has been proposed a technique for compressing the data volume of point cloud data with a large data volume by converting point cloud data output from a distance measurement sensor into a two-dimensional image having pixel values based on the position information of each distance measurement point in a three-dimensional space and compressing it by an existing image compression technique (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the technique described in Patent Document 1, since the pixel values are determined based only on the position information of each distance measurement point in the three-dimensional space obtained by the distance measurement sensor to generate a two-dimensional image, there is a problem that other attribute information of the position information of each distance measurement point cannot be retained in the two-dimensional image.

[0005] The present invention has been made in view of the above problems. An object thereof is to provide an information processing method and an information processing apparatus capable of generating a two-dimensional image that retains other attribute information in addition to the position information of each distance measurement point.

Means for Solving the Problems

[0006] In order to solve the above problems, an information processing method and an information processing apparatus according to an aspect of the present invention assign attribute information to each ranging point based on a captured image obtained by capturing the surroundings of a vehicle by an imaging unit or point cloud data regarding ranging points around the vehicle generated by a sensor, and calculate the distance from the imaging unit to the ranging point and the direction of the ranging point as seen from the imaging unit based on the point cloud data. Then, for each ranging point, a pixel value is calculated based on the distance to the ranging point and the attribute information, and a two-dimensional image is generated in which a pixel corresponding to the direction of the ranging point has the pixel value.

Advantages of the Invention

[0007] According to the present invention, it is possible to generate a two-dimensional image that holds other attribute information in addition to the position information of each ranging point.

Brief Description of the Drawings

[0008]

Fig. 1

Fig. 2

Fig. 3

Fig. 4A

Fig. 4B

Fig. 4C

Embodiments for Carrying Out the Invention

[0009] Next, embodiments of the present invention will be described in detail with reference to the drawings. In the description, the same components are denoted by the same reference numerals and redundant description is omitted.

[0010] [Configuration of Information Processing Apparatus] Referring to FIG. 1, a configuration example of the information processing apparatus 1 will be described. FIG. 1 is a block diagram showing the configuration of the information processing apparatus 1 according to the present embodiment. As shown in FIG. 1, the information processing apparatus 1 includes a sensor group 10 (sensors), a camera 11 (imaging unit), a map information acquisition unit 12, and a controller 20.

[0011] The information processing apparatus 1 may be mounted on a vehicle having an automatic driving function or a vehicle not having an automatic driving function. Further, the information processing apparatus 1 may be mounted on a vehicle capable of switching between automatic driving and manual driving. Further, the automatic driving function may be a driving support function that automatically controls only some of the vehicle control functions such as steering control, braking force control, and driving force control to assist the driver's driving. In the present embodiment, the information processing apparatus 1 will be described as being mounted on a vehicle having an automatic driving function.

[0012] Although omitted in FIG. 1, the information processing apparatus 1 may control various actuators such as a steering actuator, an accelerator pedal actuator, and a brake actuator based on the recognition result (position, shape, posture, etc. of an object) by the attribute information setting unit 24. Thereby, highly accurate automatic driving can be realized.

[0013] The sensor group 10 mainly includes sensors that measure the distance and direction to objects around the host vehicle. Such a sensor is, for example, a lidar (Laser Imaging Detection and Ranging). A lidar measures the distance and direction to an object and recognizes the shape of the object by emitting light (laser light) to an object around the host vehicle and measuring the time until the light (reflected light) hits the object and bounces back. Further, the lidar can also acquire the positional relationship of the objects three-dimensionally. It is also possible to perform mapping using the intensity of the reflected light.

[0014] For example, the lidar scans the surroundings of the host vehicle in the main scanning direction and the sub-scanning direction by changing the irradiation direction of light. As a result, light is sequentially irradiated onto a plurality of ranging points existing around the host vehicle. The irradiation of light to all the ranging points existing around the host vehicle is repeated at a predetermined time interval. The lidar generates information for each ranging point (ranging point information) obtained by the irradiation of light for each ranging point. Then, the lidar outputs point cloud data composed of a plurality of ranging point information to the controller 20.

[0015] The ranging point information includes the position information of the ranging point. The position information is information indicating the position coordinates of the ranging point. For the position coordinates, a polar coordinate system represented by the direction from the lidar to the ranging point (yaw angle, pitch angle) and the distance (depth) from the lidar to the ranging point may be used. For the position coordinates, a three-dimensional coordinate system represented by the x coordinate, y coordinate, and z coordinate with the installation position of the lidar as the origin may be used. Further, the ranging point information may include the time information of the ranging point. The time information is information indicating the time when the position information of the ranging point was generated (when the reflected light was received). In addition, the ranging point information may include information on the intensity of the reflected light from the ranging point (intensity information).

[0016] Note that the irradiation direction of light by the lidar may be controlled by the sensor control unit 27 described later.

[0017] In addition, the sensor group 10 may include a GPS receiver or a GNSS receiver that detects the position of the host vehicle. The sensor group 10 may also include a speed sensor, an acceleration sensor, a steering angle sensor, a gyro sensor, a brake hydraulic sensor, an accelerator opening sensor, etc. that detect the state of the host vehicle. The information acquired by the sensor group 10 is output to the controller 20. Hereinafter, unless otherwise specified, the lidar will be described as a representative of the sensor group 10, and the information output to the controller 20 will be described as the information acquired by the lidar.

[0018] The camera 11 has an imaging device such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor). The installation location of the camera 11 is not particularly limited. As an example, the camera 11 is installed in front of, on the side of, or behind the host vehicle. The camera 11 continuously images the surroundings of the host vehicle at a predetermined cycle. The camera 11 detects objects existing around the host vehicle (pedestrians, bicycles, motorcycles, other vehicles, etc.) and information in front of the host vehicle (lane lines, traffic lights, signs, crosswalks, intersections, etc.). The image captured by the camera 11 is output to the controller 20. Note that the image captured by the camera 11 is stored in a storage device (not shown), and the controller 20 may refer to the image stored in the storage device.

[0019] The area measured (detected) by the lidar and the area imaged (detected) by the camera 11 overlap entirely or partially. The lidar generates point cloud data regarding the ranging points included in the imaging range of the camera 11.

[0020] The map information acquisition unit 12 acquires map information indicating the structure of the road on which the vehicle travels. The map information acquired by the map information acquisition unit 12 may include the position information of traffic lights, the types of traffic lights, the position of the stop line corresponding to the traffic lights, etc. The map information acquisition unit 12 may own a map database storing the map information, or may acquire the map information from an external map data server by cloud computing. Further, the map information acquisition unit 12 may acquire the map information using vehicle-to-vehicle communication or vehicle-to-roadside communication.

[0021] The map information acquired by the map information acquisition unit 12 may include information on the road structure such as the absolute position of the lane, the connection relationship of the lanes, and the relative position relationship. Further, the map information acquired by the map information acquisition unit 12 may also include facility information such as parking lots and gas stations.

[0022] The road information around the host vehicle is composed of the image captured by camera 11, the information obtained by the sensor group 10, and the map information obtained by the map information acquisition unit 12. In addition, the road information around the host vehicle may be obtained from outside the host vehicle by vehicle-to-vehicle communication and road-to-vehicle communication, in addition to being obtained by the acquisition unit.

[0023] The controller 20 is a general-purpose microcomputer including a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program for causing the microcomputer to function as the information processing device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits included in the information processing device 1. The controller 20 processes the data acquired from the sensor group 10 and the camera 11.

[0024] Here, an example is shown in which a plurality of information processing circuits included in the information processing device 1 are realized by software. Of course, it is also possible to prepare dedicated hardware for executing each of the information processes described below to configure the information processing circuit. Also, a plurality of information processing circuits may be configured by individual hardware.

[0025] The controller 20 includes, as an example of a plurality of information processing circuits (information processing functions), a point cloud acquisition unit 21, a coordinate conversion unit 22, an image acquisition unit 23, an attribute information setting unit 24, a pixel value calculation unit 25, an image generation unit 26, and a sensor control unit 27. Note that the controller 20 may be expressed as an ECU (Electronic Control Unit).

[0026] The point cloud acquisition unit 21 acquires point cloud data from the lidar. The point cloud acquisition unit 21 outputs the acquired point cloud data to the coordinate conversion unit 22.

[0027] Based on the acquired point cloud data, the coordinate conversion unit 22 calculates the distance from the camera 11 to the measured point and the direction of the measured point as seen from the camera 11. For example, the coordinate conversion unit 22 may convert the position information based on the installation position of the lidar into the position information based on the position of the camera 11 based on the deviation between the installation positions of the lidar and the camera 11.

[0028] Also, the coordinate conversion unit 22 may convert the position information based on the installation position of the lidar into the position information based on the position of the camera 11 based on the imaging timing by the camera 11, the generation timing of the point cloud data by the lidar, and the movement information of the host vehicle (the movement direction and vehicle speed of the host vehicle). The process of converting the position information based on these timings is useful when the imaging timing by the camera 11 and the generation timing of the point cloud data by the lidar are not synchronized.

[0029] The image acquisition unit 23 acquires the captured image captured by the camera 11. The image acquisition unit 23 outputs the acquired captured image to the attribute information setting unit 24. If the region where there is a high possibility that the object to be recognized exists is known, the image acquisition unit 23 may extract and output only that region.

[0030] The attribute information setting unit 24 performs recognition processing on the captured image acquired from the image acquisition unit 23 for objects and signs (hereinafter referred to as "objects, etc.") around the vehicle. This recognition processing is an example of image processing, and is a process of detecting and identifying objects, etc. around the host vehicle (mainly in front of the host vehicle), and associating attribute information, which is a value that uniquely identifies the objects, etc., with each pixel. Such image processing can use, for example, semantic segmentation that estimates the likelihood of each object, etc. for each pixel. Also, attributes may be classified and identified according to the type and color of the objects, etc. In the present embodiment, the "objects, etc." include, for example, moving objects such as passenger cars, trucks, buses, motorcycles, pedestrians, etc., and stationary objects such as pylons, small animals, falling objects, buildings, walls, guardrails, frames at construction sites, etc. Furthermore, the "objects, etc." may include road markings marked on the road surface with a special paint for guiding, guiding, warning, regulating, instructing, etc. for road traffic. Examples of road markings include lane lines (broken white lines), crosswalks, stop lines, direction-of-travel arrow lines, etc.

[0031] Note that since the attribute information setting unit 24 associates attribute information with each image constituting the captured image acquired from the image acquisition unit 23, as a result, the attribute information setting unit 24 can assign attribute information to each ranging point reflected in the captured image.

[0032] In addition, the attribute information setting unit 24 may perform recognition processing on objects and signs (hereinafter referred to as objects, etc.) around the vehicle based on point cloud data. In particular, since the shape of the objects around the vehicle is represented by the point cloud data, the attribute information setting unit 24 may assign attribute information to each ranging point using a method such as pattern matching. In this way, the attribute information setting unit 24 assigns attribute information to each ranging point based on the captured image or the point cloud data.

[0033] The pixel value calculation unit 25 calculates a pixel value for a pixel corresponding to a distance measurement point based on the distance to the distance measurement point and the attribute information assigned to the distance measurement point. More specifically, the pixel value calculation unit 25 calculates a pixel value based on distance information corresponding to the distance to the distance measurement point and attribute information, which is a value that uniquely identifies an object or the like reflected in the captured image. In addition, the pixel value calculation unit 25 may calculate a pixel value based on intensity information indicating the intensity of the reflected light from the distance measurement point. Note that the pixel value calculation unit 25 identifies a pixel corresponding to the distance measurement point in the captured image based on the direction of the distance measurement point as viewed from the camera 11, which is calculated by the coordinate conversion unit 22.

[0034] An example of the calculation of the pixel value by the pixel value calculation unit 25 will be described with reference to FIG. 3. FIG. 3 is a diagram showing a configuration example of the pixel value set for each pixel corresponding to the distance measurement point.

[0035] In FIG. 3, it is shown that the distance information, the intensity information, and the attribute information are each 8-bit data, and 24-bit data representing the pixel value is configured by concatenating these pieces of information. In this way, the pixel value calculation unit 25 combines the data of the distance information, the intensity information, and the attribute information to calculate the pixel value. Note that the pixel value calculation unit 25 may combine the data of the distance information and the attribute information to calculate the pixel value.

[0036] In addition, in FIG. 3, an example in which the pixel value is 24-bit data is shown, but the present invention is not limited to this. The pixel value may have a bit number other than 24 bits. An example in which the distance information, the intensity information, and the attribute information are each 8-bit data is shown, but the present invention is not limited to this. These pieces of information may have a bit number other than 8 bits.

[0037] Note that, after the pixel value calculation unit 25 sets the correspondence between the distance to the distance measurement point and the pixel value, it may calculate distance information (pixel value) corresponding to the distance to the distance measurement point based on the correspondence (hereinafter, the distance information and the pixel value will be described without particular distinction). More specifically, when the pixel value calculation unit 25 discretizes the distance and represents the distance by distance information consisting of a predetermined number of bits, it may change the resolution of the distance according to the distance. An example of the correspondence set by the pixel value calculation unit 25 will be described with reference to FIGS. 4A, 4B, and 4C.

[0038] FIG. 4A is a diagram showing a first example of the correspondence between the distance and the distance information. In FIG. 4A, the state where the value (pixel value) indicated by the distance information increases in proportion to the distance is shown. That is, according to the correspondence shown in FIG. 4A, the resolution is constant regardless of the distance.

[0039] FIG. 4B is a diagram showing a second example of the correspondence between the distance and the distance information. In FIG. 4B, the state where the change amount of the value (pixel value) indicated by the distance information per unit change amount of the distance becomes larger as the distance is shorter is shown. That is, according to the correspondence shown in FIG. 4B, the resolution becomes higher as the distance is shorter.

[0040] FIG. 4C is a diagram showing a third example of the correspondence between the distance and the distance information. In FIG. 4C, compared with the case where the distance is not within the predetermined range, when the distance is within the predetermined range (in FIG. 3C, the distance is in the medium range), the change amount of the value (pixel value) indicated by the distance information per unit change amount of the distance becomes larger. Also, the state where the change amount of the value indicated by the distance information per unit change amount of the distance becomes smaller when the distance is not within the predetermined range is shown. That is, according to the correspondence shown in FIG. 4C, the resolution becomes higher when the distance is within the predetermined range, and the resolution becomes lower when the distance is not within the predetermined range.

[0041] Thus, the pixel value calculation unit 25 may set various correspondence relationships. Note that the pixel value calculation unit 25 may set the correspondence relationship based on the situation around the vehicle or the behavior of the vehicle.

[0042] For example, in order to accurately represent the distance for the distance measurement points near the vehicle, the pixel value calculation unit 25 may set the correspondence relationship between the distance and the distance information such that the change amount of the distance information (pixel value) per unit change amount of the distance becomes larger as the distance becomes shorter. For example, the pixel value calculation unit 25 may set a correspondence relationship that increases the distance resolution in the vicinity of the vehicle when the host vehicle decelerates or when there is another vehicle approaching in front of the host vehicle.

[0043] Also, the pixel value calculation unit 25 may select the first attribute information from among the attribute information based on the behavior of the vehicle. Then, the pixel value calculation unit 25 may set the correspondence relationship between the distance and the pixel value such that the change amount of the pixel value per unit change amount of the distance at the distance to the distance measurement point to which the first attribute information is assigned becomes larger than a predetermined value.

[0044] For example, depending on the behavior of the vehicle, the object for which it is necessary to accurately represent the distance may vary. When the vehicle changes lanes, the lane marker indicating the boundary with the adjacent lane, the curb on the side of the road, the guardrail, etc. become the objects. Also, when the vehicle accelerates or decelerates, the stop line, traffic signal, other vehicles, etc. in front of the host vehicle become the objects. The pixel value calculation unit 25 may determine a predetermined range based on the distance at which such an object is located, and may set a correspondence relationship that increases the distance resolution within the predetermined range.

[0045] Furthermore, when the speed of the vehicle is greater than a predetermined speed, the pixel value calculation unit 25 may set the correspondence relationship between the distance and the pixel value such that the change amount of the pixel value per unit change amount of the distance at a distance greater than a predetermined distance is greater than a predetermined value. When the speed of the vehicle is high, there may be a case where it is necessary to accurately represent the distance for an object at a greater distance from the vehicle compared to an object at a smaller distance from the vehicle. Therefore, when the speed of the vehicle is greater than a predetermined speed, the pixel value calculation unit 25 may set a correspondence relationship that increases the distance resolution in the far field of the vehicle.

[0046] The image generation unit 26 generates a two-dimensional image based on the direction of the distance measurement point and the pixel value calculated by the pixel value calculation unit 25 for the distance measurement point. More specifically, the image generation unit 26 generates a two-dimensional image such that the pixel corresponding to the direction of the distance measurement point viewed from the camera 11 has the pixel value calculated by the pixel value calculation unit 25.

[0047] When the number of distance measurement points by the lidar is not sufficiently large, since the number of pixels corresponding to the distance measurement points is smaller than the number of pixels constituting the captured image, the image generation unit 26 may generate a two-dimensional image using a dummy value indicating that the pixel is not a processing target as the pixel value for pixels in the two-dimensional image that do not correspond to the distance measurement points.

[0048] Note that the two-dimensional image generated by the image generation unit 26 may be output outside the information processing apparatus 1 and used for the vehicle control function. When outputting the two-dimensional image outside, information specifying the correspondence relationship set in the pixel value calculation unit 25 may be output outside the information processing apparatus 1.

[0049] The sensor control unit 27 controls the sensor group 10. In particular, the sensor control unit 27 controls the irradiation direction of light by the lidar and increases the spatial density of measurement points in an area where the number of measurement points is insufficient (hereinafter referred to as the "specific area"). That is, the sensor control unit 27 performs control of the lidar and increases the spatial density of measurement points in the specific area after control as compared with the spatial density of measurement points in the specific area before control.

[0050] Examples of the specific area include measurement points to which previously set second attribute information is assigned and areas in the vicinity of the measurement points. In this case, the sensor control unit 27 may determine whether the second attribute information is included in the attribute information, and when it is determined that the second attribute information is included, control the lidar so as to increase the spatial density of measurement points at the position of the measurement points to which the second attribute information is assigned.

[0051] Also, examples of the specific area include specific objects existing around the vehicle. Specific objects include, for example, stationary objects such as pylons, small animals, falling objects, buildings, walls, guardrails, and frames at construction sites. The presence or absence of stationary objects may be determined based on the map information acquired by the map information acquisition unit 12. That is, the sensor control unit 27 may determine whether a specific object exists around the vehicle based on the map information, and when it is determined that a specific object exists, control the lidar so as to increase the spatial density of measurement points at the position of the specific object.

[0052] Finally, a specific example of the control of the lidar by the sensor control unit 27 is shown. For example, the sensor control unit 27 increases the number of scans in the main scanning direction or the sub-scanning direction by the lidar to increase the number of light irradiations per unit solid angle for a specific area around the host vehicle. As a result, the spatial density of measurement points in the specific area becomes higher than the spatial density of measurement points in other areas. By increasing the spatial density of measurement points in the specific area, it is possible to accurately detect the position and identify the object located in the specific area.

[0053] [Processing Procedure of Information Processing Device] Next, the processing procedure of the information processing apparatus 1 according to the present embodiment will be described with reference to the flowchart of FIG. 2. FIG. 2 is a flowchart showing the processing of the information processing apparatus 1 according to the present embodiment. The processing of the information processing apparatus 1 shown in FIG. 2 may be repeatedly executed at a predetermined cycle.

[0054] First, in step S101, the image acquisition unit 23 acquires a captured image captured by the camera 11.

[0055] In step S103, the point cloud acquisition unit 21 acquires point cloud data from the lidar.

[0056] In step S105, the attribute information setting unit 24 assigns attribute information to each distance measurement point.

[0057] In step S107, the coordinate conversion unit 22 calculates the distance from the camera 11 to the distance measurement point and the direction of the distance measurement point as viewed from the camera 11 based on the acquired point cloud data.

[0058] In step S109, the pixel value calculation unit 25 sets the correspondence between the distance to the distance measurement point and the pixel value.

[0059] In step S111, the pixel value calculation unit 25 calculates the pixel value for the pixel corresponding to the distance measurement point based on the distance to the distance measurement point and the attribute information assigned to the distance measurement point.

[0060] In step S113, the image generation unit 26 generates a two-dimensional image based on the direction of the distance measurement point and the pixel value calculated by the pixel value calculation unit 25 for the distance measurement point.

[0061] In step S115, the sensor control unit 27 determines whether there is an area with insufficient distance measurement points. If it is determined that there is an area with insufficient distance measurement points (YES in step S115), then in step S117, the sensor control unit 27 controls the sensor group 10 to increase the spatial density of the distance measurement points in the area with insufficient distance measurement points.

[0062] On the other hand, if it is determined that there is no area with insufficient distance measurement points (NO in step S115), then in step S119, it is determined whether to continue the process of generating a two-dimensional image. If it is determined to continue the process (YES in step S119), the process returns to step S101. If it is determined not to continue the process (NO in step S119), the flowchart in FIG. 2 is terminated.

[0063] [Effect of Embodiment] As described in detail above, the information processing method and information processing apparatus according to the present embodiment assign attribute information to each distance measurement point based on the captured image obtained by imaging the surroundings of the vehicle by the imaging unit or the point cloud data regarding the distance measurement points around the vehicle generated by the sensor, and calculate the distance from the imaging unit to the distance measurement point and the direction of the distance measurement point as seen from the imaging unit based on the point cloud data. Then, for each distance measurement point, a pixel value is calculated based on the distance to the distance measurement point and the attribute information, and a two-dimensional image is generated in which pixels corresponding to the direction of the distance measurement point have the pixel value.

[0064] Thereby, a two-dimensional image that retains other attribute information in addition to the position information of each distance measurement point can be generated. Also, the attribute information of an object that can be recognized only from the captured image by the camera 11, which cannot be discriminated from the sensor group 10, can be output together with the three-dimensional position information of each distance measurement point. Further, since it is generated as a two-dimensional image, the data capacity can be reduced using existing image compression techniques.

[0065] Specifically, when the camera 11 is used as compared with the lidar, an imaging image with high resolution and including color information can be obtained. Therefore, stationary objects such as pylons, small animals, falling objects, buildings, walls, guardrails, frames at construction sites, etc., and signs with meanings such as white lines, signs, and traffic lights can be recognized more efficiently using the imaging image. Therefore, by assigning attribute information based on the imaging image and outputting it together with the position information of each ranging point, it can be used to determine whether the recognized object or the like affects the running of the vehicle.

[0066] Further, the information processing method and the information processing apparatus according to the present embodiment may set the correspondence relationship between the distance and the pixel value such that the change amount of the pixel value per unit change amount of the distance becomes larger as the distance becomes shorter. Thereby, the distance of an object or a road shape in the region near the host vehicle that has a great influence on the control of the vehicle can be accurately expressed. As a result, more appropriate vehicle control can be performed.

[0067] Furthermore, the information processing method and the information processing apparatus according to the present embodiment may set the correspondence relationship between the distance and the pixel value such that the change amount of the pixel value per unit change amount of the distance at a distance greater than a predetermined distance becomes larger than a predetermined value when the speed of the vehicle is greater than a predetermined speed. Thereby, when the speed of the vehicle is high, the distance of an object with a large distance from the vehicle can be accurately expressed as compared with an object with a small distance from the vehicle. As a result, more appropriate vehicle control can be performed.

[0068] In addition, the information processing method and information processing apparatus according to the present embodiment may select first attribute information from among the attribute information based on the behavior of the vehicle, and set the correspondence relationship between the distance and the pixel value such that the amount of change in the pixel value per unit change in the distance at the distance to the measurement point to which the first attribute information is assigned becomes larger than a predetermined value. In vehicle control, the region where the position accuracy of the object recognition result is important changes according to the behavior of the vehicle (such as vehicle speed and yaw rate). Therefore, by selecting the region where the position accuracy of the object recognition result is important based on the behavior of the vehicle and increasing the resolution of the distance in that region, the distance can be expressed accurately. As a result, more appropriate vehicle control can be performed.

[0069] Furthermore, the information processing method and information processing apparatus according to the present embodiment may determine whether second attribute information is included in the attribute information, and control the sensor to increase the spatial density of the measurement points at the positions of the measurement points to which the second attribute information is assigned when it is determined that the second attribute information is included. Thereby, it is possible to improve the detection position accuracy of an object that has a great influence on vehicle control, such as a vehicle approaching the host vehicle or a pedestrian crossing the path of the host vehicle.

[0070] In addition, an object smaller than or thinner than the interval between the irradiation directions of the waves (laser light, millimeter wave, etc.) of the lidar may not be detected because the light is not irradiated, but there is a possibility that it can be detected by the camera 11 with higher resolution than the lidar. Therefore, by controlling the irradiation direction of the lidar light based on the image recognition result, it is possible to obtain position information of an object smaller than or thinner than the interval between the irradiation directions of the light by the lidar.

[0071] In addition, the information processing method and information processing apparatus according to the present embodiment may determine whether a specific object exists around the vehicle based on the map information, and control the sensor to increase the spatial density of the measurement points at the position of the specific object when it is determined that the specific object exists. Thereby, it is possible to improve the detection position accuracy of an object that has a great influence on vehicle control, such as a stop line or a crosswalk.

[0072] Furthermore, the information processing method and the information processing apparatus according to the present embodiment may calculate position information of ranging points based on the imaging timing by the imaging unit, the generation timing of point cloud data by the sensor, and the movement information of the vehicle, with reference to the position of the imaging unit at the imaging timing by the imaging unit, and calculate the distance and direction based on the position information. When the imaging timing by the camera 11 and the generation timing of the point cloud data by the lidar are not synchronized, if the host vehicle moves, the position where the captured image is acquired and the position where the point cloud data is acquired will deviate. However, by correcting the position information of the ranging points based on these timings, the deviation can be corrected.

[0073] Each function shown in the above-described embodiment can be implemented by one or more processing circuits. The processing circuit includes a programmed processor, an electric circuit, etc., and further includes a device such as an application-specific integrated circuit (ASIC) and circuit components arranged to execute the described functions.

[0074] Although the content of the present invention has been described along with the embodiments above, it is obvious to those skilled in the art that the present invention is not limited to these descriptions and various modifications and improvements are possible. It should not be understood that the discussions and drawings forming part of this disclosure limit the present invention. Various alternative embodiments, examples, and operation techniques will become apparent to those skilled in the art from this disclosure.

[0075] Needless to say, the present invention includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention-specific matters according to the legitimate claims based on the above description.

Explanation of Reference Numerals

[0076] 1 Information processing apparatus 10 Sensor group (sensor) 11 Camera (imaging unit) 12 Map information acquisition unit 20 Controller 21 Point cloud acquisition unit 22 Coordinate conversion unit 23 Image acquisition unit 24 Attribute information setting unit 25 Pixel value calculation unit 26 Image generation unit 27 Sensor control unit

Claims

1. An imaging unit that images the surroundings of a vehicle and generates an imaging image; A sensor that generates point cloud data regarding a distance measurement point included in the imaging range of the imaging unit; An information processing method for an information processing apparatus including a controller that processes data acquired from the imaging unit and the sensor, the method comprising: by the controller, based on the imaging image or the point cloud data, attribute information indicating the type of an object or a sign reflected in the imaging image is assigned to each distance measurement point; based on the point cloud data, the distance from the imaging unit to the distance measurement point and the direction of the distance measurement point as seen from the imaging unit are calculated; for each distance measurement point, a pixel value is calculated based on the distance and the attribute information; generating a two-dimensional image in which pixels corresponding to the direction have the pixel value An information processing method characterized by the above.

2. The information processing method according to Claim 1, wherein by the controller, a correspondence relationship between the distance and the pixel value is set such that as the distance becomes shorter, the amount of change in the pixel value per unit change in the distance becomes larger. An information processing method characterized by the above.

3. The information processing method according to Claim 1 or 2, wherein by the controller, when the speed of the vehicle is greater than a predetermined speed, a correspondence relationship between the distance and the pixel value is set such that the amount of change in the pixel value per unit change in the distance at a distance greater than a predetermined distance is greater than a predetermined value. An information processing method characterized by the above.

4. The information processing method according to Claim 1 or 2, wherein by the controller, based on the behavior of the vehicle, first attribute information is selected from among the attribute information; a correspondence relationship between the distance and the pixel value is set such that the amount of change in the pixel value per unit change in the distance at the distance to the distance measurement point to which the first attribute information is assigned is greater than a predetermined value. An information processing method characterized by the above.

5. The information processing method according to Claim 1 or 2, wherein by the controller, it is determined whether second attribute information is included in the attribute information; when it is determined that the second attribute information is included, the sensor is controlled so as to increase the spatial density of the distance measurement points at the positions of the distance measurement points to which the second attribute information is assigned. An information processing method characterized by the above.

6. The information processing method according to Claim 1 or 2, wherein by the controller, Determine whether a specific object exists around the vehicle based on map information, and control the sensor to increase the spatial density of the ranging points at the position of the specific object when it is determined that the specific object exists. An information processing method characterized by the above.

7. The information processing method according to claim 1 or 2, wherein the controller, based on the imaging timing by the imaging unit, the generation timing of the point cloud data by the sensor, and the movement information of the vehicle, calculates the position information of the ranging points with reference to the position of the imaging unit at the imaging timing by the imaging unit, and calculates the distance and the direction based on the position information. An information processing method characterized by the above.

8. An imaging unit that images the surroundings of a vehicle to generate an imaging image, a sensor that generates point cloud data regarding ranging points included in the imaging range of the imaging unit, and a controller that processes data acquired from the imaging unit and the sensor, wherein the controller, based on the imaging image or the point cloud data, assigns attribute information indicating the type of an object or a sign reflected in the imaging image to each ranging point, calculates the distance from the imaging unit to the ranging point and the direction of the ranging point as seen from the imaging unit based on the point cloud data, calculates a pixel value for each ranging point based on the distance and the attribute information, and generates a two-dimensional image in which pixels corresponding to the direction have the pixel value. An information processing apparatus characterized by the above.

Images