Information processing device, information processing method, and program

The information processing device enhances recognition accuracy in adverse environments by using LiDAR with SPAD pixels and 3D map data from good conditions, addressing the challenge of reduced accuracy in adverse weather or night conditions for automatic driving and driving support systems.

WO2024241896A1PCT designated stage expired Publication Date: 2024-11-28SONY GROUP CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017252
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-09
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing technologies for automatic driving and driving support, such as those using LiDAR for semantic segmentation, face challenges in maintaining recognition accuracy in adverse environments like night or bad weather, where visibility is reduced and recognition accuracy decreases.

Method used

An information processing device and method that determines the environmental state and adjusts recognition processing accordingly, using a LiDAR system with SPAD pixels for distance measurement and communication with an external device to transmit detection and recognition results, and in adverse conditions, utilizes a 3D map created in good environments to enhance recognition accuracy.

Benefits of technology

The system achieves high accuracy in recognition processing even in bad environments by leveraging 3D map data from good conditions, improving the robustness of automatic driving and driving support systems across various environmental conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017252_28112024_PF_FP_ABST
    Figure JP2024017252_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This information processing device comprises: an environment determination unit that determines the state of an environment; a recognition processing unit that performs recognition processing on the basis of a detection result of a sensor; and a communication unit that performs communication processing with an external device. When the environment is in a first state, the recognition processing unit performs the recognition processing on the basis of the detection result of the sensor, and the communication unit transmits the detection result of the sensor and the recognition result of the recognition processing unit to the external device.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program.

[0002] Currently, there is a technology that performs recognition processing such as semantic segmentation based on the detection results of LiDAR (light detection and ranging) for the purpose of improving the accuracy of autonomous driving and driver assistance in automobiles. LiDAR is a sensing technology that can detect the distance to an object and the properties of the object. In recent years, in addition to sensors such as in-vehicle cameras and millimeter-wave radar, LiDAR, which can detect the position and shape of objects such as vehicles and pedestrians with high accuracy, has become increasingly important.

[0003] Semantic segmentation is a technology that labels each pixel that makes up an image with the information it represents. Pixels are classified according to which category they belong to, and then labeled and associated with categories to indicate what is depicted. Semantic segmentation makes it possible to attach information to each pixel that describes the type of subject it constitutes.

[0004] In order to improve the accuracy of surrounding environment recognition for automatic driving and driving assistance of automobiles, a technology has been proposed in which multiple recognition models for recognizing the surrounding environment are selected based on environmental data of the surrounding environment (Patent Document 1).

[0005] Japanese Patent Application Laid-Open No. 2022-176816

[0006] However, with the technology of Patent Document 1, even if a recognition model is selected, sensing is performed in adverse environments such as at night or in bad weather, where the accuracy of recognition and sensing is reduced, and therefore it can be said that this technology is insufficient in terms of improving recognition accuracy in adverse environments.

[0007] The present technology has been developed in consideration of such problems, and aims to provide an information processing device, an information processing method, and a program that are capable of performing recognition processing with high accuracy even in adverse environments.

[0008] In order to solve the above-mentioned problems, the first technology is an information processing device that includes an environment determination unit that determines the state of the environment, a recognition processing unit that performs recognition processing based on the detection results of the sensor, and a communication unit that performs communication processing with an external device, and when the environment is in a first state, the recognition processing unit performs recognition processing based on the detection results of the sensor, and the communication unit transmits the detection results of the sensor and the recognition results of the recognition processing unit to the external device.

[0009] The second technology is an information processing method that determines the state of the environment, performs recognition processing based on the detection results of a sensor, and performs communication processing with an external device, and when the environment is in a first state, performs recognition processing based on the detection results of the sensor, and transmits the detection results of the sensor and the recognition results of the recognition processing to the external device.

[0010] Furthermore, the third technology is a program that causes a computer to execute an information processing method that determines the state of the environment, performs recognition processing based on the detection results of a sensor, and performs communication processing with an external device, in which, when the environment is in a first state, recognition processing is performed based on the detection results of the sensor, and the detection results of the sensor and the recognition results of the recognition processing are transmitted to the external device.

[0011] Furthermore, the fourth technology is an information processing device that includes a 3D map creation unit that creates a 3D map based on sensor detection results and recognition results transmitted from an external device when the environment is in a first state, a recognition information creation unit that creates recognition information from the 3D map when the environment is in a second state, and a communication unit that transmits the recognition information to the external device.

[0012] Furthermore, the fifth technology is an information processing method that generates a 3D map based on sensor detection results and recognition results transmitted from an information processing device when the environment is in a first state, and creates recognition information from the 3D map when the environment is in a second state, and transmits the recognition information to the information processing device.

[0013] Furthermore, the sixth technology is a program that causes a computer to execute an information processing method that generates a 3D map based on sensor detection results and recognition results transmitted from an information processing device when the environment is in a first state, and creates recognition information from the 3D map when the environment is in a second state and transmits the recognition information to the information processing device.

[0014] FIG. 1 is a block diagram showing the configuration of an information processing system 10. FIG. 2 is an explanatory diagram of an overview of direct time of flight. FIG. 3 is an explanatory diagram of an overview of direct time of flight. FIG. 4 is an explanatory diagram of SPAD. FIG. 5 is an explanatory diagram of output from LiDAR 30. FIG. 6 is an explanatory diagram of output from LiDAR 30. FIG. 7 is an explanatory diagram of output from LiDAR 30. FIG. 8 is a flowchart showing processing in an information processing device 100. FIG. 9 is an explanatory diagram of processing in an information processing device 100. FIG. 10 is a flowchart showing processing in a server 200. FIG. 11 is an explanatory diagram of processing in an information processing device 100. FIG. 12 is an explanatory diagram of processing in a server 200. FIG. 13 is an explanatory diagram of a modified example.

[0015] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order: <Embodiment> [Description of LiDAR 30 using SPAD] [Configuration of information processing device 100] [Configuration of server 200] [Processing in information processing system 10] <Modification>

[0016] 1 , the information processing system 10 is configured with an automobile 20, a LiDAR 30, a position information acquisition unit 40, an information processing device 100, and a server 200. The external device for the information processing device 100 is the server 200, and the external device for the server 200 is the information processing device 100.

[0017] The LiDAR 30 is connected to the information processing device 100. The LiDAR 30 may be connected to the information processing device 100 by wire or wirelessly. Wired connection methods include, for example, HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus), and wireless connection methods include, for example, Wi-Fi, Bluetooth (registered trademark), and NFC (Near Field Communication).

[0018] In this embodiment, the LiDAR 30 is installed as an on-board sensor on the body of a moving vehicle, an automobile 20, and senses the surrounding environment of the automobile 20, such as the area in front of the automobile 20.

[0019] The LiDAR 30 measures scattered light in response to irradiation with pulsed laser light, and detects the distance to the target object or substance, as well as the properties and type of the object.

[0020] In this embodiment, the LiDAR 30 measures distance using a stacked direct Time of Flight (dToF) method that uses SPAD (Single Photon Avalanche Diode) pixels, which are single photon avalanche diodes. The SPAD pixel is used as one of the light receiving elements of the dToF method, which is one of the LiDAR ranging methods, and measures distance by using a detector to detect the time of flight (time difference) of light from a light source until it reflects off an object and returns, as shown in FIG.

[0021] In the dToF LiDAR 30, as shown in Figure 3, incident photons are collected and received by the SPAD, where they undergo photoelectric conversion and charge multiplication. Next, voltage conversion is performed, charge multiplication is stopped, and waveform shaping is performed. Next, column transfer is performed, time-to-digital conversion is performed, and signal peaks are extracted from the histogram, and finally a depth map is output.

[0022] An overview of the SPAD compared with the PD (PhotoDiode) and APD (Avalanche PhotoDiode) is shown in Figure 4. The SPAD is a device that has higher sensitivity than the PD and APD and can detect single photons.

[0023] As shown in Figure 5, LiDAR 30 using SPAD pixels can obtain depth (depth information) as output data from the peak in the histogram of the time it takes for laser light to return for each pixel and the brightness (number of photon counts).

[0024] As shown in Fig. 6A, the structure of one frame of output data from the LiDAR 30 is three-dimensional data represented by S(u, v, d). Also, as shown in Fig. 6B, the depth D is expressed by the following equation 1 using the three-dimensional data.

[0025] [Formula 1]

[0026] In addition, the intensity image (Intensity), which is the output data of the LiDAR 30, can be converted into an ambient light image (Ambient).

[0027] The intensity image can be converted into an ambient light image by integrating multiple intensity images as output data from the LiDAR 30, as shown in Figure 7. An ambient light image is an integrated histogram for each pixel, and can be converted into an ambient light image by integrating multiple intensity images in the output data from the LiDAR 30. An intensity image is an image composed of peak values ​​of the histogram for each pixel. For convenience of explanation, Figure 7 shows only short-distance, medium-distance, and long-distance intensity images included in the output data from the LiDAR 30, but when actually integrating, it is preferable to integrate all intensity images included in the output data from the LiDAR 30.

[0028] The LiDAR 30 using SPAD pixels can obtain distance information based on the time it takes for the infrared laser light emitted from the light-emitting unit to be received by the image sensor in the light-receiving unit, and can also obtain the reflection intensity of the subject from the intensity of the infrared light received by the light-receiving unit. Furthermore, because the timing of light reception varies depending on the subject distance, the light-receiving unit must be left open for a certain period of time, and as a side effect, infrared ambient light can also be obtained. The distance information is obtained as a two-dimensional depth map, and the infrared reflected light and infrared ambient light are obtained as two-dimensional images. Hereinafter, the two-dimensional image of infrared reflected light will be referred to as the infrared reflected light image, and the two-dimensional image of infrared ambient light will be referred to as the infrared ambient light image. This information can be used as input for recognition processes such as semantic segmentation.

[0029] The location information acquisition unit 40 is a sensor such as a GPS (Global Positioning System) or a GNSS (Global Navigation Satellite System) for acquiring the location of the automobile 20 having the functions of the information processing device 100. The location information acquisition unit 40 is connected to the information processing device 100. The location information acquisition unit 40 supplies the acquired location information to the communication unit 104 of the information processing device 100. The location information acquisition unit 40 may be connected to the information processing device 100 via a wired or wireless connection. Examples of wired connection methods include HDMI (registered trademark) and USB, and examples of wireless connection methods include Wi-Fi, Bluetooth (registered trademark), and NFC. The location information acquisition unit 40 may be a function that the automobile 20 has in advance, or may be mounted on the automobile 20 as a device separate from the automobile 20. The location information acquisition unit 40 acquires location information in synchronization with the LiDAR 30 using a predetermined synchronization signal or the like to detect the location of the automobile 20 at the time when the LiDAR 30 performed detection.

[0030] 1, the configuration of the information processing device 100 will be described. The information processing device 100 includes a data acquisition unit 101, an environment determination unit 102, a recognition processing unit 103, and a communication unit 104.

[0031] The data acquisition unit 101 acquires the detection results output from the LiDAR 30. As described above, the detection results of the LiDAR 30 are obtained as a depth map, an infrared reflected light image, and an infrared ambient light image. The data acquisition unit 101 converts the depth map into 3D point cloud information and supplies it to the communication unit 104. Structurally, the LiDAR 30 using SPAD pixels has a design in which the direction of the light incident on each pixel is known. Using this design data, the depth map can be converted into a 3D point cloud.

[0032] The data acquisition unit 101 also supplies the infrared reflected light image and the infrared ambient light image to the recognition processing unit 103. Note that the conversion of the depth map into 3D point cloud information may be performed by a dedicated processing unit, or the depth map may be transmitted to the server 200, and the server may convert the depth map into 3D point cloud information. Note that the data acquisition unit 101 may also supply the depth map to the recognition processing unit 103 in addition to the infrared reflected light image and the infrared ambient light image. It is considered that accuracy can be improved by also using the depth map in semantic segmentation as part of the recognition processing by the recognition processing unit 103.

[0033] The environment determination unit 102 determines whether the surrounding environment of the automobile 20 is a good environment or a bad environment. A good environment is the first environmental state in the claims, and a bad environment is the second environmental state in the claims.

[0034] A good environment is an environment that is bright and has good visibility, and in which recognition processing in the recognition processing unit 103 can be performed with high accuracy based on the detection results of the LiDAR 30. A good environment is, for example, a time of day such as morning or daytime, and sunny weather. A bad environment is an environment that is dark and has poor visibility, and the detection results of the LiDAR 30 are also dark and of poor quality, resulting in reduced recognition accuracy in the recognition processing unit 103. A bad environment is, for example, a time of day such as evening or night, and poor weather such as rain, snow, or fog. In a bad environment, the visibility of the input infrared ambient light is reduced, and therefore it is expected that the performance of the recognition processing will also decrease.

[0035] The environment determination unit 102 determines whether the surrounding environment of the automobile 20 is good or bad based on the illuminance. Alternatively, the environment determination unit 102 may determine whether the surrounding environment of the automobile 20 is good or bad based on weather information such as cloud cover and precipitation amount instead of or in addition to the illuminance.

[0036] The automobile 20 may be equipped with various sensors such as an illuminance sensor and a rainfall sensor, and the environment determination unit 102 may acquire sensing data from these sensors to make a determination, or the environment determination unit 102 may acquire weather information from the Internet and make a determination. The environment determination unit 102 supplies the determination result to the recognition processing unit 103.

[0037] For example, the environment determination unit 102 determines a good environment when the illuminance is 1000 lx or more, the cloud cover is 2 or less, and the precipitation is 0 mm, and determines a bad environment when any of these conditions are not met. Note that the types and values ​​of information such as illuminance, cloud cover, and precipitation are merely examples, and the present technology is not limited to these types and values ​​of information. The information and values ​​used to determine the environment may be adjusted depending on the country, region, season, time of year, etc. Furthermore, the determination may be made using other information, such as the amount of snowfall.

[0038] The recognition processing unit 103 performs semantic segmentation as recognition processing using the infrared reflected light image and the infrared ambient light image obtained by the LiDAR 30 and supplied from the data acquisition unit 101. The recognition result of the semantic segmentation is obtained as two-dimensional label data (hereinafter, sometimes referred to as 2D label). Note that the recognition processing is not limited to semantic segmentation, and may be other recognition processing such as instance segmentation, object recognition, face recognition, moving object recognition, and scene recognition. The recognition processing unit 103 supplies the recognition result to the communication unit 104.

[0039] The recognition processing unit 103 switches between processing for a good environment and processing for a bad environment based on the determination result of the environment determination unit 102. In a good environment, the recognition processing unit 103 performs recognition processing using an infrared ambient light image and an infrared ambient light image that are the detection results of the LiDAR 30. On the other hand, in a bad environment, the recognition processing unit 103 performs recognition processing using a 2D label as recognition information transmitted from the server 200.

[0040] The communication unit 104 transmits the recognition result of the recognition processing unit 103, the 3D point cloud information, and the position information to the server 200 via a network. Communication methods include cellular communication, wireless LAN (Local Area Network), WAN (Wide Area Network), WiFi (Wireless Fidelity), 4G (Fourth Generation Mobile Communication System), 5G (Fifth Generation Mobile Communication System), Ethernet (registered trademark), etc. The communication unit 104 may use a communication function that the automobile 20 is equipped with in advance, or may perform communication by controlling a communication module separate from the automobile 20.

[0041] Although not shown, the automobile 20 includes a control unit, an interface, and a storage unit.

[0042] The control unit is composed of a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc. The ROM stores programs that are read and run by the CPU. The RAM is used as the CPU's work memory. The CPU controls the entire automobile 20 and each part by executing various processes and issuing commands according to the programs stored in the ROM.

[0043] The interface is a communication interface between an external device, the Internet, etc. Communication methods include wireless LAN, WAN, Wi-Fi, 4G, 5G, Bluetooth (registered trademark), etc.

[0044] The storage unit is, for example, a large-capacity storage medium such as a hard disk or flash memory.

[0045] The information processing device 100 is configured as described above. In this embodiment, the automobile 20 may be equipped with the functions of the information processing device 100 in advance, a processor equipped in the automobile 20 may execute the functions of the information processing device 100, or the information processing device 100 and the information processing method may be realized by the automobile 20 having the functions of a computer executing a program. The program may be installed in the automobile 20 in advance, or may be distributed by download or storage medium, etc., and installed by a user, etc. Alternatively, the information processing device 100 may be configured as a standalone device and installed in the automobile 20.

[0046] [Configuration of Server 200 ] The server 200 is configured to include a communication unit 201 , a 3D map creation unit 202 , a 3D map storage unit 203 , and a recognition information creation unit 204 .

[0047] The communication unit 201 receives the position information, 3D point cloud information, and recognition results transmitted from the information processing device 100 via the communication function or communication module provided in the server 200 and the network. The communication unit 201 also transmits 2D labels as recognition information created by the recognition information creation unit 204 to the information processing device 100.

[0048] The 3D map creation unit 202 creates a 3D map with labels attached, which are the results of semantic segmentation, based on the 3D point cloud information, position information, and recognition results transmitted from the information processing device 100. Hereinafter, this 3D map will be referred to as a labeled 3D map.

[0049] The 3D map storage unit 203 stores the labeled 3D map created by the 3D map creation unit 202. The 3D map storage unit 203 can be realized by using a storage medium or the like that is normally included in the server 200.

[0050] The recognition information creation unit 204 creates 2D labels as recognition information to be used in the recognition process of the information processing device 100 in adverse environments.

[0051] Although not shown, the server 200 includes a control unit, an interface, and a storage unit, which are similar in configuration to those included in the automobile 20 described above.

[0052] The server 200 is configured as described above. In this embodiment, the server 200 operates as a second information processing device. A processor included in the server 200 may execute the functions of the second information processing device, or the server 200 may execute a program to realize the second information processing device and its information processing method. The program may be installed in the server 200 in advance, or may be distributed by download or storage medium, etc., and installed by a user, etc.

[0053] [1-2. Processing in the Information Processing Device 100 and the Server 200] Next, processing in the information processing device 100 and the server 200 will be described with reference to FIGS. 8 to 12. FIG.

[0054] First, the processing in the information processing device 100 will be described with reference to Fig. 8. In step S101, the environment determination unit 102 determines whether the surrounding environment of the automobile 20 is a good environment or a bad environment. If the determination result is a good environment, the processing proceeds to step S102 (Yes in step S101). On the other hand, if the determination result is a bad environment, the processing proceeds to step S108 (No in step S101). In this way, the information processing device 100 performs different processing depending on whether the surrounding environment of the automobile 20 is a good environment or a bad environment.

[0055] First, the processing of the information processing device 100 in a good environment will be described. In step S102, the data acquisition unit 101 acquires a depth map, an infrared reflected light image, and an infrared ambient light image, which are detection results supplied from the LiDAR 30. The data acquisition unit 101 supplies the infrared reflected light image and the infrared ambient light image to the recognition processing unit 103. The data acquisition unit 101 also converts the depth map into 3D point cloud information by polar coordinate conversion using the design value of the laser projection direction, and supplies the 3D point cloud information to the communication unit 104.

[0056] Next, in step S103, as shown in FIG. 9 , the recognition processing unit 103 performs semantic segmentation as recognition processing using the infrared reflected light image and the infrared ambient light image. The recognition results of the semantic segmentation are obtained as 2D labels. Since it is expected that existing methods using annotations can provide sufficient recognition performance in good environments, the recognition results of the semantic segmentation are sent to the server 200 as reference data.

[0057] Next, in step S104, if the information processing device 100 is connected to the server 200, the process proceeds to step S105 (Yes in step S104).

[0058] Next, in step S105, the information processing device 100 acquires the current position information of the automobile 20 from the position information acquisition unit 40. This position information is the position of the automobile 20 at the time when the LiDAR 30 performed detection.

[0059] Next, in step S106, the information processing device 100 transmits the recognition result, the 3D point cloud information, and the position information to the server 200 as shown in FIG.

[0060] If the traveling of the automobile 20 has ended in step S107, the processing of the information processing device 100 ends (Yes in step S107). If the traveling of the automobile 20 has not ended, the information processing device 100 proceeds to step S102 (No in step S107), and repeats steps S102 to S107 until the traveling of the automobile 20 has ended.

[0061] In this manner, the information processing device 100 performs processing in a good environment.

[0062] Next, referring to FIG. 10A , the processing of the server 200 in a good environment will be described. In step S211, the 3D map creation unit 202 creates a labeled 3D map by arranging and integrating 3D point cloud information and 2D labels, which are recognition results, based on position information, as shown in FIG. 9 . This labeled 3D map is a 3D map with labels attached, which are the results of semantic segmentation. The 3D map creation unit 202 can create a large labeled 3D map by collecting and integrating multiple pieces of position information and 3D point cloud information acquired at various position angles in a good environment.

[0063] Next, in step S212, the 3D map storage unit 203 stores the labeled 3D map generated by the 3D map creation unit 202. In this way, the server 200 creates a labeled 3D map based on information in a good environment and stores it so that it can be used in a bad environment. The labeled 3D map created in a good environment has higher accuracy than a labeled 3D map created based on the detection results of the LiDAR 30, the quality of which is degraded in a bad environment.

[0064] This completes the processing of the server 200 in the good environment.

[0065] Next, the processing of the information processing device 100 in a bad environment will be described with reference to Fig. 8. If the environment determination unit 102 determines in step S101 that the environment is not good, that is, if the environment is bad, the processing proceeds to step S108 (No in step S101).

[0066] Next, in step S108, the data acquisition unit 101 acquires the depth map, infrared reflected light image, and infrared ambient light image, which are the detection results output from the LiDAR 30. The data acquisition unit 101 supplies the infrared reflected light image and infrared ambient light image to the recognition processing unit 103. The data acquisition unit 101 also converts the depth map into 3D point cloud information by polar coordinate conversion using the design value of the laser projection direction, and supplies the 3D point cloud information to the communication unit 104.

[0067] Next, in step S109, if the information processing device 100 is connected to the server 200, the process proceeds to step S110 (Yes in step S109).

[0068] Next, in step S110, the information processing device 100 acquires the current position information of the automobile 20 from the position information acquisition unit 40. This position information is the position of the automobile 20 at the time when the LiDAR 30 performed detection.

[0069] Next, in step S111, as shown in FIG. 11 , the information processing device 100 transmits 3D point cloud information and position information to the server 200. When transmitting this 3D point cloud information and position information, the information processing device 100 requests the server 200 to transmit 2D labels as recognition information generated by the server 200. The request data includes metadata (text data, etc.) indicating whether the environment is good or bad. The server 200 switches the processing to either a good environment or a bad environment by referring to the metadata in the request data. Upon receiving this request, the server 200 executes processing for the bad environment and creates recognition information. The generation of recognition information in the server 200 will be described later.

[0070] 11, the information processing device 100 receives the recognition information transmitted from the server 200. The received recognition information is supplied from the communication unit 104 to the recognition processing unit 103.

[0071] Next, in step S113, the recognition processing unit 103 performs semantic segmentation based on the recognition information and the infrared ambient light image and the infrared reflected light image, which are the detection results of the LiDAR 30 in a poor environment.

[0072] If the automobile 20 has finished traveling in step S114, the process of the information processing device 100 ends (Yes in step S114). If the automobile 20 has not finished traveling, the process proceeds to step S108 (No in step S114), and the information processing device 100 repeats steps S108 to S115 until the automobile 20 has finished traveling.

[0073] The description returns to step S109. If the information processing device 100 is not connected to the server 200 in step S109, the process proceeds to step S115 (No in step S109).

[0074] Next, in step S115, the recognition processing unit 103 performs semantic segmentation based on the infrared ambient light image and the infrared reflected light image, which are the detection results of the LiDAR 30. If the information processing device 100 and the server 200 are not connected, the information processing device 100 cannot receive recognition information from the server 200. Therefore, even in a poor environment, the recognition processing unit 103 performs semantic segmentation using the detection results of the LiDAR 30.

[0075] Next, referring to FIG. 10B, the process of the server 200 in a bad environment will be described.

[0076] First, in step S221, the server 200 acquires the request data, 3D point cloud information, and location information transmitted from the information processing device 100. The server 200 switches the processing to execute processing in a bad environment by referring to metadata included in the request data that indicates whether the environment is good or bad. Note that if the data transmitted from the information processing device 100 is 3D point cloud information and location information and does not include a recognition result, the server 200 may switch the processing to execute processing in a bad environment.

[0077] Next, in step S222, the recognition information creation unit 204 estimates the position and orientation of the automobile 20 in the current adverse environment relative to the labeled 3D map in the good environment, based on the stored labeled 3D map in the good environment and the 3D point cloud information and position information in the adverse environment transmitted from the information processing device 100.

[0078] To estimate the position and orientation of the automobile 20, first, a portion of the labeled 3D map in a good environment is cut out based on the position information. The cutout range is, for example, a range with a radius of 100 m approximately centered on the position of the automobile 20 indicated by the position information as the observation range of the automobile 20. However, the radius of 100 m is merely an example value, and the cutout range may be larger or smaller.

[0079] Because the position information detected by the GPS, compass, or the like serving as the position information acquisition unit 40 contains errors and the labeled 3D map is large, it is not possible to identify the exact position of the automobile 20 on the labeled 3D map using only the position information. Therefore, by cutting out a portion of the labeled 3D map based on the position information in this way, it is possible to roughly identify the position of the automobile 20 on the labeled 3D map and efficiently match it with the 3D point cloud information described below.

[0080] Next, the extracted labeled 3D map is compared with the location information and 3D point cloud information. Because location information acquired by GPS or other devices contains errors, the point clouds in the 3D shape information and the extracted labeled 3D map do not overlap in their current state. Because the extracted labeled 3D map has vertex data and label data corresponding to each vertex, the location information, the labeled 3D point cloud, and the extracted labeled 3D map are compared by adjusting the arrangement of the 3D point cloud using a method for minimizing the distance between the point clouds, such as ICP (Iterative Closest Point), for adjusting the arrangement of the 3D point cloud.

[0081] By comparing the 3D point cloud information with the extracted labeled 3D map, it is possible to accurately estimate the position and direction on the labeled 3D map from which the 3D point cloud was detected. This makes it possible to estimate the position and direction of the LiDAR 30 detection on the labeled 3D map, i.e., the detailed position and orientation of the vehicle 20 in the current adverse environment relative to the labeled 3D map.

[0082] Next, in step S223, the recognition information creation unit 204 creates recognition information by projecting the labeled 3D map into two dimensions based on the estimated position and orientation of the automobile 20, and extracting 2D labels, which are two-dimensional images with labels attached to each pixel, from the labeled 3D map.

[0083] Then, in step S224, the server 200 transmits the 2D label as recognition information to the information processing device 100.

[0084] In step S112, the information processing device 100 receives the recognition information transmitted from the server 200 in step S224. Then, in step S113, the information processing device 100 performs semantic segmentation based on the recognition information, the infrared ambient light image, and the infrared reflected light image in the adverse environment.

[0085] Processing in the information processing device 100 and the server 200 is performed as described above. According to the present technology, in a good environment, the information processing device 100 performs recognition processing such as semantic segmentation using an infrared reflected light image and an infrared ambient light image obtained by the LiDAR 30. The server 200 then generates and stores a labeled 3D map using the recognition results in the good environment. Meanwhile, in a bad environment, the information processing device 100 performs recognition using 2D labels, which are recognition information created by the server 200 from the highly accurate labeled 3D map in the good environment, thereby obtaining highly accurate recognition results even in a bad environment. In a bad environment, the visibility of infrared ambient light, which is the input for the recognition processing, decreases, and therefore the performance of the recognition processing is expected to decrease. Therefore, by using recognition results previously obtained in a good environment for recognition processing in a bad environment, it becomes possible to perform recognition processing such as semantic segmentation with high accuracy even in a bad environment.

[0086] Although the visibility of infrared ambient light decreases in adverse environments due to the influence of the environment, the influence of the adverse environment is less on distance information and infrared reflected light than on ambient light. Therefore, by using the infrared reflected light image in the adverse environment in addition to the depth map (distance information) in the adverse environment to compare with a labeled 3D map created in a good environment, it is possible to obtain 2D labels as recognition information required for recognition processing in adverse environments.

[0087] Furthermore, in order to extract and create 2D labels as recognition information from a labeled 3D map, the system compares the location information and 3D point cloud information with the labeled 3D map, thereby improving the positional accuracy of the 2D labels extracted from the labeled 3D image.

[0088] Improving the accuracy of semantic segmentation can improve the accuracy of automated driving and driving assistance of the automobile 20. This makes it possible to realize automated driving and driving assistance that are robust in a variety of environments.

[0089] <Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible.

[0090] If the recognition processing unit 103 is equipped with an algorithm that outputs a 2D label as a recognition result and two-dimensional reliability data ranging from 0 to 1 indicating the reliability of each pixel constituting the 2D label, the reliability may be used to determine whether the environment is good or bad. For example, as shown in FIG. 13 , if reliability data is output simultaneously with the 2D label of the recognition result by the recognition processing unit 103 at a certain time t=T, and the average of the reliability data for each pixel of the 2D label is less than 0.5, the environment is determined to be bad. Then, the processing for the next frame at time t=T+1 is switched to processing for an adverse effect, and the recognition processing unit 203 performs recognition processing using the 2D label as recognition information transmitted from the server 200.

[0091] Furthermore, instead of transmitting the location information and 3D point cloud information from the information processing device 100 to the server 200 and transmitting the recognition information from the server 200 to the information processing device 100 each time, a labeled 3D map that has been cut out to a large size based on the location information may be transmitted to the information processing device 100 in advance. Then, a process of matching the labeled 3D map with the 3D point cloud information is performed within the information processing device 100. In this method, a large amount of communication traffic and communication time are required to transmit the labeled 3D map from the server 200 to the information processing device 100, but the delay between matching the labeled 3D map with the 3D point cloud information and creating 2D labels from the labeled 3D map can be reduced.

[0092] In the embodiment, in a bad environment, the information processing device 100 performs recognition processing using recognition information transmitted from the server 200, but it is also possible to perform recognition processing based on the detection results of the LiDAR 30 even in a bad environment, and upload the recognition results to the server 200 or aggregate them as a separate data set. Although the accuracy may be lower than that of recognition results in a good environment, by manually correcting the obtained recognition results, a data set of recognition results in a bad environment can be created more easily than in the past, and this can be used to improve recognition performance.

[0093] The number of automobiles 20 equipped with the functions of the information processing device 100 does not necessarily have to be one, but may be multiple. When there are multiple information processing devices 100, the server 200 may create a labeled 3D map by integrating information on good environments transmitted from the multiple information processing devices 100. By using information transmitted from the multiple information processing devices 100, a labeled 3D map of a wide area can be efficiently created.

[0094] The present technology can be applied not only to automobiles 20 but also to other mobile objects such as hybrid electric vehicles, motorcycles, bicycles, personal mobility vehicles, airplanes, drones, ships, robots, construction machinery, and agricultural machinery (tractors). The present technology can improve the accuracy of automatic driving, automatic steering, and autonomous movement of these mobile objects. Furthermore, the information processing device 100 may operate on electronic devices such as personal computers, smartphones, tablet terminals, and wearable devices.

[0095] Furthermore, the second information processing device is not limited to the server 200, and may be a personal computer, a tablet terminal, a smartphone, or the like.

[0096] Furthermore, this technology is not limited to mobile objects, but can also be applied to any process that uses the results of semantic segmentation, such as agriculture, plant and animal breeding, and aquaculture.

[0097] The present technology can also be configured as follows. (1) An information processing device including: an environment determination unit that determines an environmental state; a recognition processing unit that performs recognition processing based on a detection result of a sensor; and a communication unit that performs communication processing with an external device, wherein when the environment is in a first state, the recognition processing unit performs recognition processing based on the detection result of the sensor, and the communication unit transmits the detection result of the sensor and the recognition result of the recognition processing unit to the external device. (2) The information processing device described in (1), wherein when the environment is in a second state, the communication unit transmits the detection result of the sensor to the external device. (3) The information processing device described in (1) or (2), wherein when the environment is in the second state, the recognition processing unit performs the recognition processing based on recognition information transmitted from the external device. (4) The information processing device described in any of (1) to (3), wherein the first state is a state in which the recognition processing can be performed with high accuracy based on the detection result of the sensor, and the second state is a state in which the accuracy of the recognition processing is reduced due to the detection result of the sensor. (5) The information processing device according to any one of (1) to (4), wherein the environment determination unit determines the state of the environment based on illuminance. (6) The information processing device according to any one of (1) to (5), wherein the environment determination unit determines the state of the environment based on information related to weather. (7) The information processing device according to any one of (1) to (6), wherein the sensor is a single-photon avalanche diode (SPAD) LiDAR capable of detecting distance information, infrared reflected light, and infrared ambient light as the detection results. (8) The information processing device according to any one of (1) to (7), wherein the communication unit transmits to the external device position information of a position detected by the sensor, acquired by a position information acquisition unit. (9) The information processing device according to any one of (1) to (8), wherein the LiDAR is provided in a moving object. (10) The information processing device according to any one of (1) to (9), wherein the recognition processing unit performs semantic segmentation.(11) An information processing method for determining an environmental state, performing recognition processing based on sensor detection results, and performing communication processing with an external device, wherein, when the environment is in a first state, the recognition processing is performed based on the sensor detection results, and the sensor detection results and the recognition results of the recognition processing are transmitted to the external device. (12) A program causing a computer to execute an information processing method for determining an environmental state, performing recognition processing based on the sensor detection results, and performing communication processing with an external device, wherein, when the environment is in a first state, the recognition processing is performed based on the sensor detection results, and the sensor detection results and the recognition results of the recognition processing are transmitted to the external device. (13) An information processing device comprising: a 3D map creation unit that creates a 3D map based on the sensor detection results and the recognition results transmitted from the external device when the environment is in the first state; a recognition information creation unit that creates recognition information from the 3D map when the environment is in a second state; and a communication unit that transmits the recognition information to the external device. (14) The information processing device according to (13), wherein the recognition information creation unit creates the recognition information from the 3D map based on 3D point cloud information that is the detection result of the sensor and transmitted from the external device when the environment is in a second state. (15) The information processing device according to (13) or (14), wherein the recognition information creation unit creates the recognition information from the 3D map based on position information transmitted from the external device when the environment is in the second state. (16) The information processing device according to (15), wherein the recognition information creation unit cuts out the 3D map based on the position information and compares the cut-out 3D map with the 3D point cloud information. (17) The information processing device according to any of (13) to (16), wherein the 3D map creation unit creates the 3D map based on the recognition result, 3D point cloud information, and position information transmitted from the external device. (18) An information processing method, comprising: generating a 3D map based on sensor detection results and recognition results transmitted from an information processing device when an environment is in a first state; creating recognition information from the 3D map when the environment is in a second state; and transmitting the recognition information to the information processing device.(19) A program that causes a computer to execute an information processing method, which generates a 3D map based on sensor detection results and recognition results transmitted from an information processing device when the environment is in a first state, creates recognition information from the 3D map when the environment is in a second state, and transmits the recognition information to the information processing device.

[0098] 20... Automobile 30... LiDAR 100... Information processing device 101... Environment determination unit 103... Recognition processing unit 104... Communication unit 200... Server 201... Communication unit 202... 3D map creation unit 204... Recognition information creation unit

Claims

1. An information processing device comprising: an environment determination unit that determines the state of the environment; a recognition processing unit that performs recognition processing based on the detection results of a sensor; and a communication unit that performs communication processing with an external device, wherein when the environment is in a first state, the recognition processing unit performs recognition processing based on the detection results of the sensor, and the communication unit transmits the detection results of the sensor and the recognition results of the recognition processing unit to the external device.

2. The information processing device according to claim 1, wherein when the environment is in a second state, the communication unit transmits the detection result of the sensor to the external device.

3. The information processing device according to claim 1, wherein when the environment is in a second state, the recognition processing section performs the recognition processing based on recognition information transmitted from the external device.

4. An information processing device as described in claim 1, wherein the first state is a state in which the recognition processing can be performed with high accuracy based on the detection result of the sensor, and the second state is a state in which the accuracy of the recognition processing is reduced based on the detection result of the sensor.

5. The information processing device according to claim 1, wherein the environment determination unit determines the state of the environment based on illuminance.

6. The information processing device according to claim 1, wherein the environment determination unit determines the state of the environment based on information relating to weather.

7. The information processing device according to claim 1, wherein the sensor is a single-photon avalanche diode (SPAD) LiDAR capable of detecting distance information, infrared reflected light, and infrared ambient light as the detection results.

8. The information processing device according to claim 1, wherein the communication unit transmits to the external device the position information of the position detected by the sensor, the position information being acquired by a position information acquisition unit.

9. The information processing device according to claim 1, wherein the LiDAR is installed on a moving body.

10. The information processing device according to claim 1, wherein the recognition processing unit performs semantic segmentation.

11. An information processing method for determining an environmental state, performing recognition processing based on a sensor detection result, and performing communication processing with an external device, wherein, when the environment is in a first state, recognition processing is performed based on the sensor detection result, and the sensor detection result and the recognition result of the recognition processing are transmitted to the external device.

12. A program for causing a computer to execute an information processing method for determining an environmental state, performing recognition processing based on a detection result of a sensor, and communicating with an external device, the information processing method including, when the environment is in a first state, performing recognition processing based on the detection result of the sensor, and transmitting the detection result of the sensor and the recognition result of the recognition processing to the external device.

13. An information processing device comprising: a 3D map creation unit that creates a 3D map based on sensor detection results and recognition results transmitted from an external device when the environment is in a first state; a recognition information creation unit that creates recognition information from the 3D map when the environment is in a second state; and a communication unit that transmits the recognition information to the external device.

14. The information processing device according to claim 13, wherein the recognition information creation unit creates the recognition information from the 3D map based on 3D point cloud information, which is the detection result of the sensor transmitted from the external device when the environment is in a second state.

15. The information processing device according to claim 13, wherein the recognition information creation unit creates the recognition information from the 3D map based on position information transmitted from the external device when the environment is in a second state.

16. The information processing device according to claim 15, wherein the recognition information creation unit cuts out the 3D map based on the position information, and compares the cut-out 3D map with the 3D point cloud information.

17. The information processing device according to claim 13, wherein the 3D map creation unit creates the 3D map based on the recognition result, 3D point cloud information, and position information transmitted from the external device.

18. An information processing method, comprising: generating a 3D map based on sensor detection results and recognition results transmitted from an information processing device when the environment is in a first state; creating recognition information from the 3D map when the environment is in a second state; and transmitting the recognition information to the information processing device.

19. A program that causes a computer to execute an information processing method, which generates a 3D map based on sensor detection results and recognition results transmitted from an information processing device when the environment is in a first state, creates recognition information from the 3D map when the environment is in a second state, and transmits the recognition information to the information processing device.

Citation Information

Patent Citations

  • Map system, method, and storage medium

    WO2020045317A1