Information processing apparatus, information processing method, program, and movable object

Abstract

The present invention improves the accuracy of recognizing the external environment of a moving body. Image data having image features (region, date and time, weather, etc.) corresponding to the moving scene of the moving body is acquired. Training is performed using the image data, and coefficients of a DNN for inference for recognizing the external environment of the moving body are obtained from the image data of the moving scene. For example, the external environment is semantic segmentation, depth, etc. With the DNN for inference in which the coefficients for inference are set, the external environment of the moving body can be accurately recognized from the image data of the moving scene.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, a program, and a movable object, and more particularly, to an information processing apparatus and others for improving the recognition accuracy of the external environment of a movable object. Background Art

[0002] Generally, an autonomous vehicle is equipped with an in-vehicle camera device that recognizes the external environment based on image data of a driving scene and automatically controls driving using the recognition result. For example, the recognition result is based on semantic segmentation or depth. Since it is directly related to safety, recognizing the external environment requires very high accuracy.

[0003] To recognize the image data of a driving scene, it is known to use a deep neural network (DNN) as a machine learning technique. In this case, learning is performed using the image data of the driving scene actually captured by the in-vehicle camera device to pre-acquire the coefficients of the DNN for inference.

[0004] When using a DNN to recognize the external environment based on the image data of a driving scene, a large gap between the image data of the driving scene and the image data of the driving scene used for learning results in a decrease in the accuracy of the recognition result. Using as much image data of various scenes as possible in learning improves the accuracy of the recognition result to a certain extent. However, it is impossible to perform learning covering image data of various scenes around the world.

[0005] For example, Patent Document 1 discloses that eliminating the bias in the number of learning data at each capture position enables the acquisition of general learning coefficients.

[0006] Citation List

[0007] Patent Document

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2018-195237 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] An object of the present technology is to improve the recognition accuracy of the external environment of a movable object.

[0011] Solution to the Problem

[0012] According to the concept of the present technology, there is provided an information processing apparatus including:

[0013] an image data acquisition unit configured to acquire image data having an image feature corresponding to a movement scene of a movable object; and

[0014] A DNN unit for learning, which is configured to: perform learning using the image data acquired by the image data acquisition unit to obtain coefficients of an inference DNN for identifying the external environment of a movable object from the image data of a moving scene.

[0015] In the present technology, the image data acquisition unit acquires image data having image features corresponding to the moving scene of the movable object. For example, the image features may include position elements. In this case, for example, the image features may further include weather elements or date and time elements.

[0016] The DNN unit for learning performs learning using the image data acquired by the image data acquisition unit to obtain coefficients of an inference DNN for identifying the external environment of a movable object from the image data of a moving scene. For example, based on the coefficients of the inference DNN in the first time zone, the DNN unit for learning may perform transfer learning using the image data acquired by the image data acquisition unit to obtain coefficients of the inference DNN to be used in a second time zone after the first time zone.

[0017] As described above, in the present technology, learning is performed using image data having image features corresponding to the moving scene of the movable object, and coefficients of an inference DNN for identifying the external environment of the movable object are obtained from the image data of the moving scene. The inference DNN with the coefficients of the inference DNN set can accurately identify the external environment of the movable object from the image data of the moving scene.

[0018] Note that, in the present technology, for example, the information processing device may further include an image data receiving unit, which is configured to: receive the image data of the moving scene from the movable object in the case where position information and date and time information are added to the image data. In addition, in the present technology, for example, the information processing device may further include a coefficient sending unit, which is configured to send the coefficients of the inference DNN obtained by the DNN unit for learning to the movable object. In this case, for example, when the evaluation value of the coefficients of the inference DNN obtained by the DNN unit for learning is higher than the evaluation value of the general coefficients, the coefficient sending unit may send the coefficients of the inference DNN obtained by the DNN unit for learning to the movable object. This arrangement enables the movable object to use coefficients with a higher evaluation value.

[0019] In addition, according to another concept of the present technology, a movable object is provided, including:

[0020] An inference DNN unit, which is configured to identify the external environment from the image data of the moving scene;

[0021] A control unit, which is configured to control the movement based on the recognition result from the inference DNN unit; and

[0022] A coefficient receiving unit configured to receive, from a cloud server, coefficients of an inference DNN to be used by an inference DNN unit.

[0023] Wherein, the coefficients of the inference DNN are obtained by performing learning using image data having image features corresponding to a moving scenario.

[0024] In the present technology, the movable object includes an inference DNN unit for recognizing an external environment from image data of a moving scenario. The control unit controls the movement based on the recognition result from the inference DNN unit. Further, the coefficient receiving unit receives, from the cloud server, coefficients of the inference DNN to be used by the inference DNN unit. Here, the coefficients of the inference DNN are obtained by performing learning using image data having image features corresponding to a moving scenario.

[0025] As described above, in the present technology, the coefficients of the inference DNN are obtained by performing learning using image data having image features corresponding to a moving scenario, and the coefficients of the inference DNN to be used by the inference DNN unit are received from the cloud server. This arrangement enables the inference DNN unit to accurately recognize the external environment of the movable object from the image data of the moving scenario.

[0026] Note that, in the present technology, for example, the movable object may further include an image data transmitting unit configured to: when adding position information and date and time information to the image data, transmit the image data of the moving scenario to the cloud server. This arrangement can provide the image data of the moving scenario to the cloud server. Further, position information about the movable object can be provided to the cloud server, and it is possible to easily receive, from the cloud server, the coefficients of the inference DNN corresponding to the area where the movable object is moving.

[0027] Further, in the present technology, for example, the movable object may further include: a learning DNN unit configured to perform learning using the image data of the moving scenario to obtain coefficients of the inference DNN; and a coefficient transmitting unit configured to transmit the coefficients of the inference DNN obtained by the learning DNN unit to the cloud server. With this arrangement, in a case where it is impossible to obtain the coefficients of the inference DNN by performing learning due to insufficient collection of image data by the cloud server, the coefficients of the inference DNN obtained by the learning DNN unit can be used as a substitute.

[0028] In addition, in the present technology, for example, when a movable object moves toward the second region side in an overlapping region between the first region and the second region, when the coefficient receiving unit receives the coefficients of the DNN for inference corresponding to the second region from the cloud server, the DNN unit for inference can switch the coefficients of the DNN for inference corresponding to the first region to the coefficients of the DNN for inference corresponding to the second region. With this arrangement, even when the region where the movable object is moving changes, it can cause the DNN for inference with appropriate coefficients to operate without being affected by transmission delay.

[0029] In addition, in the present technology, the DNN unit for inference may include a first DNN for inference and a second DNN for inference. When the movable object moves in the first region, the coefficient receiving unit may receive the coefficients of the DNN for inference corresponding to the second region to which the movable object will move next, may set the coefficients of the DNN for inference corresponding to the first region to the first DNN for inference and set the coefficients of the DNN for inference corresponding to the second region to the second DNN for inference, and when the movable object moves from the first region to the second region, the DNN unit for inference can switch from the first DNN for inference in use to the second DNN for inference to be used. With this arrangement, even when the region where the movable object is moving changes, it can cause the DNN for inference with appropriate coefficients to operate without being affected by transmission delay.

[0030] In addition, in the present technology, the movable object may further include a storage device configured to store the coefficients of the DNN for inference corresponding to the region where the movable object is moving and the coefficients of the DNN for inference corresponding to another region around the region. The coefficients of the DNN for inference are received by the coefficient receiving unit respectively, and when the movable object moves from the first region to the second region, the DNN unit for inference can extract the coefficients of the DNN for inference corresponding to the second region from the storage device and can use the extracted coefficients of the DNN for inference. With this arrangement, even when the region where the movable object is moving changes, it can cause the DNN for inference with appropriate coefficients to operate without being affected by transmission delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a block diagram showing a configuration example of an autonomous driving system as an embodiment.

[0032] Figure 2 is a block diagram showing a configuration example of an autonomous driving vehicle and a cloud server.

[0033] Figure 3Shows exemplary image data of a driving scenario and an exemplary recognition result of semantic segmentation based on the image data of the driving scenario.

[0034] Figure 4 Is a block diagram showing a detailed configuration example of a cloud server.

[0035] Figure 5 Explanatorily shows the processing of an image database unit.

[0036] Figure 6 Is a block diagram showing a detailed configuration example of a DNN unit for learning.

[0037] Figure 7 Is a block diagram showing a detailed configuration example of a coefficient database unit.

[0038] Figure 8 Is a flowchart showing an exemplary processing procedure of an image database unit.

[0039] Figure 9 Is a flowchart showing an exemplary processing procedure of a DNN unit for learning.

[0040] Figure 10 Is a flowchart showing an exemplary processing procedure of a coefficient database unit.

[0041] Figure 11 Explanatorily shows an exemplary method for processing the transmission delay of coefficients of a DNN.

[0042] Figure 12 Is a block diagram showing a hardware configuration example of a cloud server. Detailed implementation

[0043] Hereinafter, a manner for implementing the present invention (hereinafter referred to as "embodiment") will be described. Note that the description will be given in the following order.

[0044] 1. Embodiment

[0045] 2. Modification

[0046] <1. Embodiment>

[0047] [Configuration of an autonomous driving system]

[0048] Figure 1 Shows a configuration example of an autonomous driving system 10 as an embodiment. The autonomous driving system 10 includes a vehicle (hereinafter appropriately referred to as "autonomous driving vehicle") 100 having a plurality of autonomous driving functions and connected to a cloud server 200 via the Internet 300.

[0049] Among a plurality of regions in the illustrated example, namely Region 1, Region 2, …, and Region N, autonomous vehicles are in motion. Each autonomous vehicle 100 periodically acquires image data of a scene having image features corresponding to the driving scene, and transmits the image data of the scene to the cloud server 200 via the Internet 300. Here, the image features corresponding to the driving scene include position elements of the driving scene (e.g., information about the region in which the vehicle is driving), and include weather elements, date and time elements, etc. of the driving scene.

[0050] Each autonomous vehicle 100 includes an inference deep neural network (DNN) unit 101 that identifies the external environment from the image data of the driving scene. The external environment identified by the inference DNN unit 101 is based on, for example, semantic segmentation or depth. In each autonomous vehicle 100, the inference DNN unit 101 controls power, braking, etc. in autonomous driving based on the recognition result of the external environment.

[0051] The cloud server 200 includes a learning DNN unit 201. Based on the image data transmitted from each autonomous vehicle 100, the learning DNN unit 201 periodically acquires the coefficients of the DNN to be set in the inference DNN unit 101 of the autonomous vehicle 100 based on the region and the weather. Then, the cloud server 200 periodically transmits the coefficients of the DNN corresponding to the region in which the autonomous vehicle 100 is driving and the weather at that time to each autonomous vehicle 100 via the Internet 300.

[0052] In this way, the coefficients of the DNN corresponding to the region in which each autonomous vehicle 100 is driving and the weather at that time are transmitted from the cloud server 200 to the autonomous vehicle 100. This arrangement enables the accuracy of the recognition result of the external environment to be improved by the inference DNN unit 101 of each autonomous vehicle 100. Therefore, power, braking, etc. in autonomous driving can be controlled more accurately.

[0053] “Configuration Example of Autonomous Vehicle and Cloud Server”

[0054] Figure 2 A configuration example of the autonomous vehicle 100 and the cloud server 200 is shown. The autonomous vehicle 100 includes an inference DNN unit 101, a capture unit 102, a position / date and time acquisition unit 103, an image data memory 104, a data transmission unit 105, a data reception unit 106, a DNN coefficient memory 107, a control unit 108, and a learning DNN unit 109.

[0055] The capture unit 102 includes a lens, a capture element such as a CCD image sensor or a CMOS image sensor, etc., and periodically acquires image data corresponding to the driving scene. The position / date and time acquisition unit 103 uses, for example, the Global Positioning System (GPS) to acquire information about the current position. In addition, the position / date and time acquisition unit 103 acquires information about the current date and time from a clock unit (not shown).

[0056] In the case where the position information and the date and time information acquired by the position / date and time acquisition unit 103 are added to the image data, the image data memory 104 temporarily stores the image data of the driving scene acquired by the capture unit 102. The data transmission unit 105 transmits the image data stored in the image data memory 104 (in the case where the position information and the date and time information are added to the image data) to the cloud server 200 via the Internet 300.

[0057] The data reception unit 106 receives the coefficients of the DNN transmitted from the cloud server 200 via the Internet 300. The DNN coefficient memory 107 temporarily stores the coefficients of the DNN received by the data reception unit 106. The coefficients of the DNN stored in the DNN coefficient memory 107 are extracted and set to the inference DNN unit 101.

[0058] Then, the inference DNN unit 101 identifies the external environment from the image data of the driving scene acquired by the capture unit 102. For example, the external environment is identified based on, for example, semantic segmentation or depth. For example, Figure 3 (a) shows exemplary image data of the driving scene. Figure 3 (b) shows an exemplary recognition result based on the semantic segmentation of the image data of the driving scene. Based on the recognition result of the external environment from the inference DNN unit 101, the control unit 108 controls power, braking, etc. in autonomous driving.

[0059] For example, in the case where communication with the cloud server 200 cannot be established due to the absence of a communication network, the learning DNN unit 109 performs learning using the image data stored in the image data memory 104 as learning data, and then acquires the coefficients of the DNN. In this case, for example, transfer learning is performed based on the coefficients of the DNN set to and used by the inference DNN unit 101 in a certain time zone (the first time zone), and the coefficients of the DNN to be used in the next time zone (the second time zone) are acquired. The coefficients of this DNN are dedicated coefficients corresponding to the region (position) and weather of the driving scene.

[0060] When communication with the cloud server 200 is enabled, the coefficients of the DNN obtained by the learning DNN unit 109 are sent from the data transmission unit 105 to the cloud server 200 via the Internet 300. Alternatively, the coefficients of the DNN obtained by the learning DNN unit 109 are set in the inference DNN unit 101 and used by the latter in the next time zone.

[0061] The cloud server 200 includes a learning DNN unit 201, a data reception unit 202, an image database unit 203, a coefficient database unit 204, and a data transmission unit 205.

[0062] "Detailed configuration example of the cloud server"

[0063] Figure 4 A detailed configuration example of the cloud server 200 is shown. The data reception unit 202 receives, via communication such as 5G, the image data of the driving scene sent from the autonomous vehicle 100 via the Internet 300 (in the case where location information and date and time information are added to the image data). In addition, the data reception unit 202 receives the coefficients (special coefficients) of the DNN sent from the autonomous vehicle 100.

[0064] "Description of the image database unit"

[0065] The image database unit 203 stores, based on the location information, date and time information, and weather information added to the image data, the image data of the driving scene received by the data reception unit 202 based on the region, date and time, and weather. In this case, the weather information can be obtained from a weather information server, or the weather information can be obtained by analyzing the image data. Note that in the above description, location information and date and time information are added to the image data of the driving scene sent from the autonomous vehicle 100; however, weather information can also be added to the image data.

[0066] In addition, the image database unit 203 configures and obtains a learning data set based on the region and weather in a certain time zone to obtain the coefficients of the DNN to be used by the inference DNN unit 101 of the autonomous vehicle 100 in the next time zone.

[0067] Figure 5Shows a configuration example of a learning data set for learning coefficients of a DNN corresponding to clear weather in a certain area to be used in the time zone from 00:30 to 01:00 (the time zone during driving). In this example, the image data ratio from 00:00 to 00:30 today (sunny day) is "3", the image data ratio from 01:00 to 01:30 on June 9 (sunny day) is "5", and the image data ratio from 01:00 to 01:30 on June 10 (cloudy) is "3". Since the weather is completely different, the image data from 01:00 to 01:30 on June 8 (rainy) is not used.

[0068] Note that in this example, the image data of the time zone during driving today (sunny day) is not included in the configuration of the learning data set; however, it is also conceivable to include the image data. In addition, it is also conceivable to refer to dates and times in units of years. For example, a learning data set for learning coefficients of a DNN corresponding to the weather on the day (snowing) in an area with little snow can be effectively configured.

[0069] In addition, in this example, the time zones are set at 30 - minute intervals. However, the length of such time zones can be defined according to the calculation speed of learning. In addition, in this example, the learning data set only includes image data of the same area. However, for example, in the case where the number of image data is small due to only including image data of the same area, it is also conceivable to refer to the image data of adjacent areas.

[0070] "Description of the DNN unit for learning"

[0071] Return reference Figure 4 , based on the learning data set based on area and weather obtained by the image database unit 203, the DNN unit 201 for learning performs learning using the learning DNN based on area and weather in a certain time zone, and then obtains the coefficients of the DNN to be set in the inference DNN unit 101 of the autonomous driving vehicle 100 and used by it in the next time zone.

[0072] In this case, transfer learning (unsupervised) based on the coefficients of the DNN in a certain time zone is performed, and the coefficients of the DNN to be used in the next time zone are obtained. The change between the driving scenario in a certain time zone and the driving scenario in the next time zone is not very large. Therefore, sequential transfer learning can achieve efficient learning with higher accuracy in a short time with a small amount of image data. For example, the learning end condition is determined based on a predefined learning time period or a predefined number of epochs (the number of times to update the coefficients).

[0073] Figure 6Shows a detailed configuration example of the DNN unit 201 for learning. The DNN unit 201 for learning includes a DNN for learning based on region and weather, and performs distributed learning in which the learning based on region and weather is executed in parallel. This distributed learning enables an improvement in the overall calculation speed. The illustrated example shows the case where the regions are from 1 to N and the weather types are from 1 to n. It is conceivable that the weather types include sunny, cloudy, rainy, snowy, etc.

[0074] Note that in the above description, in a certain time zone today, the learning of the coefficients of the DNNs for each region corresponding to all weather types in the next time zone is performed. However, it is also conceivable to omit the learning of the coefficients of the DNNs for the weather types that do not correspond at all in each region today. For example, in the case where the corresponding weather types in each region today are only sunny, it is meaningless to learn the coefficients of the DNNs corresponding to other types of weather (such as cloudy, rainy, and snowy), so it can be omitted.

[0075] "Description of the coefficient database unit"

[0076] Return to reference Figure 4 , the coefficient database unit 204 temporarily stores the coefficients (dedicated coefficients) of the DNN based on region and weather that are obtained by the learning DNN learning unit 201 in a certain time zone and are to be set to and used by the inference DNN unit 101 of the autonomous driving vehicle 100 in the next time zone. In addition, the coefficient database unit 204 also temporarily stores the coefficients (dedicated coefficients) of the DNN sent from the autonomous driving vehicle 100.

[0077] In addition, the coefficient database unit 204 determines and sends the coefficients of the DNN to be sent to each autonomous driving vehicle 100. In this case, basically, the coefficients of the DNN based on region and weather - that is, the dedicated coefficients - are determined as the coefficients of the DNN to be sent. However, in the case where an evaluation is performed based on the loss function of the DNN and the evaluation value of the dedicated coefficient is lower than the evaluation value of the general coefficient, the general coefficient is determined as the coefficient of the DNN to be sent instead of the dedicated coefficient. Here, the general coefficient is the coefficient of the DNN obtained by pre - learning using the image data of driving scenarios that satisfy various conditions (location, weather, date and time, etc.), and is the coefficient of the DNN that can correspond to various conditions.

[0078] Figure 7Shows a detailed configuration example of the coefficient database unit 204. The coefficient database unit 204 includes a storage unit 241 that stores the coefficients of such a DNN, and a DNN coefficient determination unit 242 that determines and outputs the coefficients of the DNN to be sent to each autonomous driving vehicle 100. In addition, the storage unit 241 includes: a storage unit 241a that temporarily stores the coefficients (dedicated coefficients) of the DNN based on region and weather obtained by the learning DNN unit 201; and a storage unit 241b that temporarily stores the coefficients (dedicated coefficients) of the DNN based on region and weather received by the data receiving unit 202.

[0079] In addition, the DNN coefficient determination unit 242 basically determines the coefficients of the DNN based on region and weather as the coefficients of the DNN in the next time zone to be sent to each autonomous driving vehicle 100, extracts the determined coefficients of the DNN from the storage unit 241, and then outputs the extracted coefficients of the DNN as the coefficients to be sent. In this case, the determined coefficients of the DNN are basically extracted from the storage unit 241a. However, in the case where the coefficients of the DNN do not exist in the storage unit 241a (corresponding to the case where learning cannot be performed due to a system failure or the like), if the coefficients of the DNN exist in the storage unit 241b, the coefficients of the DNN are extracted from the storage unit 241b.

[0080] Note that in the case where the determined coefficients of the DNN exist in both the storage unit 241a and the storage unit 241b, it is also conceivable that the DNN coefficient determination unit 242 is configured to: output the coefficients of the DNN with a higher evaluation value as the coefficients to be sent.

[0081] In the present embodiment, in fact, the DNN coefficient determination unit 242 outputs the coefficients of the DNN extracted from the storage unit 241 as the coefficients to be sent. In other words, the DNN coefficient determination unit 242 outputs the dedicated coefficients as the coefficients to be sent only when the evaluation value of the dedicated coefficients is higher than the evaluation value of the general coefficients. When the evaluation value of the dedicated coefficients is lower than the evaluation value of the general coefficients, the general coefficients are output as the coefficients to be sent.

[0082] For example, as a case where the evaluation value of the dedicated coefficients is lower than the evaluation value of the general coefficients, assume a case where the number of image data for learning is insufficient and sufficient learning cannot be performed. Therefore, when the dedicated coefficients are the coefficients of an inappropriate DNN, it is possible to avoid using the coefficients of the DNN on the side of the autonomous driving vehicle 100.

[0083] In addition, the DNN coefficient determination unit 242 may be configured to: when the coefficients of the determined DNN do not exist in the storage unit 241 (storage unit 241a or storage unit 241b), output a general coefficient as the coefficient of the DNN to be transmitted.

[0084] Return reference Figure 4 , the data transmission unit 205 transmits the coefficients (dedicated coefficients or general coefficients) of the DNN determined by the coefficient database unit 204 to each autonomous driving vehicle 100 via the Internet 300. In this case, the coefficients of the DNN to be used in the next time zone are transmitted to each autonomous driving vehicle 100 in a certain time zone.

[0085] Note that in the above description, when the evaluation value of the dedicated coefficient is low, the coefficient database unit 204 outputs a general coefficient as the coefficient of the DNN to be transmitted, and the data transmission unit 205 transmits the general coefficient to the autonomous driving vehicle 100. However, it is also conceivable that the coefficient database unit 204 is configured to output a command for indicating the use of the general coefficient when the evaluation value of the dedicated coefficient is low, and the data transmission unit 205 transmits the command to the autonomous driving vehicle 100. In this case, according to the command, the autonomous driving vehicle 100 uses the general coefficient that the autonomous driving vehicle 100 has.

[0086] It is also conceivable that the coefficient database unit 204 is configured to continuously output the dedicated coefficient without comparing the evaluation value of the dedicated coefficient with the evaluation value of the general coefficient, the data transmission unit 205 is configured to transmit the dedicated coefficient to the autonomous driving vehicle 100, and the autonomous driving vehicle 100 is configured to compare the evaluation value of the dedicated coefficient with the evaluation value of the general coefficient to determine which one to use.

[0087] Note that in the case where the coefficient database unit 204 outputs a general coefficient as the coefficient of the DNN for a certain region and weather, it is conceivable that in the case of obtaining the coefficients of the DNN for the region and weather in the next time zone, the DNN unit 201 for learning performs transfer learning based on the general coefficient.

[0088] "Exemplary processing procedures of the image database unit, the DNN unit for learning, and the coefficient database unit"

[0089] Figure 8The flowchart shows an exemplary process of the image database unit 203. In step ST1, the image database unit 203 obtains image data of the driving scenario of each autonomous driving vehicle 100 from the data receiving unit 202 (in the case where position information and date and time information are added to the image data). Next, in step ST2, the image database unit 203 adds weather information to the obtained image data of the driving scenario of each autonomous driving vehicle 100 (in the case where position information and date and time information are added to the image data).

[0090] Next, in step ST3, the image database unit 203 stores the obtained image data of the driving scenario of each autonomous driving vehicle 100 based on image features (region, date and time, and weather). Next, in step ST4, the image database unit 203 determines an image data set for learning the coefficients of the DNN in the next time zone based on image features (region and weather) (see Figure 5 ).

[0091] Next, in step ST5, the image database unit 203 performs preprocessing on each image data set based on image features (region and weather), and sends each image data set based on image features (region and weather) to the DNN unit 201 for learning. For example, the preprocessing includes processes such as cutting the image data into pieces, normalizing the pixel values, and shuffling the order. These are preprocessings typically performed for learning about image data.

[0092] Figure 9 The flowchart shows an exemplary process of the DNN unit 201 for learning. In step ST11, the DNN unit 201 for learning obtains an image data set based on image features (region and weather) from the image database unit 203. Next, in step ST12, using the image data set, the DNN unit 201 for learning performs learning (transfer learning) based on image features (region and weather) using the coefficients of the DNN.

[0093] Next, in step ST13, the DNN unit 201 for learning ends the learning based on a predefined learning time period or number of epochs. Next, in step ST14, the DNN unit 201 for learning sends the coefficients of the DNN learned based on image features (region and weather) to the coefficient database unit 204.

[0094] Figure 10The flowchart shows an exemplary process of the coefficient database unit 204. In step ST21, the coefficient database unit 204 obtains the coefficients of the DNN learned based on image features (region and weather) from the learning DNN unit 201. Next, in step ST22, the coefficient database unit 204 saves the obtained DNN coefficients with information about the image features (region and weather) added.

[0095] Next, in step ST23, the coefficient database unit 204 evaluates whether the accuracy of the obtained DNN coefficients (special coefficients) is higher than that of the general coefficients. This evaluation is performed based on the loss function. Next, in step ST24, if the accuracy is higher, the coefficient database unit 204 sends the obtained DNN coefficients (special coefficients) to the data sending unit 205; otherwise, if the accuracy is not high, it sends the general coefficients to the data sending unit 205, or sends a command to use the in-vehicle general coefficients to the data sending unit 205.

[0096] "Method for Handling Transmission Delay of DNN Coefficients"

[0097] When there is a delay in sending the DNN coefficients from the cloud server 200 to the autonomous driving vehicle 100, there are regions between which the autonomous driving vehicle 100 cannot receive the DNN coefficients corresponding to the driving region. For example, as methods for handling such transmission delays, the following (1) to (3) can be considered. Through these processing methods, even when the region in which the autonomous driving vehicle 100 is traveling changes, the inference DNN with the appropriately set DNN coefficients can operate without being affected by the transmission delay.

[0098] (1) In this method, an overlapping area is set between one region and another region, and a switch is made between the DNN coefficients in this area. In this case, when the autonomous driving vehicle 100 moves toward the second region side in the overlapping area between the first region and the second region, when the data receiving unit 106 receives the coefficients corresponding to the second region from the cloud server 200, the inference DNN unit 101 of the autonomous driving vehicle 100 switches from the DNN coefficients corresponding to the first region to the DNN coefficients corresponding to the second region.

[0099] (2) In this method, the area (region) that is predicted as the moving destination based on the driving direction of the autonomous driving vehicle 100, the coefficients corresponding to the area that is the moving destination are pre-applied to two separately prepared DNNs for inference, and when the autonomous driving vehicle 100 passes through this area or drives in the overlapping area, the DNN for inference to be used is switched. In this case, the DNN unit 101 for inference includes a first DNN for inference and a second DNN for inference, and when the autonomous driving vehicle 100 moves in the first area, the data receiving unit 106 receives the coefficients of the DNN corresponding to the second area that the autonomous driving vehicle 100 will move to next. Then, the coefficients of the DNN in the first area are set to the first DNN for inference, and the coefficients of the DNN in the second area are set to the second DNN for inference. When the autonomous driving vehicle 100 moves from the first area to the second area, the DNN unit 101 for inference switches from the first DNN for inference in use to the second DNN for inference to be used.

[0100] (3) In this method, on the side of the autonomous driving vehicle 100, the coefficients of a wide range of DNNs including the driving area are pre-stored in the storage device. When the autonomous driving vehicle 100 passes through this area or drives in the overlapping area, the coefficients of the DNN to be used by the DNN unit 101 for inference are switched. In this case, the autonomous driving vehicle 100 includes a storage device that stores the coefficients of the DNN corresponding to the area where the autonomous driving vehicle 100 is moving and the coefficients of the DNN corresponding to another area around this area, and the coefficients of the DNN for inference are all received by the data receiving unit 106. When the autonomous driving vehicle 100 moves from the first area where the autonomous driving vehicle 100 is driving to the second area, the DNN unit 101 for inference extracts the coefficients of the DNN in the second area from the storage device and uses the extracted coefficients of the DNN.

[0101] In this case, the range of the coefficients of the DNN stored in the storage device changes according to the driving of the autonomous driving vehicle 100. In Figure 11 , the ellipse with a dotted line indicates the area where the autonomous driving vehicle 100 is driving, and each ellipse with a solid line indicates the area around the area where the coefficients of the DNN are pre-stored in the storage device. Note that in the shown example, there is an overlapping area between the areas; however, it is also conceivable that there is no overlapping area.

[0102] "Hardware Configuration Example of Cloud Server"

[0103] Figure 12It is a block diagram showing an example of the hardware configuration of the cloud server 200. In the cloud server 200, a central processing unit (CPU) 501, a read-only memory (ROM) 502, and a random access memory (RAM) 503 are interconnected via a bus 504. In addition, an input / output interface 505 is connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0104] The input unit 506 includes a keyboard, a mouse, and a microphone. The output unit 507 includes a display and a speaker. The storage unit 508 includes a hard disk or a non-volatile memory. The communication unit 509 includes a network interface. The drive 510 drives a removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0105] In the cloud server 200 with the above configuration, the CPU 501 loads, for example, a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 to execute the program, thereby performing the above-described series of processes.

[0106] The program executed by the CPU 501 can be provided by being recorded on a removable medium 511 such as an encapsulated medium. Alternatively, the program can be provided via a wired or wireless transmission medium (such as a local area network, the Internet, or digital satellite broadcasting).

[0107] In the cloud server 200, by attaching the removable medium 511 to the drive 510, the program can be installed in the storage unit 508 via the input / output interface 505. Alternatively, the program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the storage unit 508. In addition, the program can be pre-installed in the ROM 502 or the storage unit 508.

[0108] Note that the program executed by the CPU 501 can be a program for sequentially performing processes in chronological order according to the sequence described in this specification, a program for performing processes in parallel, or a program for performing processes at a necessary time (for example, when making a call).

[0109] As described above, in Figure 1 the shown autonomous driving system 10, coefficients (dedicated coefficients) of the DNN corresponding to the area where each autonomous driving vehicle 100 is traveling and corresponding to the weather at that time are sent from the cloud server 200 to the autonomous driving vehicle 100. This arrangement enables the accuracy of the recognition result of the external environment to be improved by the inference DNN unit 101 of each autonomous driving vehicle 100. Therefore, power, braking, etc. in autonomous driving can be controlled more accurately.

[0110] <2. Modification>

[0111] Note that in the above-described embodiments, the region is not specifically mentioned; however, for a dangerous region, it is also conceivable to perform learning using a reduced region. Performing learning using a reduced region in this way can improve the accuracy of the coefficients of the learned DNN.

[0112] In addition, in the above-described embodiments, an example is given in which the movable object is the vehicle 100. However, even if the movable object is, for example, an autonomous walking robot, a flying object such as a drone, etc., the present technology is equally applicable. For example, in the case of a flying object such as a drone, it is also conceivable to define image features related to the flight altitude. For example, a height of 0 to 2 meters is close to the human perspective, while a height of not less than several tens of meters is a scene captured from the air.

[0113] The preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings; however, the technical scope of the present disclosure is not limited to the examples. Obviously, those with ordinary knowledge in the technical field of the present disclosure can conceive various types of changes or modifications within the scope of the technical idea described in the claims, and thus, it can also be naturally understood that such changes or modifications belong to the technical scope of the present disclosure.

[0114] In addition, the effects described in this specification are merely illustrative or exemplary, and thus are not restrictive. That is, the technology according to the present disclosure can exhibit other effects that are obvious to those skilled in the art from the description of this specification, together with or instead of the above effects.

[0115] Note that the present technology can also adopt the following configurations.

[0116] (1) An information processing apparatus, comprising:

[0117] An image data acquisition unit configured to acquire image data having image features corresponding to the moving scene of a movable object; and

[0118] A learning DNN unit configured to: perform learning using the image data acquired by the image data acquisition unit to obtain coefficients of an inference DNN for identifying the external environment of the movable object from the image data of the moving scene.

[0119] (2) The information processing apparatus according to (1) above,

[0120] wherein the image features include a position element.

[0121] (3) The information processing apparatus according to (2) above,

[0122] Among them, the image features further include weather elements.

[0123] (4) The information processing device according to (2) or (3) above,

[0124] Among them, the image features further include date and time elements.

[0125] (5) The information processing device according to any one of (1) to (4) above,

[0126] Among them, the learning DNN unit performs transfer learning using the image data acquired by the image data acquisition unit based on the coefficients of the inference DNN in the first time zone to obtain the coefficients of the inference DNN to be used in the second time zone after the first time zone. (6) The information processing device according to any one of (1) to (5) above, further comprising:

[0127] An image data receiving unit configured to receive the image data of the moving scene from the movable object in a state where position information and date and time information are added to the image data.

[0128] (7) The information processing device according to any one of (1) to (6) above, further comprising:

[0129] A coefficient sending unit configured to send the coefficients of the inference DNN acquired by the learning DNN unit to the movable object.

[0130] (8) The information processing device according to (7) above,

[0131] Among them, when the evaluation value of the coefficients of the inference DNN acquired by the learning DNN unit is higher than the evaluation value of the general coefficients, the coefficient sending unit sends the coefficients of the inference DNN acquired by the learning DNN unit to the movable object.

[0132] (9) An information processing method, comprising:

[0133] A process of acquiring image data having image features corresponding to the moving scene of a movable object; and

[0134] A process of performing learning using the acquired image data to obtain coefficients of an inference DNN for identifying the external environment of the movable object from the image data of the moving scene.

[0135] (10) A program for causing a computer to function as the following device:

[0136] An image data acquisition device configured to acquire image data having image features corresponding to a movement scenario of a movable object; and

[0137] A DNN device for learning, configured to: perform learning using the image data acquired by the image data acquisition device to obtain coefficients of an inference DNN for identifying an external environment of the movable object from the image data of the movement scenario.

[0138] (11) A movable object, comprising:

[0139] An inference DNN unit configured to identify an external environment from image data of a movement scenario;

[0140] A control unit configured to control movement based on an identification result from the inference DNN unit; and

[0141] A coefficient receiving unit configured to receive coefficients of an inference DNN to be used by the inference DNN unit from a cloud server,

[0142] wherein the coefficients of the inference DNN are obtained by performing learning using image data having image features corresponding to the movement scenario.

[0143] (12) The movable object according to (11) above, further comprising:

[0144] An image data sending unit configured to: send the image data of the movement scenario to the cloud server in a state where position information and date and time information are added to the image data.

[0145] (13) The movable object according to (11) or (12) above, further comprising:

[0146] A DNN unit for learning, configured to: perform learning using the image data of the movement scenario to obtain the coefficients of the inference DNN; and

[0147] A coefficient sending unit configured to send the coefficients of the inference DNN obtained by the DNN unit for learning to the cloud server.

[0148] (14) The movable object according to any one of (11) to (13) above,

[0149] Among them, when the movable object moves toward the second region side in the overlapping region between the first region and the second region, when the coefficient receiving unit receives the coefficients of the DNN for inference corresponding to the second region from the cloud server, the DNN unit for inference switches from the coefficients of the DNN for inference corresponding to the first region to the coefficients of the DNN for inference corresponding to the second region.

[0150] (15) The movable object according to any one of (11) to (13) above,

[0151] Among them, the DNN unit for inference includes a first DNN for inference and a second DNN for inference.

[0152] When the movable object moves in the first region, the coefficient receiving unit receives the coefficients of the DNN for inference corresponding to the second region to which the movable object will next move.

[0153] The coefficients of the DNN for inference corresponding to the first region are set as the first DNN for inference, and the coefficients of the DNN for inference corresponding to the second region are set as the second DNN for inference, and

[0154] When the movable object moves from the first region to the second region, the DNN unit for inference switches from the first DNN for inference in use to the second DNN for inference to be used.

[0155] (16) The movable object according to any one of (11) to (13) above further includes:

[0156] A memory configured to store the coefficients of the DNN for inference corresponding to the region in which the movable object moves and the coefficients of the DNN for inference corresponding to other regions around the region received by the coefficient receiving unit.

[0157] Among them, when the movable object moves from the first region to the second region, the DNN unit for inference extracts the coefficients of the DNN for inference corresponding to the second region from the memory and uses the extracted coefficients of the DNN for inference.

[0158] List of reference numerals

[0159] 10 Autopilot system

[0160] 100 Autonomous driving vehicle

[0161] 101 DNN unit for inference

[0162] 102 Capture unit

[0163] 103 Position / Date and Time Acquisition Unit

[0164] 104 Image Data Memory

[0165] 105 Data Transmission Unit

[0166] 106 Data Reception Unit

[0167] 107 Coefficient Memory of DNN

[0168] 108 Control Unit

[0169] 109 DNN Unit for Learning

[0170] 200 Cloud Server

[0171] 201 DNN Unit for Learning

[0172] 202 Data Reception Unit

[0173] 203 Image Database Unit

[0174] 204 Coefficient Database Unit

[0175] 205 Data Transmission Unit

[0176] 241, 241a, 241b Storage Unit

[0177] 242 Coefficient Determination Unit for Output DNN

[0178] 300 Internet

Claims

1. An information processing apparatus, comprising: An image data acquisition unit configured to acquire image data having image features corresponding to a movement scenario of a movable object; A learning deep neural network unit configured to perform learning using the image data acquired by the image data acquisition unit to obtain coefficients of an inference deep neural network for identifying an external environment of the movable object from the image data of the movement scenario; And A coefficient transmission unit configured to transmit the coefficients of the inference deep neural network obtained by the learning deep neural network unit to the movable object, wherein the coefficient transmission unit transmits the coefficients of the inference deep neural network corresponding to a second area to which the movable object will next move to the movable object when the movable object moves in a first area, wherein the learning deep neural network unit performs transfer learning using the image data acquired by the image data acquisition unit based on the coefficients of the inference deep neural network in a first time zone to obtain the coefficients of the inference deep neural network to be used in a second time zone after the first time zone, and wherein the lengths of the first time zone and the second time zone are defined at a predetermined time interval or according to the calculation speed of learning.

2. The information processing apparatus according to claim 1, wherein, The image features include position elements.

3. The information processing apparatus according to claim 2, wherein, The image features further include weather elements.

4. The information processing apparatus according to claim 2, wherein, The image features further include date and time elements.

5. The information processing apparatus according to claim 1, further comprising: An image data reception unit configured to receive the image data of the movement scenario from the movable object in a state where position information and date and time information are added to the image data.

6. The information processing apparatus according to claim 1, wherein, When an evaluation value of the coefficients of the inference deep neural network obtained by the learning deep neural network unit is higher than an evaluation value of general coefficients, the coefficient transmission unit transmits the coefficients of the inference deep neural network obtained by the learning deep neural network unit to the movable object.

7. An information processing method, comprising: A process of acquiring image data having image features corresponding to a movement scenario of a movable object; A process of performing learning using the acquired image data to obtain coefficients of an inference deep neural network for identifying an external environment of the movable object from the image data of the movement scenario; And A process of transmitting the obtained coefficients of the inference deep neural network to the movable object, wherein when the movable object moves in a first area, the coefficients of the inference deep neural network corresponding to a second area to which the movable object will next move are transmitted to the movable object, wherein transfer learning is performed using the acquired image data based on the coefficients of the inference deep neural network in a first time zone to obtain the coefficients of the inference deep neural network to be used in a second time zone after the first time zone, and wherein the lengths of the first time zone and the second time zone are defined at a predetermined time interval or according to the calculation speed of learning.

8. A program product for causing a computer to function as: An image data acquisition device configured to acquire image data having image features corresponding to a moving scene of a movable object; A deep neural network device for learning, configured to: perform learning using the image data acquired by the image data acquisition device to obtain coefficients of an inference deep neural network for identifying an external environment of the movable object from the image data of the moving scene; and A coefficient transmission device configured to transmit the coefficients of the inference deep neural network obtained by the deep neural network device for learning to the movable object, wherein, When the coefficient transmitting device determines that the movable object is moving in the first area, it transmits to the movable object the coefficients of the deep neural network for inference corresponding to the second area to which the movable object will next move. The deep neural network device for learning performs transfer learning using the image data acquired by the image data acquisition device based on the coefficients of the deep neural network for inference in the first time zone, so as to obtain the coefficients of the deep neural network for inference to be used in the second time zone after the first time zone, and the lengths of the first time zone and the second time zone are defined at predetermined time intervals or according to the computing speed of learning.

9. A movable object, comprising: A deep neural network unit for inference, configured to recognize the external environment from the image data of the moving scene; A control unit, configured to control the movement based on the recognition result from the deep neural network unit for inference; and A coefficient receiving unit, configured to receive from the cloud server the coefficients of the deep neural network for inference to be used by the deep neural network unit for inference, wherein the coefficients of the deep neural network for inference are obtained by performing learning using image data having image features corresponding to the moving scene, and when the movable object is moving toward the second area side in the overlapping area between the first area and the second area, and the coefficient receiving unit receives from the cloud server the coefficients of the deep neural network for inference corresponding to the second area, the deep neural network unit for inference switches from the coefficients of the deep neural network for inference corresponding to the first area to the coefficients of the deep neural network for inference corresponding to the second area. The coefficients of the deep neural network for inference are obtained by performing transfer learning using the acquired image data based on the coefficients of the deep neural network for inference in the first time zone, so as to obtain the coefficients of the deep neural network for inference to be used in the second time zone after the first time zone, and the lengths of the first time zone and the second time zone are defined at predetermined time intervals or according to the computing speed of learning.

10. A movable object, comprising: A deep neural network unit for inference, configured to recognize the external environment from the image data of the moving scene; A control unit, configured to control the movement based on the recognition result from the deep neural network unit for inference; and A coefficient receiving unit, configured to receive from the cloud server the coefficients of the deep neural network for inference to be used by the deep neural network unit for inference, wherein the coefficients of the deep neural network for inference are obtained by performing learning using image data having image features corresponding to the moving scene, the deep neural network unit for inference includes a first deep neural network for inference and a second deep neural network for inference, when the movable object is moving in the first area, the coefficient receiving unit receives the coefficients of the deep neural network for inference corresponding to the second area to which the movable object will next move. The coefficients of the deep neural network for inference corresponding to the first region are set to the first deep neural network for inference, the coefficients of the deep neural network for inference corresponding to the second region are set to the second deep neural network for inference, and when the movable object moves from the first region to the second region, the deep neural network unit for inference switches from the first deep neural network for inference in use to the second deep neural network for inference to be used, wherein the coefficients of the deep neural network for inference are obtained by performing transfer learning using the acquired image data based on the coefficients of the deep neural network for inference in the first time zone to obtain the coefficients of the deep neural network for inference to be used in a second time zone after the first time zone, and wherein the lengths of the first time zone and the second time zone are defined at a predetermined time interval or according to the learning calculation speed.

11. The movable object according to claim 9 or 10, further comprising: An image data sending unit configured to send the image data of the moving scenario to the cloud server in a state where position information and date and time information are added to the image data.

12. The movable object according to claim 9 or 10, further comprising: A deep neural network unit for learning configured to perform learning using the image data of the moving scenario to obtain the coefficients of the deep neural network for inference; and A coefficient sending unit configured to send the coefficients of the deep neural network for inference obtained by the deep neural network unit for learning to the cloud server.

13. A computer-readable medium having a program recorded thereon, the program, when executed by a computer, causing the computer to execute a method, the method comprising: A process of acquiring image data having image features corresponding to the moving scenario of the movable object; A process of performing learning using the acquired image data to obtain the coefficients of the deep neural network for inference for identifying the external environment of the movable object from the image data of the moving scenario; and A process of sending the obtained coefficients of the deep neural network for inference to the movable object, wherein, when the movable object moves in the first region, the coefficients of the deep neural network for inference corresponding to the second region to which the movable object will next move are sent to the movable object, wherein transfer learning is performed using the acquired image data based on the coefficients of the deep neural network for inference in the first time zone to obtain the coefficients of the deep neural network for inference to be used in a second time zone after the first time zone, and wherein the lengths of the first time zone and the second time zone are defined at a predetermined time interval or according to the learning calculation speed.

Images