Information processing device, information processing method, and program

Abstract

An information processing device according to the present technology is provided with a processing unit. The processing unit may process a plurality of tasks for the identification target, the plurality of tasks including a first task and a second task including the same feature quantity extraction. The processing unit determines whether to execute the processing of the second task using the recognition result of the recognition target of the processing of the first task.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program. Background Art

[0002] It is known to install a plurality of cameras on a vehicle to obtain information about the vehicle's surroundings in order to improve safety when driving a vehicle, etc. For example, Patent Document 1 describes detecting a moving object by performing recognition processing using images from a camera installed on a vehicle.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-123470 Summary of the invention

[0006] Technical issues

[0007] In order to recognize all surroundings of the vehicle with high accuracy, similar recognition processing is always performed on images acquired by each of a plurality of camera devices mounted on the vehicle, which results in a large total amount of calculations.

[0008] Therefore, in the field of recognition processing, a technology capable of reducing the amount of calculation without reducing recognition accuracy is required.

[0009] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program capable of reducing the amount of calculation without reducing recognition accuracy.

[0010] Solution to the problem

[0011] In order to achieve the above-mentioned object, an information processing device according to an embodiment of the present technology includes a processing section.

[0012] The processing unit is capable of processing a plurality of tasks for a recognition target, wherein the plurality of tasks include a first task and a second task that share feature extraction.

[0013] The processing unit determines whether to execute the processing of the second task using the recognition result of the recognition target from the processing of the first task.

[0014] With this configuration, since whether to execute the processing of the second task is decided based on the recognition result from the processing of the first task, the calculation amount of the overall recognition processing can be reduced without reducing the recognition accuracy.

[0015] The processing section may generate the parameter of the second task using the recognition result of the recognition target from the processing of the first task.

[0016] The processing unit may use the generated parameters to configure the neural network of the second task.

[0017] The processing section may extract a plurality of features from the recognition target, and use a recognition result of the recognition target from the processing of the first task using the plurality of features to decide whether to execute the processing of the second task and generate the parameter.

[0018] The parameters may include a region to be processed of the second task and one or more features selected from a plurality of features.

[0019] The processing unit may determine whether to execute the processing of the second task and generate the parameter using the scene feature obtained from the recognition result of the recognition object from the processing of the first task.

[0020] The recognition target may be an image acquired by an image capturing unit mounted on the mobile object, the image capturing unit capturing the surrounding environment of the mobile object, and

[0021] The scene feature may be a moving scene feature of a moving object, which refers to whether there is an object of interest in the image and whether the object of interest is a movable object.

[0022] The object of interest may be an object that hinders the movement of the moving object.

[0023] The first task may be semantic segmentation, the second task may include object detection, motion detection, and distance detection, and if there is no object of interest in the image, the processing unit may only perform distance detection, if there is an object of interest in the image and the object of interest is not a movable object, the processing unit performs object detection and distance detection, and if there is an object of interest in the image and the object of interest is a movable object, the processing unit performs object detection, motion detection, and distance detection.

[0024] The parameters may include a region to be processed of the second task and one or more features selected from a plurality of features, and

[0025] If the object of interest does not exist in the image, the processing unit may generate parameters so that the entire image area is used as the area to be processed and all of the plurality of features are used, and

[0026] If the object of interest exists in the image, the processing unit generates parameters such that a minimum area surrounding the object of interest is used as an area to be processed and one or more features selected from a plurality of features are used according to the number of pixels in the object of interest.

[0027] The distance measuring unit may be mounted on the moving object, and

[0028] In the distance detection, the distance may be estimated using an aggregation result of aggregating features extracted from an image and distance features obtained by the distance measurement section.

[0029] The distance measuring section may include one or more selected from LiDAR (Light Detection and Ranging), a stereo camera, and a millimeter wave radar.

[0030] A plurality of image capturing units may be mounted on the moving object, and

[0031] The processing section may decide whether to perform the processing of the second task and generate the parameter using the image recognition result from the processing of the first task for each image acquired by each of the plurality of image capturing sections mounted on the mobile object.

[0032] The image capturing unit may be a stereo camera or a monocular camera.

[0033] The processing unit may perform a second task on the image using a neural network of the second task configured using the generated parameters, and the information processing device may also include a presentation control unit that controls a presentation unit that provides assistance to an operator of the mobile object based on a recognition result of the second task.

[0034] One or more selected from a display section, a light emitting section, and a sound output section may be installed on the mobile object as a presentation section, and

[0035] The presentation control section may control at least one of display control of the display section, lighting control of the light emitting section, and sound output control of the sound output section.

[0036] The mobile object may be a mobile object capable of moving autonomously, and

[0037] The processing unit may perform the second task on the image using a neural network of the second task configured by using the generated parameters, and the information processing device may further include a planning unit that plans travel and action of the mobile object based on the recognition result of the second task.

[0038] The recognition target can be an image,

[0039] The first task can be semantic segmentation, and

[0040] The second task may include one or more selected from object detection, motion detection, distance detection, normal estimation, pose estimation, and trajectory estimation.

[0041] According to an embodiment of the present technology, an information processing method is performed by an information processing device that processes a first task for a recognition target and uses a recognition result of the recognition target processed by the first task to decide whether to perform a second task that shares feature extraction with the first task.

[0042] The program according to the embodiment of the present technology causes an information processing device to execute the following steps:

[0043] a step of processing a first task for identifying a target, and

[0044] The recognition result of the recognition target from the processing of the first task is used to decide whether to perform the steps of the second task that shares feature extraction with the first task. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] [ Figure 1 ] is a schematic diagram showing a configuration example of a processing unit of an information processing device according to each embodiment of the present technology.

[0046] [ Figure 2 ] is a flowchart showing the image recognition processing method (information processing method) in the above-mentioned processing unit.

[0047] [ Figure 3 ] is a diagram showing a top view of a vehicle and an example of the installation position of the sensor unit.

[0048] [ Figure 4 ] is a schematic diagram showing a configuration example of an information processing system according to the second embodiment.

[0049] [ Figure 5 ] is a flowchart showing the image recognition processing method (information processing method) in the second embodiment and the third embodiment.

[0050] [ Figure 6 ] is a schematic diagram of a neural network showing feature extraction.

[0051] [ Figure 7 ] is a schematic diagram showing a neural network for semantic segmentation.

[0052] [ Figure 8 ] is a schematic diagram of a neural network showing instance segmentation (object detection).

[0053] [ Fig. 9 ] is a schematic diagram of a neural network showing optical flow (motion detection).

[0054] [ Fig.10 ] is a schematic diagram of a neural network showing distance detection.

[0055] [ Fig.11] is a flowchart showing an image recognition processing method (information processing method) according to each embodiment.

[0056] [ Fig.12 ] is shown Fig.11 This is a flowchart of details of the second task decision processing in step 4 of the flowchart.

[0057] [ Fig.13 ] is a diagram describing a specific example of the second task determination process.

[0058] [ Fig.14 ] is shown Fig.11 A flowchart of details of the parameter generation process in step 5 of the flowchart.

[0059] [ Fig.15 ] is a diagram showing a specific example of parameter generation.

[0060] [ Fig.16 ] is a diagram showing a specific example of parameter generation.

[0061] [ Fig.17 ] is a diagram showing a configuration example of a neural network.

[0062] [ Fig.18 ] is a diagram that uses distance detection as an example to illustrate that the amount of computation on the decoder side can be reduced by reconfiguring the neural network.

[0063] [ Fig.19 ] is a schematic diagram showing a configuration example of an information processing system according to a third embodiment. DETAILED DESCRIPTION

[0064] Hereinafter, each embodiment of the present technology will be described with reference to the accompanying drawings. In the following description, similar configurations will be marked with similar symbols, and the description of the previously described configurations may be omitted. In the description of the information processing method, similar steps are numbered with similar step numbers, and the description of the previously described steps may be omitted.

[0065] In the present technology, in the recognition processing of the recognition target performed by the information processing device capable of processing a plurality of different tasks (which share feature extraction) for the recognition target, it is decided whether to execute the remaining other tasks based on the recognition result obtained in one of the tasks. This configuration makes it possible to reduce the amount of calculation without reducing the recognition accuracy.

[0066] In the following description, an example is given in which the recognition target is an image (camera image). In the first to third embodiments described below, an example is given in which the present technology is applied to recognition processing of an image acquired by an image capture unit mounted on a four-wheeled vehicle as a moving object (hereinafter referred to as a "vehicle"). The image contains vehicle surrounding environment information (vehicle external information), and the driving scene (moving scene) of the vehicle can be estimated from the image.

[0067] The first embodiment focuses on the characteristic configuration of the present technology. In the first embodiment, an overview of image recognition processing performed on a single image (hereinafter referred to as an input image) is described.

[0068] In the second and third embodiments, the present technology is applied to image recognition processing of images acquired by each of the multiple image capture units by taking the example of installing multiple image capture units on a vehicle to sense all surrounding environments of the vehicle. The image recognition processing to which the present technology is applied will be described in more detail in the second embodiment.

[0069] The image recognition processing in the information processing device according to each of the first, second and third embodiments is the same. In the second embodiment, an example of providing driving support to the driver (i.e., the operator of the vehicle) based on the image recognition processing results is given. In the third embodiment, an example is given in which the vehicle is able to move autonomously, and a vehicle driving route plan and a vehicle action plan are executed based on the image recognition processing results. Autonomous movement refers to so-called automatic driving, in which the vehicle moves autonomously without the need for driver operation. Typically, the vehicle can be switched between manual driving and automatic driving by the driver. The term "driving" means "movement of a moving object."

[0070] In addition, it may be configured to provide both the driving support described in the second embodiment and the automatic driving control described in the third embodiment.

[0071] In the following description, “right” and “left” refer to “right” and “left” viewed from a driver in a vehicle, “front” refers to a direction in which the vehicle is traveling, and “rear” refers to the opposite direction to the traveling direction.

[0072] <First Embodiment>

[0073] Figure 1 1 is a schematic diagram showing a configuration example of the processing section 3 of the information processing device 10 according to the first embodiment. Figure 21 is a flowchart showing an example of an image processing method (information processing method) in the processing section 3. The processing section 3 of the information processing apparatus 10a according to the second embodiment and the processing section 3 of the information processing apparatus 10b according to the third embodiment, which will be described below, have the same configuration as the processing section 3 of the information processing apparatus 10.

[0074] [Information processing device]

[0075] The information processing device 10 can perform multiple tasks related to image recognition processing on a single image. The image is a recognition target and is acquired by an image capture unit installed on a vehicle. Hereinafter, the image of the recognition target may be referred to as an "input image". In the information processing device 10 according to the present embodiment, multiple tasks sharing the same feature extraction can be processed simultaneously for a single input image. In other words, the information processing device 10 can perform multiple different task processing using multiple features extracted by the same feature extractor (FeatureExtractor). Hereinafter, the feature extractor is marked with symbol 37. The details are described in the second embodiment.

[0076] exist Figure 2 In , the number of tasks is set to N.

[0077] In this embodiment and the second and third embodiments described below, examples of processing four recognition tasks are given: category classification through semantic segmentation, object detection through instance segmentation, motion detection through optical flow, and distance detection.

[0078] A plurality of tasks (four tasks in this embodiment) that can be performed by the information processing device 10 are classified into a first task and a second task. The first task and the second task are recognition tasks, and more specifically, image recognition tasks in this embodiment. The image recognition task can be performed on an image acquired by an image capture unit mounted on a vehicle, and the recognition processing result can be used to perform driving support presentation, automatic driving control, etc.

[0079] Semantic segmentation (category classification) is the first task. Instance segmentation (object detection), optical flow (motion detection), and distance detection are the second tasks.

[0080] Semantic segmentation as the first task is category classification. With semantic segmentation, each pixel in the input image is classified into the object category (classification) to which the pixel belongs.

[0081] Instance segmentation as the second task is object detection (also called instance detection). In instance segmentation, objects in an input image are detected. In the following description, it is mainly referred to as "object detection".

[0082] The optical flow as the second task is motion detection. In the optical flow, the motion of an object in an input image is detected. Specifically, in the optical flow, the motion of an object of interest between two temporally consecutive input image frames is estimated. In the following description, it is mainly referred to as "motion detection".

[0083] In distance detection as the second task, the distance between the object and the vehicle (the vehicle on which the image capture unit is mounted) is estimated using features extracted by the feature extractor 37 and a LiDAR point cloud acquired by a LiDAR (Light Detection and Ranging) as a distance measuring unit (described below).

[0084] The details of each task process will be described in the second embodiment described below.

[0085] In the image recognition processing performed by the processing unit 3 of the information processing device 10, based on the recognition result obtained from the semantic segmentation (first task), it is decided whether to perform the remaining other tasks (second tasks), that is, object detection (instance segmentation), motion detection (optical flow), and distance detection. This configuration reduces the amount of calculation. A specific example is described in the second embodiment.

[0086] like Figure 1 As shown in , the information processing device 10 has an image acquisition unit 30 and a processing unit 3. The processing unit 3 has a feature extraction unit 31, a first task estimation unit 32, a second task determination unit 33, a parameter generation unit 34, a second task neural network configuration unit 35 and a second task estimation unit 36.

[0087] The image acquisition section 30 acquires the image (input image) acquired by the image capturing section.

[0088] The feature extraction unit 31 extracts a plurality of common features of a plurality of tasks (a first task and a second task) from the input image. The feature extraction unit 31 includes a feature extractor 37 (for example, see Figure 6 wait).

[0089] The first task estimation section 32 performs semantic segmentation on the input image using deep learning, and performs category classification of the object on a pixel-by-pixel basis.

[0090] The scene features of the input image can be estimated based on the semantic segmentation results (the input image recognition results from the first task processing). For example, the driving scene features (mobile scene features) can be obtained from the semantic segmentation results. In the following, the "driving scene features" are referred to as "scene features".

[0091] The scene feature indicates whether there is an object that hinders driving and whether the object is a movable object. In the following, the "object that hinders driving" may be referred to as the "object of interest".

[0092] Objects that hinder driving are, for example, vehicles (e.g., cars, motorcycles, bicycles, trains), telephone poles, traffic lights, trash cans, pillars, people, animals, etc. On the other hand, objects that do not hinder driving are, for example, roads, the sky, etc.

[0093] Objects that hinder driving can be further classified into movable objects and immovable objects (so-called stationary objects). Movable objects are, for example, vehicles, people, animals, etc.

[0094] The second task decision unit 33 decides whether to execute the second task (i.e., object detection, motion detection, and distance detection, respectively) based on the semantic segmentation result. Based on the scene features obtained from the semantic segmentation result, the second task decision unit 33 decides the second task to be executed to obtain an image recognition processing result required for obtaining information required for autonomous driving or driving support.

[0095] If there is no object of interest (object that hinders driving) in the input image, the second task decision unit 33 decides not to perform object detection and motion detection but to perform only distance detection.

[0096] If there is an object of interest but no movable object in the input image, the second task decision section 33 decides not to perform motion detection but to perform only object detection and distance detection.

[0097] If there is a movable object of interest, the second task decision section 33 decides to execute all the second tasks of object detection, motion detection, and distance detection.

[0098] Therefore, since the second task (image recognition task) to be performed is determined based on the scene features obtained from the semantic segmentation result so that the desired image recognition processing result is obtained, the computational complexity of the recognition processing can be reduced without reducing the recognition accuracy.

[0099] The parameter generation unit 34 uses the semantic segmentation result to dynamically generate parameters for determining the second task to be performed. The parameters of the second task include: a target image to be processed for performing the second task, and features to be used when performing the second task. The target image to be processed is the smallest rectangular image area containing the object of interest (described below). The features to be used when performing the second task are determined based on the number of pixels of the object of interest.

[0100] According to the semantic segmentation results, the category of the object of interest and the image region where the object of interest is located can be determined.

[0101] The parameter generation unit 34 generates parameters for object detection using the semantic segmentation result in object detection so as to process a partial neural network corresponding to the number of pixels in the image region where the object of interest is located and the object category.

[0102] When there is an object of interest in distance detection, the parameter generation unit 34 generates parameters for distance detection so that only an image region (image to be processed) including the object of interest is processed, and a partial neural network is processed according to the number of pixels of the object of interest.

[0103] On the other hand, when there is no object of interest in distance detection, the parameter generation unit 34 generates parameters for distance detection so that the entire input image is processed and all features are used.

[0104] When there is a movable object of interest in motion detection, the parameter generation unit 34 generates parameters for motion detection so that only the image area (image to be processed) including the movable object of interest is processed, and a partial neural network is processed according to the number of pixels of the object of interest.

[0105] The second-task neural network configuration section 35 uses the parameters generated by the parameter generation section 34 to configure a neural network for each of the following second tasks: object detection, motion detection, and distance detection.

[0106] If all features f1 to f5 are generated as parameters by the parameter generation unit 34, the second task neural network configuration unit 35 uses a standard neural network as a neural network for the second task processing. In this embodiment, the "standard neural network" refers to a neural network configured to use all features f1 to f5, each of which is respectively in Figures 7 to 9 As shown in the figure, it is described below.

[0107] The second task estimation section 36 performs object detection (instance segmentation) using the configured neural network for object detection.

[0108] The second task estimation section 36 performs distance detection using a neural network configured for distance detection and estimates the distance.

[0109] The second task estimation section 36 performs motion detection (optical flow) using the configured motion detection neural network and estimates the motion of the object.

[0110] The information processing device 10 has a hardware configuration required for a computer, such as a central processing unit (CPU) and a memory (RAM, ROM). In the information processing device 10, the CPU loads a program stored in a storage unit (not shown) into the RAM and executes the program to perform various processes. The storage unit stores a program for performing the image recognition process according to the present embodiment.

[0111] [Information processing method (image recognition processing method)]

[0112] Figure 2This section describes an information processing method (image recognition processing method) executed by the processing unit 3 of the information processing device 10 of this embodiment. Figure 2 In FIG. 1 , a plurality of second tasks different from each other are shown as a second task a, a second task b, a second task c, ..., and a second task N. N corresponds to the number of second tasks. For example, in Figure 2 , the second task a indicates object detection (instance segmentation), the second task b indicates motion detection (optical flow), and the second task c indicates distance detection. The number of the second tasks is not limited and may be one or more.

[0113] like Figure 2 As shown in FIG. 1 , when an image is input to the processing unit 3 , a plurality of features of the input image are extracted by the feature extraction unit 31 ( S2 ).

[0114] Next, the first task estimation unit 32 uses the multiple features extracted in S2 to perform the first task (semantic segmentation in this embodiment) (S3). The semantic segmentation result (first task processing result) is output to the second task determination unit 33. The semantic segmentation result can be used to estimate the scene features of the input image.

[0115] Next, the second task decision unit 33 uses the semantic segmentation result to decide whether to execute the second tasks a, b, c to N respectively (S4). In this embodiment, the second task decision unit 33 decides whether to execute object detection, motion detection and distance detection respectively. If there are other second tasks, it also decides whether to execute these second tasks.

[0116] Next, the parameter generating unit 34 generates parameters of the second task ( S5 ).

[0117] exist Figure 2 , the step of generating parameters for the second task a (object detection) is represented as S5a. The step of generating parameters for the second task b (motion detection) is represented as S5b. The step of generating parameters for the second task c (distance detection) is represented as S5c. The step of generating parameters for the second task N is represented as S5N. If there is no need to distinguish between these S5a, S5b, S5c, ..., S5N, they are represented as S5.

[0118] In this embodiment, if Figure 2 As shown in , the semantic segmentation result is used to generate the parameters of the second task a (S5a). The semantic segmentation result is used to generate the parameters of the second task b (S5b). The semantic segmentation result is used to generate the parameters of the second task c (S5c). The semantic segmentation result is used to generate the parameters of the second task N (S5N).

[0119] Next, the second task neural network configuration unit 35 configures the neural network of the second task using the parameters generated in S5 ( S6 ).

[0120] The neural network configured here is a standard neural network or a neural network reconfigured so that only a portion of the features are used.

[0121] exist Figure 2 , the step of configuring the neural network of the second task a is represented as S6a. The step of configuring the neural network of the second task b is represented as S6b. The step of configuring the neural network of the second task c is represented as S6c. The step of configuring the neural network of the second task N is represented as S6N. If there is no need to distinguish between these S6a, S6b, S6c, ..., S6N, they are represented as S6.

[0122] In this embodiment, if Figure 2 As shown in , the parameters generated in S5a are used to configure the neural network of the second task a (S6a). The parameters generated in S5b are used to configure the neural network of the second task b (S6b). The parameters generated in S5c are used to configure the neural network of the second task c (S6c). The parameters generated in S5N are used to configure the neural network of the second task b (S6N).

[0123] Next, the second task is executed by the second task estimation section 36 using the neural network configured in S6 ( S7 ).

[0124] exist Figure 2 In the embodiment, the step of estimating using the configured neural network of the second task a is represented as S7a. The step of estimating using the configured neural network of the second task b is represented as S7b. The step of estimating using the configured neural network of the second task c is represented as S7c. The step of estimating using the configured neural network of the second task N is represented as S7N. If there is no need to distinguish between these S7a, S7b, S7c, ..., S7N, they are represented as S7.

[0125] In this embodiment, if Figure 2 As shown in , instance segmentation is performed using the configured neural network of the second task a (object detection) ( S7a ), and an object detection result (recognition processing result of the second task a) is output.

[0126] Optical flow is performed using the configured neural network of the second task b (motion detection) ( S7 b ), and a motion detection result (recognition processing result of the second task b) is output.

[0127] Distance detection is performed using the configured neural network of the second task c (distance detection) ( S7c ), and a distance detection result (a recognition processing result of the second task c) is output.

[0128] By using the output recognition processing results (semantic segmentation results, object detection results, motion detection results, and distance detection results), driving support (described in detail in the second embodiment) and autonomous driving (described in detail in the third embodiment) can be made highly accurate while reducing power consumption.

[0129] In this embodiment and the second and third embodiments described below, an example is given in which the processing unit performs four task processing, but the number of tasks is not limited to four, but may be two or more, and may include a first task and one or more second tasks.

[0130] The processing result of the first task is used to decide whether to execute other second tasks. The first task is necessary to decide whether to execute other second tasks. In image recognition processing, semantic segmentation is usually used as the first task.

[0131] There may be one or more second tasks. Whether to execute the second task is determined based on the processing result of the first task. In image processing using the present technology, one or more tasks selected from object detection (instance segmentation), motion detection (optical flow), distance detection, normal estimation, pose estimation, trajectory estimation, etc. may be used as the second task.

[0132] In the first to third embodiments, the second task is an example of the following three tasks: object detection (instance segmentation), motion detection (optical flow), and distance detection.

[0133] As described above, in the present technology, the second task to be performed is determined based on the semantic segmentation result (the recognition result of the recognition target processed by the first task), and the parameters of the second task are also generated. Then, the neural network of the second task is configured using the generated parameters, and the configured neural network is used to perform the image recognition task.

[0134] Since the present technology uses the semantic segmentation result to decide the second task to be performed, only the recognition task required for image recognition can be performed, and the computational complexity of the overall recognition processing can be reduced without reducing the recognition accuracy.

[0135] In addition, in the present technology, by configuring the neural network using parameters generated by the semantic segmentation results, it is possible to process only part of the network according to the scene characteristics, thereby further reducing the computational complexity of the recognition processing while maintaining high recognition accuracy.

[0136] This enables to reduce power consumption and suppress processing delay, and can obtain real-time recognition processing results with high recognition accuracy. The recognition processing results can then be used to provide driving support and automatic driving control at precise timing with high accuracy.

[0137] As in the second and third embodiments described below, the present technology is also very effective in reducing the amount of calculation in a system that performs real-time image recognition processing for each of a plurality of images.

[0138] For example, for an omnidirectional sensing system in which multiple image capture units are installed on a vehicle to obtain vehicle surrounding environment information over a wide area, if four recognition tasks are always performed for each image acquired by each image capture unit, the amount of calculation becomes huge. However, by applying the image recognition processing method (information processing method) of the present technology to each image acquired by each image capture unit, the amount of calculation for the overall recognition processing can be reduced while maintaining high recognition accuracy. This makes it possible to reduce power consumption and processing delays in forms such as omnidirectional sensing systems that require image recognition processing of multiple images in real time, and enables driving support and autonomous driving control to be performed with high accuracy at precise timing.

[0139] <Second Embodiment>

[0140] In the second embodiment, the technique described in the first embodiment is applied to image recognition processing of each image acquired by each of a plurality of image capturing sections mounted on a vehicle, and the image recognition processing result is used for driving support.

[0141] [Vehicle example]

[0142] Figure 3 2 is a top view of a vehicle, and shows an example of the positions of a plurality of sensor sections 2 mounted on the vehicle. As will be described in detail below, each sensor 2 has an image capturing section 20 and a distance measuring section 21. Figure 3 The position of the sensor section 2 shown in FIG. 1 is only an example and is not limited thereto. If there is no need to distinguish between the plurality of sensor sections 2 , they are referred to as the sensor section 2 .

[0143] like Figure 3 As shown in FIG. 1 , for example, a front sensor unit 2F, two front sensor units 2Fa, a right front sensor unit 2FR, a left front sensor unit 2FL, a right sensor unit 2SR, a left sensor unit 2SL, a rear sensor unit 2R, and a rear sensor unit 2Ra are mounted on the vehicle 1. Any of the sensor units can acquire vehicle surrounding environment information.

[0144] The front sensor section 2F is located near the front bumper, and acquires vehicle surrounding environment information in front of the vehicle.

[0145] The two front sensor units 2Fa are located in front of the roof to acquire vehicle surrounding environment information in front of the vehicle.

[0146] The right front sensor unit 2FR is located in the front right side of the vehicle, and acquires vehicle surrounding environment information in the diagonally front right side of the vehicle.

[0147] The left front sensor unit 2FL is located in the front left side of the vehicle, and acquires vehicle surrounding environment information in the diagonally front left side of the vehicle.

[0148] The right side sensor portion 2SR is located rearward of the right front sensor portion 2FR, and acquires vehicle surrounding environment information at the right side of the vehicle.

[0149] The left side sensor portion 2SL is located rearward of the left front sensor portion 2FL, and acquires vehicle surrounding environment information at the left side of the vehicle.

[0150] The rear sensor section 2R is located near the rear bumper, and acquires vehicle surrounding environment information behind the vehicle.

[0151] The rear sensor section 2Ra is located behind the roof, and acquires vehicle surrounding environment information behind the vehicle.

[0152] In the second and third embodiments, for convenience, an example is shown in which the present technology is applied to recognition processing of the sensing result (image) of the image capture unit 20 from each of the following five sensor units: the front sensor unit 2F, the right front sensor unit 2FR, the left front sensor unit 2FL, the right sensor unit 2SR, and the left sensor unit 2SL. The number of sensor units 2 to which the present technology is applied is not limited to this, and one or more thereof is sufficient.

[0153] [Information Processing System]

[0154] Figure 4 1 is a schematic diagram of an information processing system 100 according to the present embodiment. In the information processing system 100, a driving support process is performed using image recognition processing results of acquired images of a plurality of image capture units 20 mounted on the vehicle 1. The information processing system 100 in the present embodiment can be restated as a driving support system.

[0155] like Figure 4 As shown in FIG. 1 , the information processing system 100 has a plurality of sensor sections 2 , an information processing device 10 a , a vehicle state detection section 5 , and a presentation section 6 . All of these are mounted on a vehicle 1 .

[0156] (Sensor part)

[0157] Each sensor section 2 includes an image capturing section 20 and a distance measuring section 21 .

[0158] The image capturing section 20 acquires an image, and is configured by, for example, a CMOS sensor. The image capturing section 20 in the present embodiment acquires an image of the surrounding environment of the vehicle 1. A monocular camera, a stereo camera, or the like can be used for the image capturing section 20.

[0159] The distance measuring unit 21 is configured to be able to measure the distance between the vehicle 1 on which the image capturing unit 20 is mounted and an object around the vehicle 1. As the distance measuring unit 21, LiDAR, a stereo camera, a millimeter wave radar, etc. can be used, and the distance measuring unit 21 is configured to include one or more selected from these. In the present embodiment, an example of using LiDAR as the distance measuring unit 21 is given.

[0160] The image acquired by the image capturing section 20 of the sensor section 2 and the 3D point cloud information acquired by the LiDAR as the distance measuring section 21 are output to the information processing device 10 a .

[0161] In each of the sensor sections 2, the image capturing section 20 and the distance measuring section 21 are generally positioned close to each other. Since the camera (image capturing section) and the LiDAR (distance measuring section) are generally installed at different positions, correspondence information between a camera coordinate system with the camera position as the origin and a LiDAR coordinate system with the LiDAR position as the origin is obtained and stored in advance. Using this correspondence information, image recognition processing related to distance detection can be performed by associating an image acquired by the camera with a 3D point cloud acquired by the LiDAR.

[0162] (Presentation)

[0163] The presentation unit 6 includes a device capable of outputting and presenting visual or auditory information to the driver of the vehicle 1. The presentation unit 6 is mounted on the vehicle 1. The presentation unit 6 can present information related to driving support to the driver of the vehicle 1, such as reporting vehicle surrounding environment information, prompting attention and warning, and suggesting a preferred speed and driving route.

[0164] The presentation section 6 includes, for example, a display section 60, a sound output section 61, and a light emitting section 62. Driving support can be provided using one or more selected from the display section 60, the sound output section 61, and the light emitting section 62. The driving support information is visually presented to the driver through display on the display section 60 and lighting or flashing on the light emitting section 62. The driving support information is auditorily presented to the driver through sound output in the sound output section 61.

[0165] Presentation of visual or auditory driving support information to the driver by the presentation section 6 is performed using the image recognition processing result in the processing section 3 of the image acquired in the sensor section 2 .

[0166] The driving support information is effective information for preventing accidents, etc., when driving a vehicle, for example. Examples of driving support include obstacle warnings, own vehicle collision warnings, own vehicle lane departure warnings, driving operation commands, speed change commands, vehicle overtaking advice, lane change advice, and notifications of driving condition information. The driver can drive more safely based on the driving support information presented by the presentation section 6.

[0167] The display unit 60 displays visual information in the driver's field of vision. The display unit 60 includes, for example, a display device, a dashboard, a wearable device (such as a glasses-type display worn by the driver), or a projector. The display unit 60 performs display under the control of the presentation control unit 4 described below.

[0168] The sound output section 61 includes, for example, a speaker, an alarm, a buzzer, etc. The sound output section 61 outputs sound information, notification sound, warning sound, etc. under the control of the presentation control section 4 described below.

[0169] The light emitting section 62 includes a light emitting device such as a lamp. The light emitting section 62 can be used as a warning lamp, for example, and the light emitting section 62 turns on or flashes light under the control of the presentation control section 4 described below for the purpose of notifying or warning the driver of various information.

[0170] (Vehicle status detection unit)

[0171] The vehicle state detection unit 5 detects the state of the vehicle. For example, the vehicle state detection unit 5 includes a gyro sensor, an acceleration sensor, and a sensor for detecting the operation amount of the accelerator pedal, the operation amount of the brake pedal, the steering angle, the engine speed, the motor speed, or the vehicle speed. The vehicle information detected by the vehicle state detection unit 5 (such as the speed and steering angle of the vehicle 1) is output to the presentation control unit 4 described below.

[0172] (Information Processing Device)

[0173] The information processing device 10a has a hardware configuration required for a computer, such as a CPU and memory (RAM, ROM). In the information processing device 10a, the CPU loads a program stored in the storage unit 7 (described below) into the RAM and executes the program to perform various processes, including image recognition processing according to the present technology.

[0174] In the information processing device 10a, semantic segmentation (first task) is performed for each input image from the image capturing unit 20 of each of the plurality of sensor units 2. The semantic segmentation result is used to decide a task (second task) to be used for image recognition processing, and a parameter of the task is generated.

[0175] The information processing device 10 a includes a processing section 3 , an image acquisition section 30 , a presentation control section 4 , a storage section 7 , and a situation analysis section 8 .

[0176] ((Image acquisition unit))

[0177] The image acquisition section 30 acquires the image acquired by the image capturing section 20 of each sensor section 2 and outputs the image to the processing section 3 .

[0178] ((Processing Department))

[0179] The processing unit 3 performs recognition processing on the image (input image) acquired by the image acquisition unit 30. At this time, as described in the first embodiment, the processing unit 3 uses the result of executing the first task on the input image to determine whether to execute the second task and generate parameters of the second task.

[0180] In addition, in this embodiment, as in the first embodiment, an example is described in which the first task is semantic segmentation and the second task is object detection (instance segmentation), motion detection (optical flow), and distance detection. Figure 4 In order to distinguish these second tasks, object detection is called the second task a, motion detection is called the second task b, and distance detection is called the second task c.

[0181] Hereinafter, it will be described in detail.

[0182] The processing unit 3 includes a feature extraction unit 31 , a first task estimation unit 32 , a second task determination unit 33 , a parameter generation unit 34 , a second task neural network configuration unit 35 , and a second task estimation unit 36 ​​.

[0183] The feature extraction unit 31 extracts a plurality of features of the input image by means of the feature extractor 37. The feature extraction will be described in detail below.

[0184] The first task estimation unit 32 performs semantic segmentation as the first task. The semantic segmentation result is output to the second task determination unit 33 and the situation analysis unit 8. The semantic segmentation will be described in detail below.

[0185] The second task decision unit 33 decides whether to execute the second task a, the second task b, and the second task c, respectively, based on the semantic segmentation result. The specific second task decision will be described below.

[0186] The parameter generation unit 34 generates parameters for executing the second task determined to be executed based on the semantic segmentation result, and stores the generated parameters in the storage unit 7. The specific parameter generation will be described below.

[0187] The second task neural network configuration unit 35 reads the parameters generated by the parameter generation unit 34 from the storage unit 7 , and configures the neural network of the second task using the parameters.

[0188] The second task estimation section 36 performs the second task processing using the configured neural network. The second task processing result (input image recognition result) is output to the situation analysis section 8.

[0189] The second task estimation unit 36 ​​has a second task a estimation unit 361, a second task b estimation unit 362, and a second task c estimation unit 363. The second task a estimation unit 361 performs instance segmentation and detects objects. The second task b estimation unit 362 performs optical flow and detects the motion of objects. The second task c estimation unit 363 performs distance detection. Object detection, motion detection, and distance detection will be described in detail below.

[0190] Figure 5 1 is a schematic flowchart showing an example of image recognition processing performed by the processing portion 3 of an image acquired by each of the plurality of sensor portions 2 mounted on the vehicle 1 in the second and third embodiments.

[0191] like Figure 5 As shown in , the image recognition processing (information processing) of the present technology is applied to an image acquired by each of the following five sensor units: the front sensor unit 2F, the right front sensor unit 2FR, the left front sensor unit 2FL, the right sensor unit 2SR, and the left sensor unit 2SL. In other words, in the processing unit 3, the processes S1 to S6 described in the first embodiment are performed for each of the input images from the plurality of sensor units 2.

[0192] In the image recognition processing for each of the images acquired by the plurality of sensor units 2, it is decided whether to execute the second task (i.e., another task) based on the recognition processing result of the first task. This makes it possible to reduce the amount of calculation of the overall recognition processing without reducing the recognition accuracy, thereby reducing power consumption and processing delay.

[0193] ((Situation Analysis Department))

[0194] The situation analysis unit 8 performs analysis processing of the surrounding environment of the vehicle based on the recognition processing result (semantic segmentation result) of the first task and the recognition processing result (one or more processing results selected from the object detection result, motion detection result and distance detection result) of the second task. The analysis result is output to the presentation control unit 4.

[0195] ((Presentation Control Unit))

[0196] The presentation control section 4 generates driving support information using the analysis result output from the situation analysis section 8 and the state information of the vehicle 1 detected by the vehicle state detection section 5 , and controls the presentation section 6 that presents the driving support information.

[0197] The presentation control section 4 includes, for example, a display control section 40 , a sound control section 41 , and a light emission control section 42 .

[0198] The display control section 40 controls display on the display section 60 .

[0199] The sound control section 41 controls the sound output at the sound output section 61 .

[0200] The light emission control unit 42 controls lighting of the light emitting unit 62 .

[0201] The analysis result of the situation analysis unit 8 described above is generated using the image recognition processing result in the processing unit 3. As described above, since the processing unit 3 can perform image recognition processing while reducing the amount of calculations and maintaining high recognition accuracy, the analysis result of the surrounding environment situation of the vehicle performed using the highly accurate image recognition processing result is highly accurate information. The driving support information generated using such highly accurate information is suitable for the situation of the vehicle 1, and the driver can drive more safely using the driving support information.

[0202] In addition, the processing section 3 of the information processing device 10 a of the present technology can reduce the amount of calculation of the image recognition process, thereby suppressing processing delay, and enables accurate driving support information to be presented at precise timing.

[0203] ((Storage Department))

[0204] The storage unit 7 stores various programs and data required for processing in the information processing device 10a. For example, the storage unit 7 stores a program for executing a series of processes according to the image recognition processing performed in the processing unit 3 of the present technology. For example, the storage unit 7 stores various parameters used in the processing according to the image recognition processing and logs related to vehicle driving, etc. For example, the storage unit 7 stores a program for executing a series of processes performed by the situation analysis unit 8 and the presentation control unit 4.

[0205] The storage section 7 includes, for example, a magnetic storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device.

[0206] [Feature extraction]

[0207] Next, feature extraction performed by the feature extraction section 31 will be described.

[0208] Figure 6 is a schematic diagram showing a configuration of an image recognition process performed by the processing section 3 of the information processing device 10 a , and is a schematic diagram showing a neural network for feature extraction.

[0209] like Figure 6 As shown in FIG. 1 , a plurality of features f1 to f5 common to the four tasks are extracted from the input image 9 acquired by the sensor unit 2 by a feature extractor 37 constituting the feature extraction unit 31. As common features of the four tasks, the feature extractor 37 extracts color features and edge features of the input image by a neural network.

[0210] Through the feature extractor 37, basic feature maps b0, b1, b2 and b3 with different resolutions are obtained from the input image 9 using processing of a plurality of combined layers of mutually different convolution operations and activation functions.

[0211] The basic feature maps b0 and b1 of the layer closer to the input image have relatively high resolution and have detailed structural information of the image. On the other hand, the basic feature maps b2 and b3 of the layer farther from the input image have relatively low resolution and have rough structural information of the image.

[0212] In addition, features f1 to f5 are extracted from two adjacent basic feature maps through a convolution operation. Features f1 to f5 are final feature maps and are referred to as "features" in this article to distinguish them from the basic feature maps.

[0213] For example, using basic feature maps b0 and b1, feature f1 is extracted through convolution operation. Using basic feature map b2 and feature f1, feature f2 is extracted through convolution operation. Using basic feature map b3 and feature f2, feature f3 is extracted through convolution operation. Using feature f3, feature f4 is extracted through convolution operation. Using feature f4, feature f5 is extracted through convolution operation.

[0214] Features f1 and f2 have relatively high resolution and more detailed color and edge information, while features f3 to f5 have relatively low resolution but broader edge information. The resolution of features decreases gradually from f1, f2, f3, f4 to f5.

[0215] The feature extractor 37 learns the sum of the loss functions of the four tasks of semantic segmentation, object detection (instance segmentation), motion detection (optical flow) and distance (depth) detection, and minimizes the loss functions of all tasks to extract common features f1 to f5 of the four tasks.

[0216] The information processing device 10 a can perform four task processes of semantic segmentation, object detection, motion detection, and distance detection using the extracted features f1 to f5 .

[0217] like Figure 6 As shown in , semantic segmentation produces a semantic segmentation result 11 as an image recognition process result. The semantic segmentation result 11 is an image in which a label or classification is associated with all pixels in the image.

[0218] Object detection (instance segmentation) produces an object detection result 12 as an image recognition processing result.

[0219] Motion detection (optical flow) generates an optical flow result 13 as an image recognition processing result.

[0220] The distance detection generates a distance detection result 14 as an image recognition processing result.

[0221] [Semantic Segmentation (Classification)]

[0222] Figure 7 : is a schematic diagram showing the neural network 101 for semantic segmentation. The first task estimation unit 32 performs semantic segmentation using the neural network 101 for semantic segmentation (first task).

[0223] The first task estimation unit 32 performs semantic segmentation and divides the input image 9 into regions such as road surface, sidewalk, sky, pedestrians, vehicles, cyclists, buildings, curbs, plants, guardrails, utility poles, signs, traffic lights, animals, trash cans, pillars, etc. on a pixel-by-pixel basis.

[0224] In semantic segmentation, N categories from 1 to N are predefined, the probability of each category 1 to N is estimated for each pixel, and the category with the highest probability is the estimated category of the pixel. Each category is assigned a category ID. In more detail, the estimation is performed as follows.

[0225] like Figure 7 As shown in , each of the features f1 to f5 of the input image 9 output from the feature extractor 37 is output to the corresponding decoder. In other words, feature f1 is output to the first decoder 111. Feature f2 is output to the second decoder 112. Feature f3 is output to the third decoder 113. Feature f4 is output to the fourth decoder 114. Feature f5 is output to the fifth decoder 115.

[0226] Next, a convolutional neural network of each decoder is used to estimate a feature map for each pixel for class estimation.

[0227] Next, the five feature maps for class estimation with different resolutions estimated by each decoder are aggregated by the feature aggregation unit 116. The aggregation is performed by summing pixels or by cascading the five feature maps for class estimation in the channel direction.

[0228] Next, the class predictor unit 117 calculates the probability of each class 1 to N for each pixel by a convolution operation so that the number of channels in the feature map aggregated by the feature aggregation unit 116 is N. The class ID corresponding to the highest probability among the N prediction values ​​is the semantic segmentation result (estimation result).

[0229] Therefore, the class IDs of all pixels in the input image 9 are estimated, and the semantic segmentation result 11 is obtained.

[0230] [Object Detection (Instance Segmentation)]

[0231] Figure 8 : is a schematic diagram showing the neural network 102 for object detection (instance segmentation). The second task a estimation unit 361 uses the neural network 102 for object detection to perform instance segmentation.

[0232] In instance segmentation, a mask of each object in the input image 9 is detected, and the type (class) of the region is also estimated. In instance segmentation, a mask of each object can be detected even if multiple objects of the same class are adjacent to each other.

[0233] like Figure 8 As shown in , each of the features f1 to f5 of the input image 9 output from the feature extractor 37 is output to the corresponding decoder. That is, feature f1 is output to the first object decoder 121. Feature f2 is output to the second object decoder 122. Feature f3 is output to the third object decoder 123. Feature f4 is output to the fourth object decoder 124. Feature f5 is output to the fifth object decoder 125.

[0234] Next, using the convolutional neural network of each decoder, the bounding box and bounding box class of each object are estimated.

[0235] Next, according to the position of the bounding box estimated by each decoder, features in the bounding box area are cut out from the features corresponding to the bounding box (any one of f1 to f5), and the cut out features are output to the mask estimation (MaskPredictor) unit 127.

[0236] Next, the mask estimation unit 127 estimates the category of each object region based on the features of the bounding box region.

[0237] As described above, the object detection result 12 subjected to the image recognition process is obtained from the input image 9 .

[0238] [Motion Detection (Optical Flow)]

[0239] Fig. 9 : is a schematic diagram showing the neural network 103 for motion detection (optical flow). The second task b estimation unit 362 uses the neural network 103 for motion detection to perform optical flow estimation.

[0240] like Fig. 9 As shown in , in the optical flow, the movement amount of each pixel between two input images 9a and 9b is calculated. The two input images 9a and 9b are images acquired at the same sensor section 2, and are, for example, an image of a current frame and an image one frame ago.

[0241] like Fig. 9 As shown in FIG. 1 , features f1 to f5 are extracted by the feature extractor 37 from the two input images 9a and 9b respectively.

[0242] Next, the feature f5 extracted from the input image 9a is matched with the feature f5 extracted from the input image 9b by the feature matching unit 136. In the feature matching unit 136, the features extracted from different images are matched with each other. The matching result is input to the fifth optical flow decoder 135. The fifth optical flow decoder 135 calculates an optical flow having the same resolution as the feature f5.

[0243] Next, the feature f4 extracted from the input image 9a is matched with the feature f4 extracted from the input image 9b by the feature matching unit 136. The matching result is output to the fourth optical flow decoder 134. In addition, after being calculated by the fifth optical flow decoder 135, the optical flow upsampled and amplified by the upsampling unit 137 is output to the fourth optical flow decoder 134. Then, the optical flow having the same resolution as the feature f4 is calculated by the fourth optical flow decoder 134. Here, when the optical flow is calculated by the fourth optical flow decoder 134, the amplified optical flow is also used after being calculated by the fifth optical flow decoder 135, so that the optical flow can be calculated more accurately.

[0244] Next, the feature f3 extracted from the input image 9a is matched with the feature f3 extracted from the input image 9b by the feature matching unit 136. The matching result is output to the third optical flow decoder 133. In addition, after being calculated by the fourth optical flow decoder 134, the optical flow is upsampled and amplified by the upsampling unit 137, and output to the third optical flow decoder 133. Then, the third optical flow decoder 133 calculates an optical flow having the same resolution as the feature f3. In the same manner as described above, when the optical flow is calculated by the third optical flow decoder 133, the amplified optical flow is also used after being calculated by the fourth optical flow decoder 134, so that the optical flow can be calculated more accurately.

[0245] Next, the feature f2 extracted from the input image 9a is matched with the feature f2 extracted from the input image 9b by the feature matching unit 136. The matching result is output to the second optical flow decoder 132. In addition, after being calculated by the third optical flow decoder 133, the optical flow is upsampled and amplified by the upsampling unit 137, and output to the second optical flow decoder 132. Then, the optical flow having the same resolution as the feature f2 is calculated by the second optical flow decoder 132. In the same manner as described above, when the optical flow is calculated by the second optical flow decoder 132, the amplified optical flow is also used after being calculated by the third optical flow decoder 133, so that the optical flow can be calculated more accurately.

[0246] Next, the feature f1 extracted from the input image 9a is matched with the feature f1 extracted from the input image 9b by the feature matching unit 136. The matching result is output to the first optical flow decoder 131. In addition, after being calculated by the second optical flow decoder 132, the optical flow is upsampled and amplified by the upsampling unit 137, and output to the first optical flow decoder 131. Then, an optical flow having the same resolution as the feature f1 is calculated by the first optical flow decoder 131. In the same manner as described above, when the optical flow is calculated by the first optical flow decoder 131, the amplified optical flow is also used after being calculated by the second optical flow decoder 132, so that the optical flow can be calculated more accurately. In this way, the optical flow is calculated in combination with the output of the optical flow decoder calculated in the previous stage, thereby obtaining a highly accurate optical flow result (motion detection result).

[0247] As described above, the optical flow result 13 is obtained from the input images 9a and 9b.

[0248] [Distance detection]

[0249] Fig.10: is a schematic diagram showing the neural network 104 for distance detection. The second task c estimation unit 363 performs distance estimation using the neural network 104 for distance detection (second task). In distance estimation, depth information (distance information between an object and a vehicle) is estimated for each pixel from the input image 9.

[0250] like Fig.10 As shown in , in the neural network 104 for distance detection, after aggregating a plurality of features f1 to f5 each having a different resolution extracted from the input image 9 and a feature extracted from the LiDAR point cloud 15 acquired by the distance measuring section 21, a depth decoder as a decoder for distance estimation calculates the distance of each pixel (the distance between the object and the vehicle (more specifically, the image capturing section)). Hereinafter, this will be described in detail.

[0251] like Fig.10 As shown in , features f1 to f5 are extracted by a feature extractor 37 from an input image 9 acquired by an image capture section 20 of a sensor section 2. In addition, RestNet (He, Kaiming et al. "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016) 16 is used for a LiDAR point cloud 15 acquired by a distance measurement section 21 of the same sensor section 2 to extract a LiDAR feature map (hereinafter referred to as LiDAR feature).

[0252] Next, the feature f5 extracted from the input image 9 and the LiDAR feature extracted from the LiDAR point cloud 15 are aggregated by the feature aggregation unit 146. Aggregation is performed by summing each pixel. Alternatively, the feature of each pixel can be cascaded in the channel direction. The aggregation result (aggregated feature) is output to the fifth depth decoder 145. The fifth depth decoder 145 calculates a distance detection result with the same resolution as the feature f5.

[0253] Next, the feature f4 extracted from the input image 9 and the LiDAR feature extracted from the LiDAR point cloud 15 are aggregated by the feature aggregation unit 146. The aggregation result is output to the fourth depth decoder 144. In addition, after being calculated by the fifth depth decoder 145, the distance detection result is upsampled and enlarged by the upsampling unit 147, and output to the fourth depth decoder 144. Then, the fourth depth decoder 144 calculates the distance detection result with the same resolution as the feature f4. Here, when the distance is calculated by the fourth depth decoder 144, the enlarged distance detection result is also used after being calculated by the fifth optical flow decoder 145, so that the distance detection result can be calculated more accurately.

[0254] Next, the feature f3 extracted from the input image 9 and the LiDAR feature extracted from the LiDAR point cloud 15 are aggregated by the feature aggregation unit 146. The aggregation result is output to the third depth decoder 143. In addition, after being calculated by the fourth depth decoder 144, the distance detection result upsampled and enlarged by the upsampling unit 147 is output to the third depth decoder 143. Then, the third depth decoder 143 calculates the distance detection result having the same resolution as the feature f3. Here, when the distance is calculated by the third depth decoder 143, the enlarged distance detection result is also used after being calculated by the fourth optical flow decoder 144, so that the distance detection result can be calculated more accurately.

[0255] Next, the feature f2 extracted from the input image 9 and the LiDAR feature extracted from the LiDAR point cloud 15 are aggregated by the feature aggregation unit 146. The aggregation result is output to the second depth decoder 142. In addition, after being calculated by the third depth decoder 143, the distance detection result upsampled and enlarged by the upsampling unit 147 is output to the second depth decoder 142. Then, the second depth decoder 142 calculates the distance detection result with the same resolution as the feature f2. Here, when the distance is calculated by the second depth decoder 142, the enlarged distance detection result is also used after being calculated by the third optical flow decoder 143, so that the distance detection result can be calculated more accurately.

[0256] Next, the feature f1 extracted from the input image 9 and the LiDAR feature extracted from the LiDAR point cloud 15 are aggregated by the feature aggregation unit 146. The aggregation result is output to the first depth decoder 141. In addition, after being calculated by the second depth decoder 142, the distance detection result is upsampled and amplified by the upsampling unit 147, and output to the first depth decoder 141. Then, the first depth decoder 141 calculates the distance detection result 14 with the same resolution as the feature f1. Here, when the distance is calculated by the first depth decoder 141, the amplified distance detection result is also used after being calculated by the second optical flow decoder 142, so that the distance detection result can be calculated more accurately. In this way, the distance detection result is calculated in combination with the output of the depth decoder calculated in the previous stage, thereby obtaining a highly accurate distance detection result.

[0257] As described above, the distance detection result 14 is obtained from the image 9 .

[0258] As described above, the four tasks (semantic segmentation, object detection, motion detection, and distance detection) can be performed in the processing section 3 of the information processing device 10 a using the common feature extractor 37 .

[0259] [Information processing method (image recognition processing method)]

[0260] Reference Fig.11 The information processing method (image recognition processing method) in the information processing device 10a is described. Fig.11 The information processing shown in .

[0261] The image 9 captured by the image capturing section 20 of the sensor section 2 is acquired by the image acquiring section 30 ( S1 ).

[0262] Next, the feature extraction unit 31 extracts a plurality of features ( f1 to f5 ) of the image 9 ( S2 ).

[0263] Next, the first task estimating unit 32 performs the first task (semantic segmentation) using the plurality of features extracted in S2 ( S3 ).

[0264] Next, the second task decision unit 33 decides whether to execute the second task based on the result of the first task (S4). In the present embodiment, the second task decision unit 33 decides whether to execute object detection, motion detection, and distance detection, respectively.

[0265] Next, the parameter generation unit 34 generates parameters for determining the second task to be executed using the semantic segmentation result ( S5 ).

[0266] Next, the second task neural network configuration unit 35 configures the neural network of the second task using the parameters generated in S5 ( S6 ).

[0267] Next, the second task (in the present embodiment, one or more tasks selected from object detection, motion detection, and distance detection) is performed by the second task estimation section 36 using the neural network configured in S6 (S7).

[0268] [Decision of the Second Mission]

[0269] Fig.12 yes Fig.11 4 (S4) of the second task determination process, which is performed by the second task determination unit 33 of the processing unit 3. Fig.12 Describe how to decide the second task to be performed. The second task decision process is based on the first task processing result (semantic segmentation result).

[0270] like Fig.12 As shown in , when the second task decision process is started, a unique category ID list (hereinafter simply referred to as ID list) is acquired from the semantic segmentation result including category ID information of each pixel (S1).

[0271] Next, using the ID list, it is determined whether there is a category ID of an object of interest (S42). When it is determined that there is an object of interest, the process proceeds to S43. When it is determined that there is no object of interest, it is decided to perform only distance detection (S44).

[0272] Here, as an example, the class IDs pre-assigned to each of a utility pole, a traffic light, a traffic sign, a trash can, a pillar, a vehicle (four-wheeled vehicle), a person, an animal, a bicycle, a motorcycle (two-wheeled vehicle), a train, a bus, and a truck are 1 to 13. These objects are objects of interest (objects that hinder driving). Among them, a vehicle with a class ID of 6, a person with a class ID of 7, an animal with a class ID of 8, a bicycle with a class ID of 9, a motorcycle with a class ID of 10, a train with a class ID of 11, a bus with a class ID of 12, and a truck with a class ID of 13 are movable objects of interest. On the other hand, a utility pole with a class ID of 1, a traffic light with a class ID of 2, a traffic sign with a class ID of 3, a trash can with a class ID of 4, and a pillar with a class ID of 5 are immovable objects of interest.

[0273] For example, in S42, if the category IDs 1 to 13 exist in the ID list, the process proceeds to S43. On the other hand, if the category IDs 1 to 13 do not exist in the ID list, the process proceeds to S44 and determines to execute only the distance detection task.

[0274] In S43, it is determined whether there is a category ID of a movable object of interest in the ID list.

[0275] When it is determined that there is a class ID of a movable object of interest, it is decided to execute three tasks of object detection, motion detection, and distance detection (S46).

[0276] On the other hand, when it is determined that there is no class ID of the movable object of interest, it is decided to execute both the object detection and the distance detection tasks ( S45 ).

[0277] For example, in the above example, if there are category IDs 5 to 13 in the ID list, the process proceeds to S46 and decides to execute three tasks of object detection, motion detection, and distance detection.

[0278] On the other hand, if the category IDs 5 to 13 do not exist in the ID list, the process proceeds to S45 and determines to execute both the object detection and the distance detection tasks.

[0279] Therefore, in this embodiment, based on the processing result (semantic segmentation result) of the first task, it is determined whether to execute the second tasks of object detection, motion detection, and distance detection respectively.

[0280] Next, refer to Fig.13 A specific example of the second task determination process is described.

[0281] Fig.13 : is a diagram for describing a specific example of the second task determination process. Fig.13 1 is a diagram for describing how to determine a second task to be executed corresponding to a result of a first task processing of an image 9 acquired by an image capturing unit 20 of each of a plurality of sensor units 2 mounted on a vehicle. In the second task determination processing, the second task to be executed is determined using scene features obtained from semantic segmentation.

[0282] exist Fig.13 In the illustrated example, the image 9SL acquired by the image capturing section 20 of the left-side sensor section 2SL and the image 9FL acquired by the image capturing section 20 of the left front sensor section 2FL are both images in which the object of interest does not exist.

[0283] Therefore, since the ID list obtained from the semantic segmentation results 11SL and 11FL (the first task estimation results of the images 9SL and 9FL, respectively) does not contain the category ID of the object of interest, it is decided to perform only distance detection.

[0284] Therefore, for example, if there is no object of interest, only distance detection can be performed to estimate the distance between the road surface and the vehicle (the vehicle on which the image capture unit is installed). Using this estimated distance information of an object that is not an obstacle, driving support information such as driving operation sequence, speed change, vehicle overtaking advice, lane change advice, etc. can be presented to the driver.

[0285] exist Fig.13 In the illustrated example, the image 9F acquired by the image capturing section 20 of the front sensor section 2F and the image 9FR acquired by the image capturing section 20 of the right front sensor section 2FR are both images in which a vehicle (ie, a movable object of interest) exists.

[0286] Therefore, since the ID list obtained from the semantic segmentation results 11F and 11FR (the first task estimation results of images 9F and 9FR, respectively) contains the category ID of the movable object of interest, it is decided to perform the tasks of object detection, motion detection, and distance detection.

[0287] Therefore, if there is a movable object of interest, object detection, motion detection, and distance detection can be performed to estimate the motion and distance of each object (the distance between the object and the vehicle). Using this estimated motion and distance information of each object (i.e., image recognition processing result), driving support information such as obstacle warning, driving operation sequence, speed change, vehicle overtaking suggestion, lane change suggestion, etc. can be presented to the driver.

[0288] exist Fig.13 In the illustrated example, the image 9SR acquired by the image capturing section 20 of the right sensor section 2SR is an image in which a fire hydrant (ie, an immovable object of interest) exists.

[0289] Therefore, since the ID list obtained from the semantic segmentation result 11SR which is the first task estimation result of the image 9SR contains the class ID of the immovable object of interest, it is decided to perform object detection and distance detection.

[0290] Therefore, if there are immovable stationary objects of interest such as fire hydrants, walls, curbs, pillars, etc., object detection and distance detection can be performed to estimate the distance between these objects and the vehicle. Using this estimated distance information (i.e., image recognition processing results), driving support information such as obstacle warnings, driving operation commands to avoid collisions with obstacles, etc. can be presented to the driver.

[0291] In an omnidirectional sensing system in which multiple image capture units are installed on a vehicle to obtain vehicle surrounding environment information in a wide area, if, for example, four tasks are always performed for one image, the amount of calculation becomes huge, which easily leads to processing delays, and real-time image recognition processing and driving assistance based on image recognition results become difficult.

[0292] In contrast, in the present embodiment, by applying the image processing method (information processing method) of the present technology to each of the images acquired by each image capturing section, it is possible to reduce the amount of calculation and processing delay in the image recognition processing, such as in the case where the image is captured by each image capturing section. Fig.13 As in the images acquired by the front sensor unit and the right front sensor unit shown in , all three second tasks are performed only when there is a movable object as an obstacle, and in other cases, only two or one second tasks are performed.

[0293] [Generate second task parameters]

[0294] Fig.14 yes Fig.11 5 (S5) of the embodiment of the present invention, which is a flowchart of the second task parameter generation process in step 5 (S5), and the second task parameter generation process is performed by the parameter generation unit 34 of the processing unit 3.

[0295] Fig.15 is a diagram describing a specific example of parameter generation.

[0296] In the following, combined Fig.14 Reference Fig.15 , how to generate parameters for use in performing the second task will be described. The parameter generation process is performed using the first task processing result (semantic segmentation result).

[0297] like Fig.14 As shown in , when the parameter generation process starts, it is determined whether there is an object of interest in the image according to the semantic segmentation result (S51). In detail, it is determined whether there is an object of interest by determining whether there is a category ID of the object of interest using an ID list obtained from the semantic segmentation result.

[0298] When it is determined at S51 that there is no object of interest, the process proceeds to S53.

[0299] When it is determined at S51 that the object of interest exists, the process proceeds to S52 .

[0300] In S53, parameters are generated so that all the features f1 to f5 are used to perform recognition processing on all regions of the input image 9. The generated parameters are stored in the storage unit 7 (S58).

[0301] As described in the second task decision process above, if it is determined that there is no object of interest, it is decided to perform only distance detection. In other words, if there is no object of interest, it is decided to perform only distance detection, no optimization is performed on the neural network for distance detection, and distance detection is performed for the entire area of ​​the input image 9 using all features f1 to f5. In other words, distance detection is performed using a standard neural network.

[0302] In S52, the classification of the object of interest is obtained. In detail, the classification can be obtained by checking the category ID of the object of interest using the ID list. There may be multiple objects in the classification acquisition result. The classification acquisition result is a category ID list of the object of interest.

[0303] Next, the process proceeds to S54 and S56.

[0304] In S54, an image region in which an object of interest exists is obtained. The image region to be obtained includes each region of a plurality of objects. A rectangular frame (also referred to as a bounding box) is set around the object of interest. The region of the frame when the size of the frame is the smallest (also referred to as the minimum region) is referred to as the final image region. The final image region is referred to as the "image to be processed". The image to be processed is the region to be processed on which the second task processing is performed.

[0305] Next, the coordinate parameters of the image to be processed are generated (S55), and the generated parameters are stored in the storage unit 7 (S58).

[0306] The coordinates of the image to be processed are expressed in the form of parameters (x1, y1, w, h). The parameters (x1, y1, w, h) are expressed using the coordinates (x1, y1) of the upper left corner of a rectangular box (bounding box) and (w, h) indicating the number of horizontal and vertical pixels of the rectangle.

[0307] In S56 , the number of pixels of each object of interest acquired in S52 is calculated, and a list of minimum pixel numbers is generated to include the minimum pixel number of each classification of each object of interest.

[0308] Next, using the list of the minimum number of pixels of the object of interest, parameters representing the features to be used for the second task are generated (S57) The generated parameters are stored in the storage unit 7 (S58).

[0309] Parameter generation is based on Fig.15 The table shown in is used to generate the parameters of the feature.

[0310] exist Fig.15In the example shown, if the object classification is a four-wheel vehicle, parameters are generated so that when the number of pixels is T0 or less, the features to be used are f1, f2, and f3, when the number of pixels is greater than T0 and less than T1, the features to be used are f2, g3, and f4, and when the number of pixels is greater than T1 and less than T2, the features to be used are f3, f4, and f5.

[0311] If the object classification is a pedestrian, parameters are generated so that when the number of pixels is T3 or less, the features to be used are f1, f2, and f3, when the number of pixels is greater than T3 and less than T4, the features to be used are f2, g3, and f4, and when the number of pixels is greater than T4 and less than T5, the features to be used are f4 and f5.

[0312] If the classification of the object of interest is a cyclist, parameters are generated so that when the number of pixels is T6 or less, the features to be used are f1, f2, and f3, when the number of pixels is greater than T6 and less than T7, the features to be used are f2, g3, and f4, and when the number of pixels is greater than T7 and less than T8, the features to be used are f4 and f5.

[0313] Next, refer to Fig.16 Describes a specific example of parameter generation.

[0314] Fig.16 (A) is a diagram showing an image 9 acquired by an image capturing unit of each of the plurality of sensor units 2 mounted on the vehicle 1, a semantic segmentation result for the image 9, an object detection result, a motion detection result, and a distance detection result. Figure 2 In the present invention, the object detection result, the motion detection result and the distance detection result are the recognition processing results using a neural network (which is reconfigured and optimized with parameters generated based on the semantic segmentation result) or a standard neural network.

[0315] Fig.16 (B) shows that based on Fig.16 FIG. 1 is an example of using the semantic segmentation result shown in (A) to decide the second task to perform and generate parameters.

[0316] exist Fig.16 In the example shown in (A) of FIG. 8 , the image 9SL acquired by the image capturing section 20 of the left sensor section 2SL is an image in which a movable object of interest exists.

[0317] Therefore, if Fig.16As shown in (B), since the ID list obtained from the semantic segmentation result of the image 9SL contains the category ID of the movable object of interest, it is decided to perform all the second tasks of object detection, motion detection, and distance detection. As the parameters of the image to be processed, the parameters [xa, ya, wa, ha] are generated to indicate the coordinates of the area A surrounded by the rectangular frame surrounding the object of interest in the figure, and the parameters are generated so that the features f1, f2, and f3 are used. As described above, the parameter generation of the features is based on Fig.15 The following also applies.

[0318] exist Fig.16 In the example shown in (A) of FIG. 8 , the image 9FL acquired by the image capturing section 20 of the left front sensor section 2FL is an image in which the object of interest does not exist.

[0319] Therefore, if Fig.16 As shown in (B), the ID list obtained from the semantic segmentation result of image 9FL does not include the category ID of the object of interest, so it is decided to perform only distance detection. In addition, parameters are generated so that the entire image area is the target image to be processed and all features f1 to f5 are used. Fig.16 In (B), the coordinates of the image to be processed of the entire image area are represented as parameters [0, 0, w, h].

[0320] exist Fig.16 In the example shown in (A) of FIG. 9 , the image 9F acquired by the image capturing section 20 of the front sensor section 2F is an image in which a movable object of interest exists.

[0321] Therefore, if Fig.16 As shown in (B), the ID list obtained from the semantic segmentation result of image 9F contains the category ID of the movable object of interest, so it is decided to perform all the second tasks of object detection, motion detection, and distance detection. As parameters of the image to be processed, parameters [xb, yb, wb, hb] indicating the coordinates of area B and parameters [xc, yc, wc, hc] indicating the coordinates of area C are generated, and the area B is surrounded by a rectangular frame around the object of interest in the figure. Parameters are generated so that features f1, f2, and f3 are used for area B, and parameters are generated so that features f3, f4, and f5 are used for area C.

[0322] exist Fig.16 In the example shown in (A) of FIG. 8 , the image 9FR acquired by the image capturing section 20 of the right front sensor section 2FR is an image in which a movable object of interest exists.

[0323] Therefore, if Fig.16As shown in (B), the ID list obtained from the semantic segmentation result of the image 9FR contains the category ID of the movable object of interest, so it is decided to perform all the second tasks of object detection, motion detection, and distance detection. As parameters of the image to be processed, parameters [xd, yd, wd, hd] are generated to indicate the coordinates of the area D surrounded by the rectangular box surrounding the object of interest in the figure, and parameters are generated so that features f1, f2, and f3 are used.

[0324] exist Fig.16 In the example shown in (A) of FIG. 8 , the image 9SR acquired by the image capturing section 20 of the right sensor section 2SR is an image in which a movable object of interest exists.

[0325] Therefore, if Fig.16 As shown in (B), the ID list obtained from the semantic segmentation result of the image 9SR contains the category ID of the movable object of interest, so it is decided to perform all the second tasks of object detection, motion detection, and distance detection. As parameters of the image to be processed, parameters [xe, ye, we, he] indicating the coordinates of the region E and parameters [xf, yf, wf, hf] indicating the coordinates of the region F are generated, and the region E is surrounded by a rectangular frame around the object of interest in the figure. Parameters are generated so that features f1, f2, and f3 are used for the region E, and parameters are generated so that features f3, f4, and f5 are used for the region F.

[0326] [Neural network configuration for the second task]

[0327] Fig.17 A diagram showing an example of a neural network configuration example executed in the second-task neural network configuration section 35 of the processing section 3 is shown.

[0328] Fig.17 Shown in the Fig.16 Each input image shown in (A) is used in the recognition task Fig.16 An example of a neural network configuration for deciding the second task and parameters to be performed is shown in (B).

[0329] By reconfiguring the neural network of the second task using the generated parameters, the estimation process of the second task decided to be performed in S33 can be performed with a smaller amount of calculation according to the size of the object of interest in the input image, and the estimation process can be more optimized. In the neural network configuration, the configuration is changed to select a decoder for each second task using the generated parameters.

[0330] In the following, we will use Fig.17 A specific example is described.

[0331] As described above, for the image 9SL acquired by the left sensor section 2SL, the second task decision processing decides that all three recognition tasks (second tasks) are to be performed: object detection, motion detection processing, and distance detection.

[0332] like Fig.17 As shown in , in performing the three second tasks, the image to be processed is only the area A [xa, ya, wa, ha]. In addition, since there are only small objects located far away in area A, it is configured to generate only high-resolution features f1, f2 and f3, and further, based on this, the neural network of each second task is reconfigured so that the decoder used in the processing of the second task is configured only by the decoders corresponding to f1, f2 and f3. In more detail, in object detection, it is reconfigured so that only the first, second and third object decoders 121 to 123 are used. In motion detection, it is reconfigured so that only the first, second and third optical flow decoders 131 to 133 are used. In distance detection, it is reconfigured so that only the first, second and third depth decoders 141 to 143 are used.

[0333] As described above, the second task determination process determines to perform only distance detection for the image 9FL acquired by the left front sensor unit 2FL. Fig.17 As shown in , in performing distance detection, the image to be processed is the entire area, and the neural network for distance detection is configured so that features f1 to f5 are generated, and further, the neural network for distance detection is configured so that the decoder used in distance detection is configured by the decoders corresponding to f1 to f5. In more detail, it is configured so that the first depth decoder 141 to the fifth depth decoder 145 are used. In other words, optimization will not be performed, but a standard neural network will be used to perform distance detection.

[0334] As described above, the second task decision processing decides to execute all three recognition tasks (second tasks) among object detection, motion detection processing, and distance detection for the image 9F acquired by the front sensor section 2F.

[0335] like Fig.17 As shown in , in executing the three second tasks, the images to be processed are only the region B [xb, yb, wb, hb] and the region C [xc, yc, wc, hc].

[0336] In region B, since there are only small objects located far away in region B, it is configured so that only high-resolution features f1, f2 and f3 are generated, and further, based on this, the neural network of each second task is reconfigured so that the decoder used in the second task is configured only by the decoders corresponding to f1, f2 and f3.

[0337] On the other hand, in region C, since there is a large object located near region C, it is configured so that only features f2, f3, and f4 having low resolution but wide range of edge information are generated, and further, based on this, the neural network of each second task is reconfigured so that the decoder used in the second task is configured only by decoders corresponding to f2, f3, and f4.

[0338] In detail, object detection is reconfigured to use only the first, second and third object decoders 121 to 123 in region B, and only the second, third and fourth object decoders 122 to 124 in region C. Motion detection is reconfigured to use only the first, second and third optical flow decoders 131 to 133 in region B, and only the second, third and fourth optical flow decoders 132 to 134 in region C. Distance detection is reconfigured to use only the first, second and third depth decoders 141 to 143 in region B, and only the second, third and fourth depth decoders 142 to 144 in region C.

[0339] As described above, for the image 9SL acquired by the left sensor section 2SL, it is determined that all three recognition tasks (second tasks): object detection, motion detection processing, and distance detection are to be performed by the second task decision processing.

[0340] like Fig.17 As shown in , in performing the three second tasks, the image to be processed is only the region D [xd, yd, wd, hd]. In addition, since there are only small objects located far away in region D, it is configured so that only high-resolution features f1, f2 and f3 are generated, and further, accordingly, the neural network of each second task is reconfigured so that the decoder used in the second task is configured only by the decoders corresponding to f1, f2 and f3. In more detail, in object detection, it is reconfigured so that only the first, second and third object decoders 121 to 123 are used. In motion detection, it is reconfigured so that only the first, second and third optical flow decoders 131 to 133 are used. In distance detection, it is reconfigured so that only the first, second and third depth decoders 141 to 143 are used.

[0341] For the image 9SR acquired by the right sensor section 2SR, it is determined that all three recognition tasks (second tasks): object detection, motion detection processing, and distance detection are to be performed by the second task decision processing.

[0342] In performing these three second tasks, the images to be processed are only the region E[xe,ye,we,he] and the region F[xf,yf,wf,hf].

[0343] In region E, since there are only small objects located far away in region E, it is configured so that only high-resolution features f1, f2 and f3 are generated, and further, based on this, the neural network of each second task is reconfigured so that the decoder used in the second task is configured only by the decoders corresponding to f1, f2 and f3.

[0344] On the other hand, in region F, since there is a large object located near region F, it is configured so that only features f2, f3 and f4 having low resolution but wide range of edge information are generated, and further, based on this, the neural network of each second task is reconfigured so that the decoder used in the second task is configured only by decoders corresponding to f2, f3 and f4.

[0345] In detail, object detection is reconfigured to use only the first, second and third object decoders 121 to 123 in region E, and only the second, third and fourth object decoders 122 to 124 in region F. Motion detection is reconfigured to use only the first, second and third optical flow decoders 131 to 133 in region E, and only the second, third and fourth optical flow decoders 132 to 134 in region F. Distance detection is reconfigured to use only the first, second and third depth decoders 141 to 143 in region E, and only the second, third and fourth depth decoders 142 to 144 in region F.

[0346] As described above, the present technology can reduce the amount of calculation by extracting five features f1 to f5 that can be used in common in multiple tasks. In addition, since the second task to be performed can be decided and the parameters can be generated according to the processing result of the first task, the neural network of the second task can be optimized. For example, this makes it possible to reduce the amount of calculation on the decoder side compared to the processing using a standard neural network.

[0347] Reference Fig.18 , taking distance detection as an example, the amount of computation on the decoder side can be reduced.

[0348] This is for Fig.16 1 and 2. An example of distance detection performed by an image 9SL acquired by the left sensor section 2SL shown in FIG. 1 , in which a movable object of interest exists.

[0349] Fig.18 (A) is a diagram showing a standard neural network 104, which is similar to the above Fig.10 The neural network shown in is similar.

[0350] Fig.18(B) is a diagram showing a neural network 104a reconstructed using features f1, f2, and f3 generated as parameters and determined to perform only distance detection based on the semantic segmentation result.

[0351] like Fig.18 (A) and Fig.18 As shown in (B), the reconstructed neural network 104a for distance detection uses fewer decoders than the standard neural network. This optimizes the distance detection process (second task processing) and reduces the amount of computation on the decoder side.

[0352] In addition, in object detection and motion detection, the neural network can be reconstructed in the same way, and the amount of computation on the decoder side can be reduced by reducing the number of decoders used compared to a standard neural network.

[0353] If no object of interest exists, a standard neural network is used to perform the task.

[0354] As described above, in the present technology, only necessary recognition tasks (second tasks) are processed according to scene features based on semantic segmentation results. In addition, parameters are generated using the semantic segmentation results, and the parameters are used to optimize the neural network of the second task. This makes it possible to reduce the amount of computation for image recognition processing while achieving highly accurate image recognition processing, thereby reducing power consumption and processing delays. As a result, while reducing the amount of computation for the vehicle's surrounding environment sensing system, it becomes possible to appropriately and in real time present driving support information based on highly accurate image recognition processing results.

[0355] <Third Embodiment>

[0356] In the third embodiment, the technology described in the first embodiment is applied to image recognition processing of each image acquired by each of the multiple image capture units installed on the vehicle, and the image recognition processing results are used for autonomous driving. The main difference between the second and third embodiments is the destination of application of the image recognition processing results, and the other configurations are almost the same. In the following, the different points are mainly described.

[0357] [Information Processing System]

[0358] Fig.19 is a schematic diagram of an information processing system 200 according to the present embodiment. In the information processing system 200, an automatic driving process is performed using image recognition processing results of a plurality of image capturing units mounted on the vehicle 1. The information processing system of the present embodiment can be restated as an automatic driving system.

[0359] like Fig.19As shown in FIG. 1 , the information processing system 200 has a plurality of sensor sections 2 , an information processing device 10 b , a vehicle state detection section 5 , and a drive system 26 . All of these are mounted on the vehicle 1 .

[0360] (Sensor part)

[0361] Each sensor section 2 includes an image capturing section 20 and a distance measuring section 21 .

[0362] The image capturing unit 20 acquires an image. A monocular camera, a stereo camera, or the like can be used for the image capturing unit 20.

[0363] The distance measuring unit 21 is configured to be able to measure the distance between the vehicle 1 and an object around the vehicle 1. As the distance measuring unit 21, LiDAR, a stereo camera, a millimeter wave radar, etc. can be used, and the distance measuring unit 21 is configured to include one or more selected from these. In the present embodiment, an example of using LiDAR as the distance measuring unit 21 is given.

[0364] The image acquired by the image capturing section 20 of the sensor section 2 and the 3D point cloud information acquired by the LiDAR as the distance measuring section 21 are output to the information processing device 10 b .

[0365] (Vehicle status detection unit)

[0366] The vehicle state detection unit 5 detects the state of the vehicle. For example, the vehicle state detection unit 5 includes a gyro sensor, an acceleration sensor, and a sensor for detecting the operation amount of the accelerator pedal, the operation amount of the brake pedal, the steering angle, the engine speed, the motor speed, or the vehicle speed. The vehicle information detected by the vehicle state detection unit 5 (such as the speed and steering angle of the vehicle 1) is output to the presentation control unit 24 described below.

[0367] (Drive system)

[0368] The drive system 26 includes various devices related to the drive system of the vehicle (own vehicle) 1. For example, the drive system includes: a drive force generator such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle, a brake device for generating a braking force, ABS (anti-lock braking system), ESC (electronic stability control), and an electric power steering system.

[0369] As described below, the drive system 26 is controlled based on various control signals supplied from the drive system control portion 25 .

[0370] (Information Processing Device)

[0371] The information processing device 10b has a hardware configuration required for a computer, such as a CPU and memory (RAM, ROM). The CPU loads a program stored in a storage unit 7 (described below) into the RAM and executes the program to perform various processes including the image recognition processing according to the present technology.

[0372] In the information processing device 10b, semantic segmentation (first task) is performed for each input image from the image capturing section 20 of each of the plurality of sensor sections 2. The semantic segmentation result is used to decide a task (second task) to be used for image recognition processing and to generate parameters for the task.

[0373] The information processing device 10 b includes a processing unit 3 , an image acquisition unit 30 , a situation analysis unit 8 , a planning unit 24 , a drive system control unit 25 , and a storage unit 27 .

[0374] ((Image acquisition unit))

[0375] The image acquisition section 30 acquires the image acquired by the image capturing section 20 of each sensor section 2 and outputs the image to the processing section 3 .

[0376] ((Processing Department))

[0377] The processing unit 3 performs image recognition on the image (input image) acquired by the image acquisition unit 30. At this time, the processing unit 3 uses the result of executing the first task on the input image to determine whether to execute the second task and generate parameters of the second task as described in the first and second embodiments.

[0378] ((Situation Analysis Department))

[0379] The situation analysis unit 8 performs analysis processing of the vehicle's surrounding environment based on the recognition processing result of the first task (semantic segmentation result) and the recognition processing result of the second task (one or more processing results selected from object detection results, motion detection results and distance detection results).

[0380] Output the analysis results to the planning department 24

[0381] ((Planning Department))

[0382] The planning unit 24 plans the route and actions of the vehicle 1 so as to drive to the destination in a timely and safe manner according to the route. In the planning unit 24, the route and actions are planned so that when the vehicle is driven by automatic driving, the own vehicle avoids collision or reduces impact, follows the vehicle based on the distance between the vehicles, maintains the vehicle speed, etc.

[0383] The planning unit 24 includes a route planning unit 240 and an action planning unit 241 .

[0384] The route planning section 240 plans a route to the destination using map information and the state information of the vehicle 1 detected by the vehicle state detection section 5. The route planning section 240 also appropriately changes the route using the analysis result of the situation analysis section 8. The route planning section 240 outputs data indicating the planned route to the action planning section 241.

[0385] The action planning unit 241 plans the action of the vehicle 1 to safely travel the route planned by the route planning unit 240 within the planned time. For example, the action planning unit 241 plans starting, stopping, driving direction (e.g., forward, backward, left turn, right turn, change of direction, etc.), driving lane, driving speed, and overtaking. The action planning unit 241 supplies data indicating the planned action of the vehicle 1 to the drive system control unit 25. The action planning unit 241 also uses the analysis results of the situation analysis unit 8 to appropriately change the action plan.

[0386] For example, if only an immovable object of interest is identified in the input image, object detection and distance detection are performed in the input image recognition process. Then, based on the distance information between the vehicle and the object of interest estimated by the distance detection, the steering angle and braking can be automatically controlled to avoid a collision with the object of interest.

[0387] On the other hand, if a movable object of interest is identified in the input image, object detection, motion detection, and distance detection are performed in the input image recognition process. Then, based on the motion information and distance information of the object of interest estimated by each detection, the steering angle and braking can be automatically controlled to avoid collision with the object of interest.

[0388] Therefore, in the image recognition processing according to the present embodiment, if there is only an immovable object of interest, motion detection is not performed, thereby enabling the amount of calculation in the image recognition processing to be reduced without reducing the recognition accuracy. The image recognition result can then be used to implement the automatic steering and automatic braking functions.

[0389] The image recognition processing result in the processing unit 3 is used to generate the analysis result of the surrounding environment of the vehicle by the above-mentioned situation analysis unit 8. The processing unit 3 of the information processing device 10b in this embodiment can perform image recognition processing without reducing the accuracy of image recognition. Therefore, the analysis result of the surrounding environment of the vehicle performed using the highly accurate image recognition processing result is highly accurate information. The route plan and action plan formulated using this highly accurate information are more suitable for the situation of the vehicle 1, and also improve the safety of autonomous driving.

[0390] Furthermore, the processing section 3 of the information processing device 10 b of the present technology can reduce the amount of calculation in the image recognition process, thereby suppressing processing delays, and realize automatic driving based on a more accurate route plan and action plan at an accurate timing.

[0391] (Drive system control unit)

[0392] The drive system control unit 25 generates various control signals based on the data indicating the behavior of the vehicle 1 planned by the behavior planning unit 241 , and supplies the control signals to the drive system 26 .

[0393] ((Storage Department))

[0394] The storage unit 27 stores various programs and data required for processing in the information processing device 10b. For example, the storage unit 27 stores a program for executing a series of processes according to the image recognition processing performed in the processing unit 3 of the present technology. For example, the storage unit 27 stores various parameters used in the processing according to the image recognition processing and logs related to vehicle driving, etc. For example, the storage unit 27 stores a program for executing a series of processes performed by the situation analysis unit 8, the presentation control unit 24, and the drive system control unit 25.

[0395] The storage section 27 includes, for example, a magnetic storage device (such as a ROM, a RAM, and a HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device.

[0396] A series of information processing methods in the processing section 3 are the same as those in the second embodiment.

[0397] In this embodiment, as in the first and second embodiments, the amount of calculation can be reduced by extracting five features f1 to f5 that can be used in common in multiple tasks. In addition, since the second task to be performed can be determined based on the processing result of the first task and the parameters can be generated, the neural network of the second task can be optimized. For example, this makes it possible to reduce the amount of calculation on the decoder side compared to processing using a standard neural network.

[0398] This configuration enables highly accurate image recognition processing while reducing the amount of calculations based on image recognition processing. This enables reduced power consumption and processing delays, and enables accurate, real-time autonomous driving (autonomous driving) control based on image recognition processing results.

[0399] <Others>

[0400] The above is a description of the embodiment of the present invention. The present invention is not limited to the above embodiment, and various changes can be made within the scope not departing from the gist of the present invention.

[0401] In the above embodiment, an example of using LiDAR as a distance measuring unit is given, but for example, a stereo camera device may be used instead of LiDAR, and image recognition processing may be performed using a 3D point cloud obtained from a stereo image acquired with the stereo camera device. In this configuration, the semantic segmentation (first task) result can be used to reduce the number of operations for stereo disparity estimation, and the amount of calculation can be reduced.

[0402] Alternatively, a depth map may be predicted from a stereo image acquired by a stereo camera, and then each pixel may be projected to a LiDAR coordinate to obtain a pseudo LiDAR point cloud converted from the depth map image to a point cloud.

[0403] In the above embodiment, an example of installing an image capture unit for acquiring images used for image recognition processing of the present technology on a mobile object is given, but it is not limited to this. The mobile object on which the image capture unit is installed may be other vehicles, such as two-wheeled vehicles, cleaning robots, toy robots, drones, etc.

[0404] In the above embodiment, an example is given in which the processing unit 3 that performs the image recognition processing according to the present technology is installed on a vehicle (mobile object), but it is not limited to this, and may be, for example, a server existing on an external network. From the perspective of reducing processing delays, it is preferable to install the processing unit on a mobile object on which the image capture unit is installed.

[0405] The present technology may also have the following structures. (1)

[0407] An information processing device, comprising:

[0408] A processing unit, wherein the processing unit is capable of processing multiple tasks for the recognition target, the multiple tasks including a first task and a second task for shared feature extraction, wherein:

[0409] The processing unit determines whether to execute the processing of the second task using the recognition result of the recognition target from the processing of the first task. (2)

[0411] The information processing device according to (1), wherein:

[0412] The processing section generates a parameter of the second task using a recognition result of the recognition target from the processing of the first task. (3)

[0414] The information processing device according to (2), wherein:

[0415] The processing unit configures the neural network of the second task using the generated parameters. (4)

[0417] The information processing device according to (2) or (3), wherein:

[0418] The processing section extracts a plurality of features from the recognition target, and uses a recognition result of the recognition target from the processing of the first task by using the plurality of features to decide whether to execute the processing of the second task and generate the parameter. (5)

[0420] The information processing device according to (4), wherein:

[0421] The parameters include a region to be processed of the second task and one or more features selected from the plurality of features. (6)

[0423] The information processing device according to any one of (2) to (5), wherein:

[0424] The processing unit determines whether to execute the processing of the second task and generates the parameter using the scene feature obtained from the recognition result of the recognition object from the processing of the first task. (7)

[0426] The information processing device according to (6), wherein:

[0427] The recognition target is an image acquired by an image capturing unit mounted on a mobile object, the image capturing unit capturing the surrounding environment of the mobile object, and

[0428] The scene feature is a moving scene feature of the moving object, and the moving scene feature refers to whether there is an object of interest in the image and whether the object of interest is a movable object. (8)

[0430] The information processing device according to (7), wherein:

[0431] The object of interest is an object that hinders the movement of the moving object. (9)

[0433] The information processing device according to (7) or (8), wherein:

[0434] The first task is semantic segmentation,

[0435] The second task includes object detection, motion detection and distance detection, and

[0436] If there is no object of interest in the image, the processing unit only performs the distance detection.

[0437] If an object of interest exists in the image and the object of interest is not a movable object, the processing unit performs the object detection and the distance detection, and

[0438] If an object of interest exists in the image and the object of interest is a movable object, the processing section performs the object detection, the motion detection, and the distance detection. (10)

[0440] The information processing device according to (9), wherein:

[0441] The parameters include a region to be processed of the second task and one or more features selected from the plurality of features, and

[0442] If the object of interest does not exist in the image, the processing unit generates the parameters so that the entire image area is used as the area to be processed and all of the plurality of features are used, and

[0443] If an object of interest exists in the image, the processing unit generates the parameters so that a minimum area surrounding the object of interest is used as an area to be processed and one or more features selected from the plurality of features are used according to the number of pixels in the object of interest. (11)

[0445] The information processing device according to (9) or (10), wherein:

[0446] A distance measuring unit is installed on the mobile object, and

[0447] In the distance detection, the distance is estimated using an aggregation result of aggregating the feature extracted from the image and the distance feature obtained by the distance measurement section. (12)

[0449] The information processing device according to (11), wherein:

[0450] The distance measuring section includes one or more selected from light detection and ranging LiDAR, a stereo camera device, and a millimeter wave radar. (13)

[0452] The information processing device according to any one of (7) to (12), wherein:

[0453] A plurality of the image capturing units are mounted on the moving object, and

[0454] The processing section uses the image recognition result from the processing of the first task for each image acquired by each of the plurality of image capturing sections mounted on the mobile object to decide whether to execute the processing of the second task and generate the parameter. (14)

[0456] The information processing device according to any one of (7) to (13), wherein:

[0457] The image capturing unit is a stereo camera or a monocular camera. (15)

[0459] The information processing device according to any one of (7) to (14), wherein:

[0460] The processing unit performs the second task on the image using a neural network of the second task configured by using the generated parameters, and

[0461] The information processing apparatus further includes a presentation control section that controls a presentation section that provides assistance to an operator of the mobile object based on a recognition result of the second task. (16)

[0463] The information processing device according to (15), wherein:

[0464] installing one or more selected from a display unit, a light emitting unit, and a sound output unit as the presentation unit on the mobile object, and

[0465] The presentation control section controls at least one of display control of the display section, lighting control of the light emitting section, and sound output control of the sound output section. (17)

[0467] The information processing device according to any one of (7) to (16), wherein:

[0468] The mobile object is a mobile object capable of autonomous movement, and

[0469] The processing unit performs the second task on the image using a neural network of the second task configured by using the generated parameters, and

[0470] The information processing device further includes a planning unit that plans travel and action of the mobile object based on the recognition result of the second task. (18)

[0472] The information processing device according to any one of (1) to (17), wherein:

[0473] The recognition target is an image,

[0474] The first task is semantic segmentation, and

[0475] The second task includes one or more selected from object detection, motion detection, distance detection, normal estimation, pose estimation and trajectory estimation. (19)

[0477] An information processing method executed by an information processing device, comprising:

[0478] processing a first task for identifying an object, and

[0479] A recognition result of the recognition target from the processing of the first task is used to decide whether to execute a second task that shares feature extraction with the first task. (20)

[0481] A program causing an information processing device to execute:

[0482] a step of processing a first task for identifying a target, and

[0483] The step of using the recognition result of the recognition target from the processing of the first task to decide whether to perform a second task that shares feature extraction with the first task.

[0484] Reference numerals list

[0485] 1 Vehicle (moving object)

[0486] 3. Processing Unit

[0487] 4. Presentation Control Unit

[0488] 40 Display control unit

[0489] 41 Sound Control Unit

[0490] 42 Lighting control unit

[0491] 6 Presentation Department

[0492] 60 Display unit

[0493] 61 Sound output unit

[0494] 62 light emitting part

[0495] 9 images, input image (recognition target)

[0496] 10, 10a, 10b information processing device

[0497] 20 Camera device (image capturing unit)

[0498] 21LiDAR (distance measurement unit)

[0499] 24 Planning Department

[0500] 102 Neural Network for Object Detection (Neural Network for the Second Task)

[0501] 103 Neural Network for Motion Detection (Neural Network for the Second Task)

[0502] 104, 104a Neural network for distance detection (neural network for the second task)

Claims

1. An information processing device, comprising: A processing unit, wherein the processing unit is capable of processing multiple tasks for the recognition target, the multiple tasks including a first task and a second task for shared feature extraction, wherein: The processing unit determines whether to execute the processing of the second task using the recognition result of the recognition target from the processing of the first task.

2. The information processing device according to claim 1, wherein: The processing section generates a parameter of the second task using a recognition result of the recognition target from the processing of the first task.

3. The information processing device according to claim 2, wherein: The processing unit configures the neural network of the second task using the generated parameters.

4. The information processing device according to claim 2, wherein: The processing section extracts a plurality of features from the recognition target, and uses a recognition result of the recognition target from the processing of the first task by using the plurality of features to decide whether to execute the processing of the second task and generate the parameter.

5. The information processing device according to claim 4, wherein: The parameters include a region to be processed of the second task and one or more features selected from the plurality of features.

6. The information processing device according to claim 2, wherein: The processing unit determines whether to execute the processing of the second task and generates the parameter using the scene feature obtained from the recognition result of the recognition object from the processing of the first task.

7. The information processing device according to claim 6, wherein: The recognition target is an image acquired by an image capturing unit mounted on a mobile object, the image capturing unit capturing the surrounding environment of the mobile object, and The scene feature is a moving scene feature of the moving object, and the moving scene feature refers to whether there is an object of interest in the image and whether the object of interest is a movable object.

8. The information processing device according to claim 7, wherein: The object of interest is an object that hinders the movement of the moving object.

9. The information processing device according to claim 7, wherein: The first task is semantic segmentation. The second task includes object detection, motion detection and distance detection, and If there is no object of interest in the image, the processing unit only performs the distance detection. If an object of interest exists in the image and the object of interest is not a movable object, the processing unit performs the object detection and the distance detection, and If an object of interest exists in the image and the object of interest is a movable object, the processing section performs the object detection, the motion detection, and the distance detection.

10. The information processing device according to claim 9, wherein: The parameters include a region to be processed of the second task and one or more features selected from the plurality of features, and If the object of interest does not exist in the image, the processing unit generates the parameters so that the entire image area is used as the area to be processed and all of the plurality of features are used, and If an object of interest exists in the image, the processing unit generates the parameters so that a minimum area surrounding the object of interest is used as an area to be processed and one or more features selected from the plurality of features are used according to the number of pixels in the object of interest.

11. The information processing device according to claim 9, wherein: A distance measuring unit is installed on the mobile object, and In the distance detection, the distance is estimated using an aggregation result of aggregating the feature extracted from the image and the distance feature obtained by the distance measurement section.

12. The information processing device according to claim 11, wherein: The distance measuring section includes one or more selected from light detection and ranging LiDAR, a stereo camera device, and a millimeter wave radar.

13. The information processing device according to claim 7, wherein: A plurality of the image capturing units are mounted on the moving object, and The processing section uses the image recognition result from the processing of the first task for each image acquired by each of the plurality of image capturing sections mounted on the mobile object to decide whether to execute the processing of the second task and generate the parameter.

14. The information processing device according to claim 7, wherein: The image capturing unit is a stereo camera or a monocular camera.

15. The information processing device according to claim 7, wherein: The processing unit performs the second task on the image using a neural network of the second task configured by using the generated parameters, and The information processing apparatus further includes a presentation control section that controls a presentation section that provides assistance to an operator of the mobile object based on a recognition result of the second task.

16. The information processing device according to claim 15, wherein: installing one or more selected from a display unit, a light emitting unit, and a sound output unit as the presentation unit on the mobile object, and The presentation control section controls at least one of display control of the display section, lighting control of the light emitting section, and sound output control of the sound output section.

17. The information processing device according to claim 7, wherein: The mobile object is a mobile object capable of autonomous movement, and The processing unit performs the second task on the image using a neural network of the second task configured by using the generated parameters, and The information processing device further includes a planning unit that plans travel and action of the mobile object based on the recognition result of the second task.

18. The information processing device according to claim 1, wherein: The recognition target is an image, The first task is semantic segmentation, and The second task includes one or more selected from object detection, motion detection, distance detection, normal estimation, pose estimation and trajectory estimation.

19. An information processing method performed by an information processing device, comprising: processing a first task for identifying an object, and A recognition result of the recognition target from the processing of the first task is used to decide whether to execute a second task that shares feature extraction with the first task.

20. A program causing an information processing device to execute: a step of processing a first task for identifying a target, and The step of using the recognition result of the recognition target from the processing of the first task to decide whether to perform a second task that shares feature extraction with the first task.

Images