Image processing device

Abstract

To provide an image processing device capable of enhancing recognition accuracy.SOLUTION: An image processing device according to one embodiment of this disclosure includes: an encoder which has a first convolution part that can generate second feature amount data by performing first convolution processing about first feature amount data on the basis of picked-up image data, the first feature amount data corresponding to the picked-up image data and including depth image data being map data about a depth value of a subject, and first control data corresponding to the depth image data and including map data about the effectiveness of the depth value, and can generate a plurality of feature amount data including feature amount data corresponding to the second feature amount data; and a decoder which can generate an inference result of an environment around an imaging device for generating the pick-up image data on the basis of the plurality of feature amount data.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to an image processing device that performs recognition processing based on a captured image. [Background technology]

[0002] Some image processing devices perform machine learning processing based on captured images and depth images. For example, Patent Literature 1 discloses a technology for performing machine learning processing to generate a depth image based on a captured image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2023-503827 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, some image processing devices recognize the surrounding environment of the camera that generated the captured image, and it is desirable for such image processing devices to recognize the surrounding environment with high accuracy.

[0005] It is desirable to provide an image processing device that can improve recognition accuracy. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present disclosure includes an encoder and a decoder. The encoder has a first convolution unit capable of generating second feature data by performing a first convolution process on the first feature data based on the first feature data and the first control data, and is capable of generating a plurality of feature data including feature data corresponding to the second feature data. The first feature data includes captured image data and depth image data corresponding to the captured image data and being map data regarding the depth value of the subject. The first control data corresponds to the depth image data and includes map data regarding the validity of the depth value. The decoder is capable of generating an inference result of the surrounding environment of the imaging device that generated the captured image data based on the plurality of feature data. Effect of the Invention

[0007] According to an image processing device according to an embodiment of the present disclosure, recognition accuracy can be improved. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of a configuration of a surrounding environment recognition device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating an example of the configuration of the surrounding environment recognition device illustrated in FIG. [Diagram 3] FIG. 3 is a block diagram illustrating an example of the configuration of the recognition processing unit illustrated in FIG. [Figure 4] FIG. 4 is a block diagram illustrating an example of a configuration of an information processing device that generates a machine learning model used by the recognition processing unit illustrated in FIG. [Diagram 5] FIG. 5 is a block diagram showing an example of the configuration of a convolutional neural network in the processing unit shown in FIGS. [Figure 6] FIG. 6 is a block diagram illustrating an example of the configuration of one of the two encoders shown in FIG. [Figure 7]FIG. 7 is a block diagram showing an example of the configuration of the other of the two encoders shown in FIG. [Figure 8A] FIG. 8A is a block diagram illustrating an example of the configuration of the convolution unit illustrated in FIG. [Figure 8B] FIG. 8B is a block diagram illustrating another configuration example of the convolution unit illustrated in FIG. [Figure 8C] FIG. 8C is a block diagram illustrating another configuration example of the convolution unit illustrated in FIG. [Figure 9] FIG. 9 is a block diagram illustrating an example of the configuration of the decoder illustrated in FIG. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of a captured image. [Figure 11] FIG. 11 is an explanatory diagram illustrating an example of map data in the mask data. [Figure 12] FIG. 12 is an explanatory diagram illustrating an example of inference result data related to training data. [Figure 13] FIG. 13 is an explanatory diagram illustrating an example of inference result data generated by a machine learning model. [Figure 14] FIG. 14 is a block diagram illustrating a configuration example of a convolutional neural network according to a reference example. [Figure 15] FIG. 15 is a block diagram illustrating an example of the configuration of the encoder illustrated in FIG. [Figure 16] FIG. 16 is an explanatory diagram illustrating an example of inference result data generated by a machine learning model according to a reference example. [Figure 17] FIG. 17 is a block diagram illustrating an example of the configuration of a convolutional neural network according to another reference example. [Figure 18] FIG. 18 is a block diagram illustrating an example of the configuration of the encoder illustrated in FIG. [Figure 19] FIG. 19 is an explanatory diagram illustrating an example of inference result data generated by a machine learning model according to a reference example. [Figure 20] FIG. 20 is a block diagram illustrating a configuration example of a convolutional neural network according to a modified example. [Figure 21] FIG. 21 is a block diagram illustrating an example of the configuration of the encoder illustrated in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment> [Configuration example] 1 and 2 show an example of a configuration of a surrounding environment recognition device 10 equipped with an image processing device according to an embodiment. In this example, the surrounding environment recognition device 10 is mounted on a vehicle 1 such as an automobile. The surrounding environment recognition device 10 is configured to recognize the surrounding environment of the vehicle 1. Here, the surrounding environment includes moving objects, stationary objects, and conditions related to driving. The moving objects include other vehicles, bicycles, pedestrians, animals, and the like around the vehicle 1, and the stationary objects include buildings, road signs, road markings, and the like around the vehicle 1. The conditions related to driving include weather conditions around the vehicle 1, and conditions such as wetness and ice on the road surface. The surrounding environment recognition device 10 includes a stereo camera 11 and an image recognition device 20.

[0011] The stereo camera 11 is configured to capture an image of the area in front of the vehicle 1, thereby generating a set of image data (left image data PL and right image data PR) having a parallax between them. The stereo camera 11 has a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are disposed in the vehicle 1, near the upper part of the windshield of the vehicle 1, and spaced apart from each other by a predetermined distance in the width direction of the vehicle 1. The left camera 11L generates left image data PL, and the right camera 11R generates right image data PR. The left image data PL and the right image data PR constitute stereo image data PIC. The stereo camera 11 performs an imaging operation at a predetermined frame rate (for example, 60 [fps]), thereby generating a series of stereo image data PIC, and supplies the generated stereo image data PIC to the image recognition device 20.

[0012] The image recognition device 20 is configured to recognize the environment around the vehicle 1 based on the stereo image data PIC supplied from the stereo camera 11. In the vehicle 1, for example, based on information about an object recognized by the image recognition device 20, the vehicle 1 can perform, for example, driving control of the vehicle 1, or display information about the recognized object on a console monitor. The image recognition device 20 is configured, for example, by a CPU (Central Processing Unit) that executes a program, a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores the program. The image recognition device 20 has a depth image generation unit 21 and a recognition processing unit 22.

[0013] The depth image generating unit 21 is configured to generate depth image data DD by performing predetermined image processing including stereo matching processing based on the left image data PL and the right image data PR. The depth image data DD is map data regarding the depth of the subject. The pixel value of the depth image data DD indicates a depth value that is the distance from the stereo camera 11 to the subject in the three-dimensional real space.

[0014] The depth image generating unit 21 performs stereo matching processing to detect corresponding points including image points in the left image and image points in the right image that correspond to each other, thereby obtaining a parallax, and calculates a depth based on the parallax. The depth image generating unit 21 may not be able to obtain a depth in a certain image region due to, for example, occlusion. In this case, the depth image generating unit 21 sets the pixel values ​​in that image region in the depth image data DD to a predetermined value indicating an error, instead of the depth value.

[0015] The recognition processing unit 22 is configured to generate a recognition result RES by recognizing the environment around the vehicle 1 based on the left image data PL and right image data PR and the depth image data DD generated by the depth image generation unit 21.

[0016] 3 shows an example of the configuration of the recognition processing unit 22. The recognition processing unit 22 has a mask data generating unit 23 and an image processing unit 24. The mask data generating unit 23 and the image processing unit 24 perform processing based on red image data DR, green image data DG, and blue image data DB included in one of the left image data PL and the right image data PR, and on depth image data DD.

[0017] The mask data generating unit 23 is configured to generate mask data DM based on the depth image data DD. The mask data DM is tensor data including map data indicating the validity of the depth in the depth image data DD. As described later, the image processing unit 24 performs processing based on tensor data related to four channels of image data (image data DR, DG, DB and depth image data DD), so the mask data generating unit 23 generates tensor data including map data for four channels as the mask data DM. For example, the map data for four channels can be the same. The pixel value of the map data in the mask data DM can be "1" or "0". "1" indicates that the pixel value of the depth image data DD at that pixel position is a depth value, and "0" indicates that the pixel value of the depth image data DD at that pixel position is a predetermined value indicating an error. That is, in an image area where the pixel value of the map data is "0", the depth value of the depth image data DD is missing. The mask data generating unit 23 is configured to generate mask data DM including such map data based on the depth image data DD.

[0018] The image processing unit 24 is configured to generate inference result data DRES indicating an inference result of the surrounding environment using a machine learning model based on, for example, image data DR, DG, DB included in one of the left image data PL and the right image data PR and mask data DM. The inference result data DRES is tensor data including map data corresponding to the image data. The machine learning model used in this image processing unit 24 is a model of a convolutional neural network (CNN). The recognition processing unit 22 performs processing based on this inference result data DRES to generate a recognition result RES.

[0019] The image processing unit 24 can perform various recognition processes using a convolutional neural network. Specifically, the image processing unit 24 may perform object detection by, for example, specifying the position of an object that is a subject in an image using a rectangle and performing class classification to classify what kind of object the object is. The image processing unit 24 may also perform so-called semantic segmentation, which performs class classification on a pixel-by-pixel basis. The image processing unit 24 may also perform so-called instance segmentation, which performs class classification on a pixel-by-pixel basis and also classifies instances. The image processing unit 24 may also perform image classification by, for example, performing class classification on the entire image.

[0020] 4 shows an example of the configuration of an information processing device 90 that generates a machine learning model used in the image processing unit 24. The information processing device 90 is, for example, a personal computer. The information processing device 90 has an image processing unit 94 and a storage 95.

[0021] The image processing unit 94 is configured using a CPU, a RAM, etc., and is configured to generate a machine learning model by performing machine learning processing using a plurality of data sets supplied from the storage 95. Each of the plurality of data sets includes image data DR, DG, DB, depth image data DD, mask data DM, and teacher data.

[0022] The storage 95 is configured using, for example, a solid state drive (SSD) or a hard disk drive (HDD), and is configured to store a plurality of data sets. The plurality of data sets are prepared in advance, for example, by an engineer, and stored in the storage 95. The information processing device 90 then performs machine learning processing using the plurality of data sets to generate a machine learning model.

[0023] (About convolutional neural networks) 5 shows an example of a convolutional neural network in the image processing unit 24 (FIG. 3) and the image processing unit 94 (FIG. 4). This convolutional neural network has encoders 31 and 32 and a decoder 33. In this example, this convolutional neural network has a so-called U-Net structure. This U-net structure is described in, for example, the following document: Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

[0024] In this example, the feature data related to the image data DR, DG, DB and the depth image data DD, and the control data related to the mask data DM are treated as three-dimensional tensor data. Therefore, in Fig. 5, these data are shown as cubes. In this Fig. 5, the feature data are shown as shaded cubes, and the control data are shown as unshaded cubes.

[0025] The encoder 31 is configured to generate three pieces of feature amount data based on feature amount data related to the image data DR, DG, and DB. The size of the feature amount data input to the encoder 31 is "3×H×W". Here, "3" is the number of channels and the number of image data (three pieces of image data DR, DG, and DB). "H" indicates the number of pixels in the vertical direction of the image, and "W" indicates the number of pixels in the horizontal direction of the image. The image sizes of the three pieces of image data DR, DG, and DB are the same. The sizes of the three pieces of feature amount data output by the encoder 31 are "64×H×W", "128×H / 2×W / 2", and "256×H / 4×W / 4", respectively.

[0026] The encoder 32 is configured to generate feature data based on feature data related to the image data DR, DG, DB and depth image data DD, and control data related to the mask data DM. The image size of the depth image data DD is the same as the image size of each of the three image data DR, DG, DB, and the image size of the mask data DM is the same as the image size of each of the three image data DR, DG, DB. The size of the feature data related to the image data DR, DG, DB and depth image data DD is "4×H×W". Here, "4" is the number of image data (three image data DR, DG, DB and one depth image data DD). The size of the control data related to the mask data DM is the same as the size of the feature data related to the image data DR, DG, DB and depth image data DD, which is "4×H×W". The size of the feature data generated by the encoder 32 is "512×H / 8×W / 8".

[0027] In this example, the image data DR, DG, DB, depth image data DD, and mask data DM are all the same size, but they may be partially different. In this case, the image sizes can be matched by performing upsampling or downsampling as appropriate.

[0028] The decoder 33 is configured to generate inference result data DRES based on the three feature data supplied from the encoder 31 and one feature data supplied from the encoder 32. For example, in the case of semantic segmentation, the size of the inference result data DRES is "Cclass x H x W". This "Cclass" is the number of classes that can be classified by classification. In other words, the inference result data DRES includes map data for each class classified by classification.

[0029] 6 shows an example of the configuration of the encoder 31. The encoder 31 has a convolution unit 311, a downsampling unit 312, a convolution unit 313, a downsampling unit 314, and a convolution unit 315.

[0030] The convolution unit 311 is configured to generate feature data by performing a convolution process based on the feature data related to the image data DR, DG, and DB input to the encoder 31. The size of the feature data input to the convolution unit 311 is “3×H×W”, and the size of the feature data output by the convolution unit 311 is “64×H×W”.

[0031] The downsampling unit 312 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 311. The size of the feature data output by the downsampling unit 312 is "64×H / 2×W / 2". The convolution unit 313 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 312. The size of the feature data output by the convolution unit 313 is "128×H / 2×W / 2".

[0032] The downsampling unit 314 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 313. The size of the feature data output by the downsampling unit 314 is "128×H / 4×W / 4". The convolution unit 315 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 314. The size of the feature data output by the convolution unit 315 is "256×H / 4×W / 4".

[0033] With this configuration, the encoder 31 repeats convolution processing based on the feature data related to the image data DR, DG, and DB input to the encoder 31. As a result, the encoder 31 compresses information about the image data DR, DG, and DB. The encoder 31 then outputs the three pieces of feature data generated by the convolution units 311, 313, and 315.

[0034] 7 shows an example of the configuration of the encoder 32. The encoder 32 has a convolution unit 321, a downsampling unit 322, a convolution unit 323, a downsampling unit 324, a convolution unit 325, a downsampling unit 326, and convolution units 327 and 328.

[0035] The convolution unit 321 is configured to generate feature data by performing a convolution process based on the feature data related to the image data DR, DG, DB and depth image data DD and the control data related to the mask data DM input to the encoder 32, and to update the input control data. Specifically, the convolution unit 321 generates feature data by performing a convolution process on a portion of the feature data related to the image data DR, DG, DB and depth image data DD that has no missing depth values, based on the control data related to the mask data DM. The size of the feature data and the control data input to the convolution unit 321 is "4×H×W", and the size of the feature data and the control data output by the convolution unit 321 is "64×H×W".

[0036] The downsampling unit 322 is configured to generate feature data and control data by performing downsampling processing based on the feature data and control data supplied from the convolution unit 321. The feature data and control data output by the downsampling unit 312 each have a size of "64×H / 2×W / 2". The convolution unit 323 is configured to generate feature data by performing convolution processing based on the feature data and control data supplied from the downsampling unit 322, and to update the input control data. The feature data and control data output by the convolution unit 323 each have a size of "128×H / 2×W / 2".

[0037] The downsampling unit 324 is configured to generate feature data and control data by performing a downsampling process based on the feature data and control data supplied from the convolution unit 323. The feature data and control data output by the downsampling unit 324 each have a size of "128×H / 4×W / 4". The convolution unit 325 is configured to generate feature data by performing a convolution process based on the feature data and control data supplied from the downsampling unit 324, and to update the input control data. The feature data and control data output by the convolution unit 325 each have a size of "256×H / 4×W / 4".

[0038] The downsampling unit 326 is configured to generate feature data and control data by performing downsampling processing based on the feature data and control data supplied from the convolution unit 325. The feature data and control data output by the downsampling unit 326 each have a size of "256×H / 8×W / 8". The convolution unit 327 is configured to generate feature data by performing convolution processing based on the feature data and control data supplied from the downsampling unit 326, and to update the input control data. The feature data and control data output by the convolution unit 327 each have a size of "512×H / 8×W / 8".

[0039] The convolution unit 328 is configured to generate feature data by performing a convolution process based on the feature data and control data supplied from the convolution unit 327, and to update the input control data. The size of the feature data and control data output by the convolution unit 328 is "512×H / 8×W / 8", respectively.

[0040] With this configuration, the encoder 32 repeats the convolution process based on the feature data related to the image data DR, DG, DB and depth image data DD and the control data related to the mask data DM input to the encoder 32. As a result, the encoder 32 compresses image information on the image data DR, DG, DB and depth image data DD. At that time, the encoder 32 processes image parts with no missing depth values ​​based on the mask data DM, so that the effect of missing depth values ​​on the feature data can be suppressed. Then, the encoder 32 outputs the feature data from among the feature data and control data generated by the convolution unit 328.

[0041] Next, the configuration of the convolution units 321, 323, 325, and 328 will be described with some examples.

[0042] 8A shows an example of the configuration of the convolution unit 320A, which is an example of the convolution units 321, 323, 325, and 328. The convolution unit 320A has a Hadamard product calculation unit 41, a convolution integral calculation unit 42, an activation calculation unit 43, and a rule-based calculation unit 44. The convolution unit 320A generates feature data Fout by performing a convolution process based on the feature data Fin and the control data Min, and generates control data Mout by updating the control data Min. The respective sizes of the feature data Fin and the control data Min are "Cin×Hin×Win", and the respective sizes of the feature data Fout and the control data Mout are "Cout×Hin×Win". Here, "Cin" is the number of channels of the feature data Fin and the control data Min, "Hin" is the number of data in the vertical direction of the feature data Fin, Fout and the control data Min, Mout, "Win" is the number of data in the horizontal direction of the feature data Fin, Fout and the control data Min, Mout, and "Cout" is the number of channels of the feature data Fout and the control data Mout. The number of channels Cout may be the same as the number of channels Cin or may be different.

[0043] The Hadamard product calculation unit 41 is configured to generate feature data by calculating the Hadamard product of the feature data Fin and the control data Min. Specifically, the Hadamard product calculation unit 41 generates feature data by multiplying values ​​at the same position based on the feature data Fin and the control data Min. As a result, the convolution unit 320A performs processing on an image portion that has no missing depth values ​​based on the mask data DM, so that the effect of missing depth values ​​on the feature data can be suppressed. The size of the feature data output by the Hadamard product calculation unit 41 is "Cin x Hin x Win", which is the same as the size of the feature data Fin and the control data Min. In FIG. 8A, the calculation result of the Hadamard product calculation unit 41 is shown as "Fin*Min".

[0044] The convolution integral calculation unit 42 is configured to generate feature data by performing a convolution integral f based on the feature data supplied from the Hadamard product calculation unit 41. The size of the feature data output by the convolution integral calculation unit 42 is "Cout x Hin x Win". In FIG. 8A, the calculation result of the convolution integral calculation unit 42 is shown as "f(Fin*Min)" using the convolution integral f. The calculation parameters of this convolution integral f are adjusted by the information processing device 90 performing machine learning processing. The calculation parameters include, for example, a weight parameter and a bias parameter.

[0045] The activation calculation unit 43 is configured to generate feature data Fout by performing activation calculation using an activation function φ based on the feature data supplied from the convolution integral calculation unit 42. In Fig. 8A, the calculation result of the activation calculation unit 43 is shown as "φ(f(Fin*Min))" using the activation function φ.

[0046] The rule-based calculation unit 44 is configured to generate the control data Mout by performing a predetermined rule-based calculation based on the control data Min. The rule-based calculation unit 44 can perform, for example, an expansion process using a morphological process.

[0047] When using this convolution unit 320A, the mask data generation unit 23 (FIG. 3) can generate the mask data DM by, for example, copying the map data generated based on the depth image data DD in the channel direction. That is, in this case, the map data for the four channels in the mask data DM are the same.

[0048] In this convolution unit 320A, the calculation parameters of the convolution integral f related to the feature amount data are adjusted by machine learning processing, thereby improving the recognition accuracy of the recognition processing unit 22. Also, in the convolution unit 320A, the rule-based calculation unit 44 performs calculations based on rules designed by humans, so that calculations that are easier for humans to interpret can be performed compared to the case of using a calculation method obtained by machine learning processing.

[0049] 8B shows an example of the configuration of convolution section 320B, which is another example of convolution sections 321, 323, 325, and 328. Convolution section 320B has a Hadamard product calculation section 41, a convolution integral calculation section 42, an activation calculation section 43, a convolution integral calculation section 45, and an activation calculation section 46. The Hadamard product calculation section 41, the convolution integral calculation section 42, and the activation calculation section 43 are the same as those in convolution section 320A (FIG. 8A).

[0050] The convolution integral calculation unit 45 is configured to generate control data by performing a convolution integral g based on the feature data supplied from the Hadamard product calculation unit 41. The size of the feature data output by the convolution integral calculation unit 45 is "Cout x Hin x Win". In FIG. 8B, the calculation result of the convolution integral calculation unit 45 is shown as "g(Fin*Min)" using the convolution integral g. The calculation parameters of this convolution integral g are adjusted by the information processing device 90 performing machine learning processing. The calculation parameters include, for example, a weight parameter and a bias parameter.

[0051] The activation calculation unit 46 is configured to generate the control data Mout by performing activation calculation using the activation function σ based on the control data supplied from the convolution integral calculation unit 45. The activation function σ is a nonlinear function whose output value is between 0 and 1, and for example, a sigmoid function can be used. In FIG. 8B, the calculation result of the activation calculation unit 46 is shown as "σ(g(Fin*Min))" using the activation function σ.

[0052] When using this convolution unit 320B, the mask data generation unit 23 (FIG. 3) can generate the mask data DM by, for example, copying the map data generated based on the depth image data DD in the channel direction. That is, in this case, the map data for the four channels in the mask data DM are the same as each other. In addition, the mask data generation unit 23 may generate the map data corresponding to the depth image data DD among the map data for the four channels based on the depth image data DD, and generate the mask data DM by setting all pixel values ​​in the three map data corresponding to the image data DR, DG, and DB to "1".

[0053] In the convolution unit 320B, in addition to the calculation parameters of the convolution integral f related to the feature amount data, the calculation parameters of the convolution integral g related to the control data are also adjusted by machine learning processing. Therefore, the convolution integral g is adjusted to be an optimal calculation, so that the recognition accuracy of the recognition processing unit 22 can be further improved.

[0054] 8C shows an example of the configuration of a convolution unit 320C, which is another example of the convolution units 321, 323, 325, and 328. The convolution unit 320C has a Hadamard product calculation unit 41, a convolution integral calculation unit 42, an activation calculation unit 43, a convolution integral calculation unit 47, and an activation calculation unit 46. The Hadamard product calculation unit 41, the convolution integral calculation unit 42, and the activation calculation unit 43 are the same as those in the convolution unit 320A (FIG. 8A). The activation calculation unit 46 is the same as those in the convolution unit 320B (FIG. 8B).

[0055] The convolution integral calculation unit 47 is configured to generate control data by performing a convolution integral h based on the control data Min. The size of the control data output by the convolution integral calculation unit 47 is "Cout x Hin x Win". In FIG. 8C, the calculation result of the convolution integral calculation unit 47 is shown as "h(Min)" using the convolution integral h. The calculation parameters of this convolution integral h are adjusted by the information processing device 90 performing machine learning processing. The calculation parameters include, for example, a weight parameter and a bias parameter.

[0056] When using this convolution unit 320C, the mask data generation unit 23 (FIG. 3) can generate the mask data DM by, for example, copying the map data generated based on the depth image data DD in the channel direction. That is, in this case, the map data for the four channels in the mask data DM are the same as each other. In addition, the mask data generation unit 23 may generate the map data corresponding to the depth image data DD among the map data for the four channels based on the depth image data DD, and generate the mask data DM by setting all pixel values ​​in the three map data corresponding to the image data DR, DG, and DB to "1".

[0057] In this convolution unit 320C, in addition to the calculation parameters of the convolution integral f related to the feature data, the calculation parameters of the convolution integral h related to the control data are also adjusted by machine learning processing. Therefore, the convolution integral h is adjusted to be an optimal calculation, so that the recognition accuracy of the recognition processing unit 22 can be further improved. Also, in this convolution unit 320C, the feature data Fin is not used in the calculation to generate the control data Mout. Therefore, in the convolution unit 320C, the convolution integral h is adjusted to be an optimal calculation without being affected by the feature data Fin, so that the recognition accuracy of the recognition processing unit 22 can be further improved.

[0058] Fig. 9 shows an example of the configuration of the decoder 33 shown in Fig. 5. The decoder 33 has an upsampling unit 331, a connection unit 332, a convolution unit 333, an upsampling unit 334, a connection unit 335, a convolution unit 336, an upsampling unit 337, a connection unit 338, and convolution units 339 and 340.

[0059] The upsampling unit 331 is configured to generate feature data by performing an upsampling process based on the feature data supplied from the encoder 32. The size of the feature data supplied from the encoder 32 is "512×H / 8×W / 8". The size of the feature data output by the upsampling unit 331 is "512×H / 4×W / 4". The linking unit 332 is configured to link the feature data supplied from the encoder 31 and the feature data supplied from the upsampling unit 331 in the channel direction. The size of the feature data supplied from the encoder 31 is "256×H / 4×W / 4". The size of the feature data output by the linking unit 332 is "768×H / 4×W / 4". The convolution unit 333 is configured to generate feature data by performing a convolution process based on the feature data supplied from the linking unit 332. The size of the feature data output by the convolution unit 333 is “256×H / 4×W / 4”.

[0060] The upsampling unit 334 is configured to generate feature data by performing an upsampling process based on the feature data supplied from the convolution unit 333. The size of the feature data output from the upsampling unit 334 is “256×H / 2×W / 2”. The concatenation unit 335 is configured to concatenate the feature data supplied from the encoder 31 and the feature data supplied from the upsampling unit 334 in the channel direction. The size of the feature data supplied from the encoder 31 is “128×H / 2×W / 2”. The size of the feature data output from the concatenation unit 335 is “384×H / 2×W / 2”. The convolution unit 336 is configured to generate feature data by performing a convolution process based on the feature data supplied from the concatenation unit 335. The size of the feature data output from the convolution unit 336 is “128×H / 2×W / 2”.

[0061] The upsampling unit 337 is configured to generate feature data by performing an upsampling process based on the feature data supplied from the convolution unit 336. The size of the feature data output by the upsampling unit 337 is “128×H×W”. The linking unit 338 is configured to link the feature data supplied from the encoder 31 and the feature data supplied from the upsampling unit 337 in the channel direction. The size of the feature data supplied from the encoder 31 is “64×H×W”. The size of the feature data output by the linking unit 338 is “192×H×W”. The convolution unit 339 is configured to generate feature data by performing a convolution process based on the feature data supplied from the linking unit 338. The size of the feature data output by the convolution unit 339 is “64×H×W”.

[0062] The convolution unit 340 is configured to generate inference result data DRES by performing a convolution process based on the feature data supplied from the convolution unit 339. For example, in the case of semantic segmentation, the size of the feature data output by the convolution unit 340 is "Cclass x H x W". This "Cclass" is the number of classes that can be classified by class classification.

[0063] With this configuration, the decoder 33 repeats the convolution process based on the four feature data supplied from the encoders 31 and 32. This causes the size of the compressed image to return to its original size. The decoder 33 then generates inference result data DRES according to applications such as object detection, semantic segmentation, instance segmentation, and image classification.

[0064] With this configuration, the information processing device 90 generates a machine learning model, and the surrounding environment recognition device 10 uses this machine learning model to recognize the surrounding environment of the vehicle 1 based on the image data DR, DG, DB, the depth image data DD, and the mask data DM. In this way, in the information processing device 90 and the surrounding environment recognition device 10, the mask data DM including map data indicating the validity of the depth in the depth image data DD is input to the machine learning model, so that it is possible to improve the recognition accuracy.

[0065] Here, the image processing units 24 and 94 correspond to a specific example of an "image processing device" in an embodiment of the present disclosure. The image data DR, DG, and DB correspond to a specific example of "captured image data" in an embodiment of the present disclosure. The depth image data DD corresponds to a specific example of "depth image data" in an embodiment of the present disclosure. The image data DR, DG, and DB and the depth image data DD correspond to a specific example of "first feature amount data" in an embodiment of the present disclosure. The mask data DM corresponds to a specific example of "first control data" in an embodiment of the present disclosure. The convolution integral f corresponds to a specific example of "first convolution processing" in an embodiment of the present disclosure. The convolution unit 321 corresponds to a specific example of "first convolution unit" in an embodiment of the present disclosure. The encoders 31 and 32 correspond to a specific example of "encoder" in an embodiment of the present disclosure. The decoder 33 corresponds to a specific example of "decoder" in an embodiment of the present disclosure. The convolution unit 311 corresponds to a specific example of "second convolution unit" in an embodiment of the present disclosure. The convolution integral g corresponds to a specific example of a "third convolution process" in an embodiment of the present disclosure. The convolution integral h corresponds to a specific example of a "fourth convolution process" in an embodiment of the present disclosure.

[0066] [Actions and Actions] Next, the operation and function of the information processing device 90 and the surrounding environment recognition device 10 according to this embodiment will be described.

[0067] (Overview of overall operation) First, an overview of the overall operation of the surrounding environment recognition device 10 and the information processing device 90 will be described with reference to FIGS.

[0068] The image processing unit 94 of the information processing device 90 generates a machine learning model by performing machine learning processing using multiple data sets including image data DR, DG, DB, depth image data DD, mask data DM, and teacher data supplied from the storage 95.

[0069] The stereo camera 11 captures an image of the area in front of the vehicle 1, thereby generating a set of image data (left image data PL and right image data PR) having a parallax between them. The depth image generation unit 21 of the image recognition device 20 generates depth image data DD by performing a predetermined image processing including a stereo matching process based on the left image data PL and right image data PR. The recognition processing unit 22 recognizes the environment around the vehicle 1 based on the left image data PL and right image data PR and the depth image data DD generated by the depth image generation unit 21. Specifically, the image processing unit 24 of the recognition processing unit 22 generates inference result data DRES indicating an inference result of the surrounding environment using a machine learning model based on the image data DR, DG, DB included in one of the left image data PL and the right image data PR, and the mask data DM.

[0070] (Details of operation) In the convolutional neural network in the image processing unit 24 and the image processing unit 94, as shown in Fig. 5, the encoder 31 generates three pieces of feature amount data based on the feature amount data related to the image data DR, DG, DB. The encoder 32 generates one piece of feature amount data based on the feature amount data related to the image data DR, DG, DB and the depth image data DD, and the control data related to the mask data DM. The decoder 33 generates inference result data DRES based on the three pieces of feature amount data supplied from the encoder 31 and the one piece of feature amount data supplied from the encoder 32.

[0071] An example of machine learning processing using the convolutional neural network shown in Fig. 5 will be described below. In this example, semantic segmentation is performed to perform class classification on a pixel-by-pixel basis. In this example, the convolution units 321, 323, 325, and 328 shown in Fig. 7 are configured using the convolution unit 320C (Fig. 8C).

[0072] Fig. 10 shows an example of captured image data. Fig. 11 shows an example of map data included in mask data DM. This map data corresponds to the captured image shown in Fig. 10. White image parts in the map data shown in Fig. 10 indicate that pixel values ​​of the depth image data DD are depth values, and black image parts in the map data shown in Fig. 10 indicate that pixel values ​​of the depth image data DD are not depth values.

[0073] Fig. 12 shows an example of training data. In this training data, a class is assigned according to the subject. Specifically, for example, a floor class is assigned to a floor, a bed mattress is assigned to a bed class, a wall is assigned to a wall class, and paintings and photographs hung on the wall are assigned to an art class. In this example, an unclassified class is assigned to parts that do not require classification, shown in black in Fig. 12.

[0074] In this processing example, a machine learning model was generated by performing machine learning processing using multiple data sets including such data.Then, using the generated machine learning model, inference result data DRES was generated based on other captured image data.

[0075] An example of the authentication result data DRES is shown in Figure 13. In this example, it was confirmed that the shapes of floors, beds, walls, paintings, photographs, etc. were correctly recognized and that their classes were correctly classified.

[0076] Next, the machine learning process using the convolutional neural network according to the reference example will be described in detail with some examples.

[0077] (Reference example E1) 14 shows an example of a convolutional neural network according to a reference example E1. This convolutional neural network has an encoder 51 and a decoder 33. The convolutional neural network according to the reference example E1 has a U-Net structure, similar to the convolutional neural network according to the present embodiment (FIG. 5). The decoder 33 is the same as the decoder 33 according to the present embodiment (FIGS. 5 and 9).

[0078] The encoder 51 is configured to generate four pieces of feature amount data based on the feature amount data related to the image data DR, DG, and DB. That is, in the present embodiment, as shown in FIG. 5, the encoder 31 generates three pieces of feature amount data based on the feature amount data related to the image data DR, DG, and DB, and the encoder 32 generates one piece of feature amount data based on the feature amount data related to the image data DR, DG, and DB and the depth image data DD, and the control data related to the mask data DM, but in this reference example E1, the encoder 51 generates four pieces of feature amount data based on the feature amount data related to the image data DR, DG, and DB. The size of the feature amount data input to the encoder 51 is "3×H×W". Here, "3" is the number of channels and the number of image data (three image data DR, DG, and DB). The sizes of the four feature data output by the encoder 51 are "64×H×W", "128×H / 2×W / 2", "256×H / 4×W / 4", and "512×H / 8×W / 8", respectively.

[0079] 15 shows an example of the configuration of the encoder 51. The encoder 51 has a convolution unit 511, a downsampling unit 512, a convolution unit 513, a downsampling unit 514, a convolution unit 515, a downsampling unit 516, and convolution units 517 and 518.

[0080] The convolution unit 511 is configured to generate feature data by performing a convolution process based on the feature data related to the image data DR, DG, and DB input to the encoder 51. The size of the feature data input to the convolution unit 511 is “3×H×W”, and the size of the feature data output by the convolution unit 511 is “64×H×W”.

[0081] The downsampling unit 512 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 511. The size of the feature data output by the downsampling unit 512 is "64×H / 2×W / 2". The convolution unit 513 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 512. The size of the feature data output by the convolution unit 513 is "128×H / 2×W / 2".

[0082] The downsampling unit 514 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 513. The size of the feature data output by the downsampling unit 514 is "128×H / 4×W / 4". The convolution unit 515 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 514. The size of the feature data output by the convolution unit 515 is "256×H / 4×W / 4".

[0083] The downsampling unit 516 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 515. The size of the feature data output by the downsampling unit 516 is "256×H / 8×W / 8". The convolution unit 517 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 516. The size of the feature data output by the convolution unit 517 is "512×H / 8×W / 8".

[0084] The convolution unit 518 is configured to generate feature data by performing a convolution process based on the feature data supplied from the convolution unit 517. The size of the feature data output by the convolution unit 518 is “512×H / 8×W / 8”.

[0085] With this configuration, the encoder 51 repeats the convolution process based on the feature data related to the image data DR, DG, and DB input to the encoder 51. Then, the encoder 51 outputs the four feature data generated by the convolution units 511, 513, 515, and 518.

[0086] 16 shows an example of the authentication result data DRES when the convolutional neural network according to the reference example E1 is used. In this example, the shapes of the bed, painting, and photograph, for example, cannot be correctly recognized. Thus, the recognition accuracy is low in the recognition process using only the image data DR, DG, and DB.

[0087] (Reference example E2) 17 shows an example of a convolutional neural network according to a reference example E2. This convolutional neural network has an encoder 61 and a decoder 33. The convolutional neural network according to the reference example E2 has a U-Net structure, similar to the convolutional neural network according to the present embodiment (FIG. 5). The decoder 33 is the same as the decoder 33 according to the present embodiment (FIGS. 5 and 9).

[0088] The encoder 61 is configured to generate four pieces of feature amount data based on the feature amount data related to the image data DR, DG, DB and the depth image data DD. That is, in the present embodiment, as shown in FIG. 5, the encoder 31 generates three pieces of feature amount data based on the feature amount data related to the image data DR, DG, DB, and the encoder 32 generates one piece of feature amount data based on the feature amount data related to the image data DR, DG, DB and the depth image data DD and the control data related to the mask data DM, but in this reference example E2, the encoder 61 generates four pieces of feature amount data based on the feature amount data related to the image data DR, DG, DB and the depth image data DD. The size of the feature amount data input to the encoder 61 is "4×H×W". Here, "4" is the number of channels and the number of image data (three image data DR, DG, DB and one depth image data DD). The sizes of the four feature data output by the encoder 61 are "64×H×W", "128×H / 2×W / 2", "256×H / 4×W / 4", and "512×H / 8×W / 8", respectively.

[0089] 18 shows an example of the configuration of the encoder 61. The encoder 61 has a convolution unit 611, a downsampling unit 612, a convolution unit 613, a downsampling unit 614, a convolution unit 615, a downsampling unit 616, and convolution units 617 and 618.

[0090] The convolution unit 611 is configured to generate feature data by performing a convolution process based on the feature data related to the image data DR, DG, and DB input to the encoder 61. The size of the feature data input to the convolution unit 611 is “4×H×W”, and the size of the feature data output by the convolution unit 611 is “64×H×W”.

[0091] The downsampling unit 612 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 611. The size of the feature data output by the downsampling unit 612 is "64×H / 2×W / 2". The convolution unit 613 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 612. The size of the feature data output by the convolution unit 613 is "128×H / 2×W / 2".

[0092] The downsampling unit 614 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 613. The size of the feature data output by the downsampling unit 614 is "128×H / 4×W / 4". The convolution unit 615 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 614. The size of the feature data output by the convolution unit 615 is "256×H / 4×W / 4".

[0093] The downsampling unit 616 is configured to generate feature data by performing downsampling processing based on the feature data supplied from the convolution unit 615. The size of the feature data output by the downsampling unit 616 is "256×H / 8×W / 8". The convolution unit 617 is configured to generate feature data by performing convolution processing based on the feature data supplied from the downsampling unit 616. The size of the feature data output by the convolution unit 617 is "512×H / 8×W / 8".

[0094] The convolution unit 618 is configured to generate feature data by performing a convolution process based on the feature data supplied from the convolution unit 617. The size of the feature data output by the convolution unit 618 is “512×H / 8×W / 8”.

[0095] With this configuration, the encoder 61 repeats the convolution process based on the feature data related to the image data DR, DG, DB and the depth image data DD input to the encoder 61. Then, the encoder 61 outputs the four feature data generated by the convolution units 611, 613, 615, and 618.

[0096] FIG. 19 shows an example of authentication result data DRES when the convolutional neural network according to reference example E2 is used. In this example, the shapes of the bed, painting, and photograph are correctly recognized compared to the authentication result data DRES (FIG. 16) according to reference example E1. However, for example, a part of the bed is classified into a class different from the bed class, and a part of the painting and photograph is classified into a class different from the art class. In this way, in the recognition process using the image data DR, DG, DB and the depth image data DD, the recognition accuracy can be improved compared to the case where only the image data DR, DG, DB are used (FIG. 16), but the recognition accuracy is still insufficient.

[0097] Thus, in the convolutional neural networks of reference examples E1 and E2, the recognition accuracy is insufficient, as shown in Figures 16 and 19. For example, the DICE score in reference example E1 (Figure 16) was 0.62, and the DICE score in reference example E2 (Figure 19) was 0.654. ​​This DICE score is an index representing the recognition accuracy, and the higher the value, the higher the recognition accuracy.

[0098] On the other hand, in the convolutional neural network according to the present embodiment, as shown in FIG. 13, the shapes of the bed, painting, and photograph are correctly recognized, and the bed, painting, and photograph are classified into the correct classes. The DICE score in this embodiment (FIG. 13) was 0.668. Thus, in the convolutional neural network according to the present embodiment, the recognition process is performed using the image data DR, DG, DB, the depth image data DD, and the mask data DM. This makes it possible to suppress the effect of missing depth values ​​on the feature amount data, thereby improving the recognition accuracy.

[0099] In this way, the convolutional neural network in the image processing unit 24, 94 has a first convolution unit (convolution unit 321) capable of generating second feature data by performing a first convolution process on the first feature data based on the captured image data (image data DR, DG, DB), the first feature data including depth image data DD which is map data on the depth value of the subject corresponding to the captured image data (image data DR, DG, DB), and the first control data (mask data DM) which corresponds to the depth image data DD and includes map data on the validity of the depth value, and is provided with encoders 31, 32 capable of generating a plurality of feature data including feature data according to the first feature data, and a decoder 33 capable of generating an inference result of the surrounding environment of the imaging device which generated the captured image data (image data DR, DG, DB) based on the plurality of feature data. As a result, the first convolution unit processes an image part having no missing depth values ​​based on the mask data DM, for example, so that the influence of missing depth values ​​on the second feature data can be suppressed. As a result, this convolutional neural network can improve recognition accuracy.

[0100] In the convolutional neural network in the image processing unit 24, 94, the encoder 31, 32 further includes a second convolution unit (convolution unit 311) capable of generating fourth feature data by performing a second convolution process on the third feature data based on the third feature data including the captured image data (image data DR, DG, DB). The plurality of feature data includes feature data corresponding to the fourth feature data. This allows processing to be performed on image parts with missing depth values, so that data to be processed can be secured when, for example, the image parts with missing depth values ​​are large. As a result, the convolutional neural network can improve the recognition accuracy.

[0101] In addition, in the convolution neural network in the image processing unit 24, 94, the first convolution unit (convolution unit 321) can generate multiplied data by multiplying values ​​at corresponding coordinate positions in the first feature data (image data DR, DG, DB and frequency image data DD) and the first control data (mask data DM), and can generate second feature data by performing a first convolution process based on the multiplied data. As a result, the first convolution unit processes image parts with no missing depth values ​​based on the mask data DM, so that the effect of missing depth values ​​on the second feature data can be suppressed. As a result, this convolution neural network can improve the recognition accuracy.

[0102] [effect] As described above, in this embodiment, the device has a first convolution unit capable of generating second feature data by performing a first convolution process on the first feature data based on the captured image data, first feature data including depth image data that corresponds to the captured image data and is map data on the depth value of the subject, and first control data that corresponds to the depth image data and includes map data on the validity of the depth value, an encoder capable of generating multiple feature data including feature data corresponding to the first feature data, and a decoder capable of generating an inference result of the environment around the imaging device that generated the captured image data based on the multiple feature data, thereby improving recognition accuracy.

[0103] In this embodiment, the encoder further includes a second convolution unit capable of generating fourth feature data by performing a second convolution process on the third feature data based on the third feature data including the captured image data. The plurality of feature data includes feature data corresponding to the fourth feature data. This can improve the recognition accuracy.

[0104] In this embodiment, the first convolution unit can generate multiplied data by multiplying values ​​at corresponding coordinate positions in the first feature data and the first control data, and can generate second feature data by performing a first convolution process based on the multiplied data, thereby improving recognition accuracy.

[0105] [Variation 1] In the above embodiment, the convolution units 320A to 320C (FIGS. 8A to 8C) perform convolution integrals and activation calculations, but the present invention is not limited to these, and other calculations may be performed. Specifically, for example, the convolution units 320A to 320C may perform upsampling processing to increase the resolution of the feature amount, for example, using bilinear interpolation or nearest neighbor. Furthermore, the convolution units 320A to 320C may perform downsampling processing to decrease the resolution of the feature amount, for example, using max pooling or average pooling. Furthermore, the convolution units 320A to 320C may perform normalization processing, for example, using batch normalization or instance normalization. Furthermore, the convolution units 320A to 320C may perform downsampling processing by devising a convolution integral method, for example, by setting the stride to 2 or more or by using pixel unshuffle. Furthermore, the convolution units 320A to 320C may perform upsampling processing by devising a convolution integral method, such as using Transposed Convolution or Pixel shuffle. In the examples of Figs. 8A to 8C, the convolution units 320A to 320C have the same resolution (Hin, Win) of the input feature amount data and control data and the same resolution (Hin, Win) of the output feature amount data and control data, but when performing such upsampling processing or downsampling processing, the resolution of the input feature amount data and control data and the resolution of the output feature amount data and control data may be different from each other.

[0106] [Variation 2] In the above embodiment, the encoder 31 generates three pieces of feature data based on the feature data related to the image data DR, DG, and DB, and the encoder 32 generates one piece of feature data based on the feature data related to the image data DR, DG, and DB and the depth image data DD, and the control data related to the mask data DM, but this is not limited to the above. The convolutional neural network according to this modification will be described in detail below.

[0107] Fig. 20 shows an example of a convolutional neural network according to this modification. This convolutional neural network has an encoder 71 and a decoder 33. The convolutional neural network according to this modification has a U-Net structure, similar to the convolutional neural network according to the above embodiment (Fig. 5). The decoder 33 is the same as the decoder 33 according to this embodiment (Figs. 5 and 9).

[0108] The encoder 71 is configured to generate four pieces of feature data based on feature data related to the image data DR, DG, DB and depth image data DD, and control data related to the mask data DM. The size of the feature data and control data input to the encoder 71 is "4×H×W". The size of the four pieces of feature data output by the encoder 71 is "64×H×W", "128×H / 2×W / 2", "256×H / 4×W / 4", and "512×H / 8×W / 8", respectively.

[0109] 21 shows an example of the configuration of the encoder 71. The encoder 71 has a convolution unit 711, a downsampling unit 712, a convolution unit 713, a downsampling unit 714, a convolution unit 715, a downsampling unit 716, and convolution units 717 and 718.

[0110] The convolution unit 711 is configured to generate feature data by performing a convolution process based on the feature data related to the image data DR, DG, DB and depth image data DD and the control data related to the mask data DM input to the encoder 71, and to update the input control data. Specifically, the convolution unit 711 generates feature data by performing a convolution process on a portion of the feature data related to the image data DR, DG, DB and depth image data DD that has no missing depth values, based on the control data related to the mask data DM. The size of the feature data and the control data input to the convolution unit 711 is "4×H×W", and the size of the feature data and the control data output by the convolution unit 711 is "64×H×W".

[0111] The downsampling unit 712 is configured to generate feature data and control data by performing downsampling processing based on the feature data and control data supplied from the convolution unit 711. The feature data and control data output by the downsampling unit 712 each have a size of "64×H / 2×W / 2". The convolution unit 713 is configured to generate feature data by performing convolution processing based on the feature data and control data supplied from the downsampling unit 712, and to update the input control data. The feature data and control data output by the convolution unit 713 each have a size of "128×H / 2×W / 2".

[0112] The downsampling unit 714 is configured to generate feature data and control data by performing a downsampling process based on the feature data and control data supplied from the convolution unit 713. The feature data and control data output by the downsampling unit 714 each have a size of "128×H / 4×W / 4". The convolution unit 715 is configured to generate feature data by performing a convolution process based on the feature data and control data supplied from the downsampling unit 714, and to update the input control data. The feature data and control data output by the convolution unit 715 each have a size of "256×H / 4×W / 4".

[0113] The downsampling unit 716 is configured to generate feature data and control data by performing a downsampling process based on the feature data and control data supplied from the convolution unit 715. The feature data and control data output by the downsampling unit 716 each have a size of "256×H / 8×W / 8". The convolution unit 717 is configured to generate feature data by performing a convolution process based on the feature data and control data supplied from the downsampling unit 716, and to update the input control data. The feature data and control data output by the convolution unit 717 each have a size of "512×H / 8×W / 8".

[0114] The convolution unit 718 is configured to generate feature data by performing a convolution process based on the feature data and control data supplied from the convolution unit 717, and to update the input control data. The size of the feature data and control data output by the convolution unit 718 is "512×H / 8×W / 8", respectively.

[0115] With this configuration, the encoder 71 repeats convolution processing based on the feature data related to the image data DR, DG, DB and depth image data DD and the control data related to the mask data DM input to the encoder 71. As a result, the encoder 71 compresses image information for the image data DR, DG, DB and depth image data DD. At that time, the encoder 71 processes image parts with no missing depth values ​​based on the mask data DM, so that the effect of missing depth values ​​on the feature data can be suppressed. The encoder 71 then outputs the four feature data generated by the convolution units 711, 713, 715, and 718.

[0116] Convolution sections 711, 713, 715, and 718 are configured using, for example, convolution sections 320A to 320C (FIGS. 8A to 8C).

[0117] [Other variations] Also, two or more of these variations may be combined.

[0118] Although the present technology has been described above by giving embodiments and some modified examples, the present technology is not limited to these embodiments and the like, and various modifications are possible.

[0119] For example, in the above embodiment, the encoder 31 shown in FIG. 6 is used, but the present invention is not limited to this, and an encoder with another configuration may be used. In the above embodiment, the encoder 32 shown in FIG. 7 is used, but the present invention is not limited to this, and an encoder with another configuration may be used. In the above embodiment, the decoder 33 shown in FIG. 9 is used, but the present invention is not limited to this, and an encoder with another configuration may be used. In the above embodiment, the convolution units 320A to 320C shown in FIGS. 8A to 8C are used, but the present invention is not limited to this, and a convolution unit with a different configuration may be used.

[0120] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0121] Furthermore, the present disclosure may take the following aspects.

[0122] (1) an encoder having a first convolution unit capable of generating second feature data by performing a first convolution process on the first feature data based on captured image data, first feature data including depth image data corresponding to the captured image data and being map data regarding depth values ​​of a subject, and first control data corresponding to the depth image data and including map data regarding validity of depth values, and capable of generating a plurality of feature data including feature data corresponding to the second feature data; a decoder capable of generating an inference result of an environment surrounding an imaging device that generated the captured image data based on the plurality of feature amount data; An image processing device comprising: (2) the encoder further includes a second convolution unit configured to perform a second convolution process on third feature data based on the captured image data to generate fourth feature data, The plurality of feature amount data includes feature amount data corresponding to the fourth feature amount data. The image processing device according to (1) above. (3) The first convolution unit includes: multiplication data can be generated by multiplying values ​​at corresponding coordinate positions in the first feature amount data and the first control data; The second feature amount data can be generated by performing the first convolution process based on the multiplied data. The image processing device according to (1) or (2). (4) The first convolution unit includes: A second control data is generated by performing rule-based processing based on the first control data; The second feature data and the second control data can be output. The image processing device according to (3) above. (5) The first convolution unit includes: A third convolution process is performed based on the multiplied data to generate second control data; The second feature data and the second control data can be output. The image processing device according to (3) above. (6) The first convolution unit includes: A fourth convolution process is performed on the first control data to generate second control data based on the first control data; The second feature data and the second control data can be output. The image processing device according to (3) above. [Explanation of symbols]

[0123] 1...vehicle, 10...surrounding environment recognition device, 11...stereo camera, 11L...left camera, 11R...right camera, 20...image recognition device, 21...depth image generation unit, 22...recognition processing unit, 23...mask data generation unit, 24...image processing unit, 31, 32, 71...encoder, 33...decoder, 41...Hadamard product calculation unit, 42...convolution integral calculation unit, 43...activation calculation unit, 44...rule-based calculation unit, 45...convolution integral calculation unit, 46...activation calculation unit, 47...convolution integral calculation unit, 90...information processing device, 94...image processing unit, 95...storage, 311, 313, 315...convolution unit, 312, 314...downsampling unit, 320A, 320 B, 320C...convolution unit, 321, 323, 325, 327, 328...convolution unit, 322, 324, 326...downsampling unit, 331, 334, 337...upsampling unit, 332, 335, 338...connection unit, 333, 336, 339, 340...convolution unit, 711, 713, 715, 717, 718...convolution unit, 712, 714, 716...downsampling unit, DD...depth image data, DM...mask data, DRES...inference result data, Fin, Fout...feature data, Min, Mout...control data, PIC...stereo image data, PL...left image data, PR...right image data, RES...recognition result.

Claims

1. an encoder having a first convolution unit capable of generating second feature data by performing a first convolution process on the first feature data based on captured image data, first feature data including depth image data corresponding to the captured image data and being map data regarding depth values ​​of a subject, and first control data corresponding to the depth image data and including map data regarding validity of depth values, and capable of generating a plurality of feature data including feature data corresponding to the second feature data; a decoder capable of generating an inference result of an environment surrounding an imaging device that generated the captured image data based on the plurality of feature amount data; An image processing device comprising:

2. the encoder further includes a second convolution unit configured to perform a second convolution process on third feature data based on the captured image data to generate fourth feature data, The plurality of feature amount data includes feature amount data corresponding to the fourth feature amount data. The image processing device according to claim 1 .

3. The first convolution unit includes: multiplication data can be generated by multiplying values ​​at corresponding coordinate positions in the first feature amount data and the first control data; The second feature amount data can be generated by performing the first convolution process based on the multiplied data. The image processing device according to claim 1 .

4. The first convolution unit includes: A second control data is generated by performing a rule-based process based on the first control data; The second feature amount data and the second control data can be output. The image processing device according to claim 3 .

5. The first convolution unit includes: A third convolution process is performed based on the multiplication data to generate second control data; The second feature amount data and the second control data can be output. The image processing device according to claim 3 .

6. The first convolution unit includes: A fourth convolution process is performed on the first control data to generate second control data based on the first control data; The second feature amount data and the second control data can be output. The image processing device according to claim 3 .

Images