Machine Learning Equipment

Abstract

A machine learning device according to one embodiment of the present disclosure comprises: a road surface detection processing unit that detects, on the basis of a first captured image and a first distance image corresponding to the first captured image, a road surface included in the first captured image; a distance value selection unit that selects, on the basis of the processing result from the road surface detection processing unit, one or more distance values to be processed from among a plurality of distance values included in the first distance image; and a learning processing unit that performs machine learning processing on the basis of the first captured image and the one or more distance values to thereby generate a learning model which receives input of a second captured image and which outputs a second distance image corresponding to the second captured image.

Description

[Technical field]

[0001] The present disclosure relates to a machine learning device that performs learning processing based on captured images and range images. [Background technology]

[0002] In a vehicle, the external environment is often detected, and the vehicle is controlled based on the detection result. In recognizing the external environment, the distance from the vehicle to surrounding three-dimensional objects is often detected. Patent Document 1 discloses a technology for performing arithmetic processing of a neural network based on a captured image and a distance image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-147286 A Summary of the Invention

[0004] By the way, there is a learning model that generates a distance image based on a captured image. It is desirable for the generated distance image to be highly accurate, and further improvement in accuracy is expected.

[0005] It is desirable to provide a machine learning device capable of generating a learning model that generates a highly accurate range image.

[0006] A machine learning device according to an embodiment of the present disclosure includes: A processing section; A road surface detection processing unit; A three-dimensional object detection unit; The device includes a distance value selection unit and a learning processing unit. The processing unit is configured to generate a first distance image including multiple distance values ​​obtained by matching image portions in the left image having an edge strength equal to or greater than a predetermined strength with image portions in the right image having an edge strength equal to or greater than a predetermined strength, of the left image and right image included in the first captured image, which is a stereo image, and to generate a second distance image based on the left image, right image, and first distance image by grouping multiple points whose distances in three-dimensional space are equal to or less than a predetermined distance from each other. The road surface detection processing unit detects the first captured image. Statue Yo And the second Distance image of Multiple distance values ​​in The road surface included in the first captured image is detected based on the above. The three-dimensional object detection unit is configured to detect a three-dimensional object located above a road surface included in the first captured image, based on a plurality of distance values ​​included in the first captured image and the second distance image. The distance value selection unit: Among the multiple distance values ​​included in the second distance image, Road surface detection processing section and the detection process in the three-dimensional object detection processing unit. The learning processing unit is configured to select one or more distance values. an input image which is a clear image of the left image and the right image included in the first captured image; and Selected by the distance value selection unit Distance value greater than or equal to 1 Using By performing machine learning processing, a second captured image is input and a corresponding Third The method is configured to generate a learning model from which a distance image of

[0007] According to the machine learning device according to an embodiment of the present disclosure, it is possible to generate a learning model that generates a highly accurate range image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example configuration of an exterior environment recognition system in which learning data generated by a machine learning device according to an embodiment of the present disclosure is used. [Diagram 2] FIG. 1 is a block diagram illustrating an example configuration of a machine learning device according to an embodiment of the present disclosure. [Diagram 3] 3 is an explanatory diagram illustrating an example of an operation of a road surface detection processing unit illustrated in FIG. 2. [Figure 4] 3 is another explanatory diagram illustrating an operation example of the road surface detection processing unit illustrated in FIG. 2. [Diagram 5] 3 is another explanatory diagram illustrating an operation example of the road surface detection processing unit illustrated in FIG. 2. [Figure 6] FIG. 3 is an explanatory diagram illustrating an example of a configuration of a neural network related to the learning model shown in FIG. 2. [Figure 7] 3 is an image diagram illustrating an example of an operation of the machine learning device illustrated in FIG. 2. [Figure 8] 3 is another image diagram illustrating an example of an operation of the machine learning device shown in FIG. 2. [Figure 9] 3 is another image diagram illustrating an example of an operation of the machine learning device shown in FIG. 2. [Figure 10] 3 is another image diagram illustrating an example of an operation of the machine learning device shown in FIG. 2. [Figure 11] 2 is an image diagram illustrating an example of a captured image in the vehicle exterior environment recognition system illustrated in FIG. 1. [Figure 12]2 is an image diagram showing an example of a distance image according to a reference example, which is generated in the vehicle exterior environment recognition system shown in FIG. 1. [Figure 13] 2 is an image diagram illustrating an example of a distance image generated in the vehicle exterior environment recognition system shown in FIG. 1. [Figure 14] FIG. 13 is a block diagram illustrating an example of the configuration of a machine learning device according to a modified example. [Figure 15] FIG. 13 is a block diagram illustrating an example configuration of a machine learning device according to another modified example. [Figure 16] FIG. 13 is a block diagram illustrating an example configuration of a machine learning device according to another modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment Mode> [Configuration example] 1 illustrates an example of a configuration of an exterior environment recognition system 10 in which processing is performed using a learning model generated by a machine learning device (machine learning device 20) according to an embodiment. The exterior environment recognition system 10 is mounted on a vehicle 100 such as an automobile. The exterior environment recognition system 10 includes a stereo camera 11 and a processing unit 12.

[0011] The stereo camera 11 is configured to capture an image of the front of the vehicle 100, thereby generating a pair of images (a left image PL1 and a right image PR1) having a parallax. The stereo camera 11 has a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are disposed in the vehicle 100 near the upper part of the windshield of the vehicle 100, spaced apart by a predetermined distance in the width direction of the vehicle 100. The left camera 11L generates a left image PL1, and the right camera 11R generates a right image PR1. The left image PL1 and the right image PR1 constitute a stereo image PIC1. The stereo camera 11 performs an imaging operation at a predetermined frame rate (for example, 60 [fps]), thereby generating a series of stereo images PIC1, and supplies the generated stereo images PIC1 to the processing unit 12.

[0012] The processing unit 12 is configured with, for example, one or more processors that execute programs, one or more RAMs (Random Access Memories) that temporarily store processing data, one or more ROMs (Read Only Memories) that store programs, etc. The processing unit 12 has distance image generation units 13 and 14 and an exterior environment recognition unit 15.

[0013] The distance image generating unit 13 is configured to generate a distance image PZ13 by performing a predetermined image processing including a stereo matching process and a filtering process based on the left image PL1 and the right image PR1. Specifically, the distance image generating unit 13 identifies corresponding points including two image points (left image points and right image points) that correspond to each other based on the left image PL1 and the right image PR1. The left image points include, for example, 16 pixels arranged in 4 rows and 4 columns in the left image PL1, and the right image points include, for example, 16 pixels arranged in 4 rows and 4 columns in the right image PR1. The difference between the abscissa value of the left image point in the left image PL1 and the abscissa value of the right image point in the right image PR1 corresponds to a distance value in a three-dimensional real space. The distance image generating unit 13 is configured to generate a distance image PZ13 based on the identified corresponding points. The distance image PZ13 includes a plurality of distance values. Each of the multiple distance values ​​may be an actual distance value in three-dimensional real space, or may be a disparity value which is the difference between the abscissa value of a left image point in the left image PL1 and the abscissa value of a right image point in the right image PR1.

[0014] The distance image generation unit 14 is configured to generate a distance image PZ14 using a learning model M based on a captured image, which is one of the left image PL1 and the right image PR1 in this example. The learning model M is a neural network model to which the captured image is input and which outputs the distance image PZ14. This learning model M is generated in advance by a machine learning device 20, which will be described later, and stored in the distance image generation unit 14 of the vehicle 100. The distance image PZ14 includes a plurality of distance values, similar to the distance image PZ13.

[0015] The exterior environment recognition unit 15 is configured to recognize the exterior environment of the vehicle 100 based on the left image PL1, the right image PR1, and the distance images PZ13 and PZ14. For example, the vehicle 100 is configured to perform driving control of the vehicle 100 based on information about a three-dimensional object outside the vehicle recognized by the exterior environment recognition unit 15, or to display information about the recognized three-dimensional object on a console monitor.

[0016] 2 shows an example of the configuration of a machine learning device 20 that generates a learning model M. The machine learning device 20 is, for example, a server device. The machine learning device 20 includes a storage unit 21 and a processing unit 22.

[0017] The storage unit 21 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 21 stores image data DT and a learning model M.

[0018] The image data DT is image data of a plurality of stereo images PIC2. Each of the plurality of stereo images PIC2 is generated by a stereo camera, similar to the stereo image PIC1 shown in Fig. 1, and stored in the storage unit 21. Each of the plurality of stereo images PIC2 includes a left image PL2 and a right image PR2, similar to the stereo image PIC1 shown in Fig. 1.

[0019] The learning model M is a model used in the distance image generating unit 14 (FIG. 1) of the vehicle 100. This learning model M is generated by the processing unit 22 and stored in the memory unit 21. The learning model M stored in the memory unit 21 is set in the distance image generating unit 14 of the vehicle 100.

[0020] Processing unit 22 is configured with, for example, one or more processors that execute programs, one or more RAMs that temporarily store processed data, etc. Processing unit 22 has an image data acquisition unit 23, a distance image generation unit 24, and an image processing unit 25.

[0021] The image data acquisition section 23 is configured to acquire a plurality of stereo images PIC2 from the storage section 21, and to supply the left image PL2 and the right image PR2 included in each of the plurality of stereo images PIC2 to the distance image generation section 24 in sequence.

[0022] Similar to the distance image generation unit 13 (Figure 1) in the vehicle 100, the distance image generation unit 24 is configured to generate a distance image PZ24 by performing predetermined image processing including stereo matching processing and filtering processing based on the left image PL2 and the right image PR2.

[0023] The image processing unit 25 is configured to perform predetermined image processing based on the left image PL2, the right image PR2, and the distance image PZ24, to generate a learning model M. The image processing unit 25 has an image edge detection unit 31, a grouping processing unit 32, a road surface detection processing unit 33, a three-dimensional object detection processing unit 34, a distance value selection unit 35, an image selection unit 36, and a learning processing unit 37.

[0024] The image edge detection unit 31 is configured to detect image parts with strong edge strength in the left image PL2 and detect image parts with strong edge strength in the right image PR2. Then, the image edge detection unit 31 specifies distance values ​​obtained based on the detected image parts included in the distance image PZ24. That is, since the distance image generation unit 24 performs stereo matching processing based on the left image PL2 and the right image PR2, the distance values ​​obtained based on the image parts with strong edge strength in the left image PL2 and the right image PR2 are expected to be highly accurate. Therefore, the image edge detection unit 31 specifies a plurality of distance values ​​expected to be highly accurate among the plurality of distance values ​​included in the distance image PZ24. Then, the image edge detection unit 31 is configured to generate a distance image PZ31 including the specified plurality of distance values.

[0025] The grouping processor 32 is configured to generate a distance image PZ32 by grouping a plurality of points that are close to each other in a three-dimensional space based on the left image PL2, the right image PR2, and the distance image PZ31. That is, when the distance image generator 24 performs a stereo matching process, depending on the image, there are cases where an incorrect corresponding point is identified due to a mismatch. For example, the distance value related to the mismatch in the distance image PZ31 may deviate from the distance values ​​of the surroundings. The grouping processor 32 is configured to remove such distance values ​​related to the mismatch to some extent by performing a grouping process.

[0026] The road surface detection processing unit 33 is configured to detect the road surface based on the left image PL2, the right image PR2, and the distance image PZ32.

[0027] 3 to 5 show an example of the operation of the road surface detection processing unit 33. First, as shown in FIG. 3, the road surface detection processing unit 33 sets a calculation target area RA based on, for example, one of the left image PL2 and the right image PR2. In this example, the calculation target area RA is an area sandwiched between two partition lines 90L, 90R that divide a lane. Then, as shown in FIG. 3, the road surface detection processing unit 33 sequentially selects horizontal lines HL in the distance image PZ32, and generates a histogram for distance based on the distance values ​​within the calculation target area RA at each horizontal line HL. The histogram H shown in FIG. j is the jth horizontal line from the bottom, HL j The horizontal axis indicates the value of the coordinate z in the vehicle length direction, and the vertical axis indicates the frequency. In this example, j The road surface detection processing unit 33 detects the coordinate value z j Let jth horizontal line HL jThe road surface detection processing unit 33 obtains the representative distances in the plurality of horizontal lines HL in this way. Then, as shown in FIG. 5, the road surface detection processing unit 33 plots these representative distances as distance points D on the zj plane. In this example, a plurality of distance points D are plotted on the zj plane, including a distance point D0 (z0,0) indicating the representative distance of the 0th horizontal line HL0, a distance point D1 (z1,1) indicating the representative distance of the 1st horizontal line HL1, and a distance point D2 (z2,2) indicating the representative distance of the 2nd horizontal line HL2. In this example, these distance points D are arranged on a substantially straight line. The road surface detection processing unit 33 obtains a function indicating the road surface by performing a fitting process, for example, based on these distance points D. In this way, the road surface detection processing unit 33 detects the road surface.

[0028] Furthermore, the road surface detection processing unit 33 supplies information on the multiple distance values ​​adopted in this road surface detection process, among the multiple distance values ​​included in the distance image PZ32, to the distance value selection unit 35. That is, as described above, the road surface detection processing unit 33 detects the road surface based on the representative distance for each of the multiple horizontal lines HL. Therefore, for each of the multiple horizontal lines HL, the multiple distance values ​​constituting the representative distance are adopted in the road surface detection process, and the multiple distance values ​​not constituting the representative distance are not adopted in the road surface detection process. The road surface detection processing unit 33 supplies information on the multiple distance values ​​adopted in this road surface detection process to the distance value selection unit 35.

[0029] The three-dimensional object detection processing unit 34 is configured to detect a three-dimensional object based on the left image PL2, the right image PR2, and the distance image PZ32. The three-dimensional object detection processing unit 34 detects a three-dimensional object by grouping a plurality of points that are close to each other in a three-dimensional space above the road surface obtained by the road surface detection processing unit 33. Specifically, the three-dimensional object detection processing unit 34 can detect a three-dimensional object by grouping a plurality of points that are within a distance of, for example, 0.1 m in a three-dimensional space.

[0030] In addition, the three-dimensional object detection processing unit 34 supplies information on the multiple distance values ​​adopted in this three-dimensional object detection process among the multiple distance values ​​included in the distance image PZ32 to the distance value selection unit 35. As described above, the three-dimensional object detection processing unit 34 detects a three-dimensional object by grouping multiple points that are close to each other in the three-dimensional space above the road surface. Therefore, a desired distance value near the three-dimensional object is adopted in the three-dimensional object detection process, and for example, as described later, a distance value related to a mismatch near the three-dimensional object or a distance value related to a specular reflection when the road surface is wet is not adopted in the three-dimensional object detection process. The three-dimensional object detection processing unit 34 supplies information on the multiple distance values ​​adopted in this three-dimensional object detection process to the distance value selection unit 35.

[0031] The distance value selection unit 35 is configured to select a plurality of distance values ​​to be supplied to the learning processing unit 37 from among the plurality of distance values ​​included in the distance image PZ32 supplied from the grouping processing unit 32. The distance value selection unit 35 can select, for example, a plurality of distance values ​​used in the road surface detection process from among the plurality of distance values ​​included in the distance image PZ32, as a plurality of distance values ​​to be supplied to the learning processing unit 37. The distance value selection unit 35 can also select, for example, a plurality of distance values ​​used in the three-dimensional object detection process from among the plurality of distance values ​​included in the distance image PZ32, as a plurality of distance values ​​to be supplied to the learning processing unit 37. The distance value selection unit 35 can also select, for example, a plurality of distance values ​​used in the three-dimensional object detection process and the road surface detection process from among the plurality of distance values ​​included in the distance image PZ32, as a plurality of distance values ​​to be supplied to the learning processing unit 37. The distance value selection unit 35 is then configured to supply the distance image PZ35 including the plurality of selected distance values ​​to the learning processing unit 37.

[0032] The image selection unit 36 ​​is configured to supply the captured image P2, which is one of the left image PL2 and the right image PR2, to the learning processing unit 37. The image selection unit 36 ​​is configured to select, for example, a clear image from the left image PL2 and the right image PR2 as the captured image P2.

[0033] The learning processing unit 37 is configured to perform machine learning processing using a neural network based on the captured image P2 and the distance image PZ35, thereby generating a learning model M. The captured image P2 is supplied to the learning processing unit 37, and the distance image PZ35 is supplied as an expected value. The learning processing unit 37 performs machine learning processing based on these images, thereby generating a learning model M to which the captured image is input and a distance image is output.

[0034] FIG. 6 shows an example of a configuration of a neural network. In this example, a captured image is input from the left of FIG. 6, and a distance image is output from the right of FIG. 6. In this neural network, for example, a compression process A1 is performed based on the captured image, and a convolution process A2 is performed based on the compressed data. In the neural network, the compression process A1 and the convolution process A2 are repeated multiple times. Then, an upsampling process B1 is performed based on the generated data, and a convolution process B2 is performed based on the data that has been subjected to the upsampling process B1. In the neural network, the upsampling process B1 and the convolution process B2 are repeated multiple times. In the convolution processes A2 and B2, a filter of a predetermined size (for example, 3 pixels×3 pixels) is used.

[0035] The learning processing unit 37 inputs the captured image P2 to the neural network and calculates the difference values ​​between the multiple distance values ​​in the output distance image and the multiple distance values ​​in the distance image PZ35, which are expected values. The learning processing unit 37 then adjusts the filter values ​​used in the convolution processes A2 and B2, for example, so that these difference values ​​become sufficiently small. In this manner, the learning processing unit 37 performs machine learning processing.

[0036] The learning processing unit 37 can set, for example, whether or not to perform learning processing for each image region. Specifically, the learning processing unit 37 can perform machine learning processing for an image region to which a distance value is supplied from the distance value selection unit 35, and can not perform machine learning processing for an image region to which a distance value is not supplied from the distance value selection unit 35. For example, the learning processing unit 37 can forcibly set the difference value of the distance value in an image region to which a distance value is not supplied from the distance value selection unit 35 to "0", thereby not performing machine learning processing for this image region.

[0037] For example, in the neural network shown in Fig. 6, if the number of layers is larger, it can become a learning model with a global view. By inputting a blurred captured image to such a neural network and performing machine learning processing, it is possible to generate a learning model M that can obtain more distance values ​​based on a captured image with less texture, for example.

[0038] Where: The image edge detection unit 31 and the grouping processing unit 32 correspond to a specific but not limitative example of a "processing unit" in the disclosure. The road surface detection processing unit 33 corresponds to a specific example of a "road surface detection processing unit" in this disclosure. The three-dimensional object detection processing unit 34 corresponds to a specific example of a "three-dimensional object detection processing unit" in this disclosure. The distance value selection unit 35 corresponds to a specific example of a "distance value selection unit" in this disclosure. The learning processing unit 37 corresponds to a specific example of a "learning processing unit" in this disclosure. The stereo image PIC2 corresponds to a specific example of a "first captured image" in this disclosure. The distance image PZ 31 corresponds to a specific example of a "first distance image" in this disclosure. The distance image PZ32 corresponds to a specific but not limitative example of a "second distance image" in this disclosure.

[0039] [Actions and Actions] Next, the operation and function of the machine learning device 20 and the exterior environment recognition system 10 of this embodiment will be described.

[0040] (Overview of overall operation) First, the operation of the machine learning device 20 will be described with reference to FIG. 2. The machine learning device 20 stores image data DT including a plurality of stereo images PIC2 generated by, for example, a stereo camera in the storage unit 21. The image data acquisition unit 23 of the processing unit 22 acquires the plurality of stereo images PIC2 from the storage unit 21, and sequentially supplies the left image PL2 and the right image PR2 included in each of the plurality of stereo images PIC2 to the distance image generation unit 24. The distance image generation unit 24 generates a distance image PZ24 by performing a predetermined image processing including a stereo matching process and a filtering process based on the left image PL2 and the right image PR2. The image edge detection unit 31 of the image processing unit 25 detects an image portion with a strong edge strength in the left image PL2 and detects an image portion with a strong edge strength in the right image PR2. Then, the image edge detection unit 31 identifies a distance value obtained based on the detected image portion included in the distance image PZ24, and generates a distance image PZ31 including the identified plurality of distance values. The grouping processing unit 32 generates a distance image PZ32 by grouping points that are close to each other in a three-dimensional space based on the left image PL2, the right image PR2, and the distance image PZ31. The road surface detection processing unit 33 detects a road surface based on the left image PL2, the right image PR2, and the distance image PZ32. The road surface detection processing unit 33 also supplies information on a plurality of distance values ​​adopted in this road surface detection processing among a plurality of distance values ​​included in the distance image PZ32 to the distance value selection unit 35. The three-dimensional object detection processing unit 34 detects a three-dimensional object based on the left image PL2, the right image PR2, and the distance image PZ32. The three-dimensional object detection processing unit 34 also supplies information on a plurality of distance values ​​adopted in this three-dimensional object detection processing among a plurality of distance values ​​included in the distance image PZ32 to the distance value selection unit 35. The distance value selection unit 35 selects a plurality of distance values ​​to be supplied to the learning processing unit 37 among a plurality of distance values ​​included in the distance image PZ32 supplied from the grouping processing unit 32. The image selection unit 36 ​​supplies the learning processing unit 37 with the captured image P2, which is one of the left image PL2 and the right image PR2.The learning processing unit 37 performs machine learning processing using a neural network based on the captured image P2 and the distance image PZ35 to generate a learning model M. Then, the processing unit 22 stores this learning model M in the memory unit 21. Then, the learning model M generated in this manner is set in the distance image generating unit 14 of the exterior environment recognition system 10.

[0041] Next, the operation of the vehicle exterior environment recognition system 10 will be described with reference to FIG. 1. The stereo camera 11 captures an image in front of the vehicle 100 to generate a left image PL1 and a right image PR1 having a parallax. The distance image generation unit 13 of the processing unit 12 performs predetermined image processing, including stereo matching processing and filtering processing, based on the left image PL1 and the right image PR1 to generate a distance image PZ13. The distance image generation unit 14 generates a distance image PZ14 using the learning model M generated by the machine learning device 20 based on the captured image, which is one of the left image PL1 and the right image PR1 in this example. The vehicle exterior environment recognition unit 15 recognizes the vehicle exterior environment of the vehicle 100 based on the left image PL1, the right image PR1, and the distance images PZ13 and PZ14.

[0042] (Details of operation) Next, the operation of the image processing unit 25 (FIG. 2) in the machine learning device 20 will be described in detail.

[0043] First, the image edge detection unit 31 detects image parts with strong edge strength in the left image PL2 and detects image parts with strong edge strength in the right image PR2. Then, the image edge detection unit 31 specifies distance values ​​obtained based on the detected image parts included in the distance image PZ24. That is, since the distance image generation unit 24 performs stereo matching processing based on the left image PL2 and the right image PR2, the distance values ​​obtained based on the image parts with strong edge strength in the left image PL2 and the right image PR2 are expected to be highly accurate. Therefore, the image edge detection unit 31 specifies multiple distance values ​​expected to be highly accurate among the multiple distance values ​​included in the distance image PZ24. Then, the image edge detection unit 31 generates a distance image PZ31 including the specified multiple distance values.

[0044] FIG. 7 shows an example of a distance image PZ31. In FIG. 7, the shading indicates the parts having distance values. The shade of the shading indicates the density of the distance values. That is, the density of the obtained distance values ​​is low in the lightly shaded parts, and the density of the obtained distance values ​​is high in the darkly shaded parts. For example, the road surface has little texture, and it is difficult to detect corresponding points in stereo matching, so the density of the distance values ​​is low. On the other hand, for example, the lane markings on the road surface and three-dimensional objects such as vehicles are easy to detect corresponding points in stereo matching, so the density of the distance values ​​is high.

[0045] Next, the grouping processor 32 generates a distance image PZ32 by grouping a plurality of points that are close to each other in three-dimensional space, based on the left image PL2, the right image PR2, and the distance image PZ31.

[0046] FIG. 8 shows an example of a distance image PZ32. In this distance image PZ32, distance values ​​are removed from areas where the density of the obtained distance values ​​is low, for example, compared to the distance image PZ31 shown in FIG. 7. When the distance image generating unit 24 performs stereo matching processing, there is a possibility that an erroneous corresponding point is identified due to a mismatch depending on the image. For example, in an area with little texture, such as a road surface, there are few corresponding points, and there are many corresponding points related to such mismatches. The distance value related to the mismatch may deviate from the distance values ​​around it. The grouping processing unit 32 can remove such distance values ​​related to the mismatch to some extent by performing grouping processing.

[0047] In Fig. 8, the road surface is wet due to rain, for example, and specular reflection occurs on the road surface. Part W1 shows an image of the tail lamp of the leading vehicle 9 reflected on the road surface. The distance value in part W1 may correspond to the distance from the vehicle to the leading vehicle 9. However, this image itself occurs on the road surface. Such a virtual image may be included in the distance image PZ32.

[0048] Next, the road surface detection processing unit 33 detects the road surface based on the left image PL2, the right image PR2, and the distance image PZ32. The road surface detection processing unit 33 also supplies information on the multiple distance values ​​adopted in this road surface detection process, among the multiple distance values ​​included in the distance image PZ32, to the distance value selection unit 35.

[0049] Fig. 9 shows a distance image showing the distance values ​​adopted in the road surface detection process among the distance values ​​included in the distance image PZ32. As shown in Fig. 9, each of the distance values ​​adopted in the road surface detection process is located in a portion corresponding to the road surface. That is, each of the distance values ​​indicates the distance from the vehicle to the road surface.

[0050] In this distance image, as shown in part W2, the distance value caused by the virtual image of the specular reflection is removed. That is, as described above, the distance value in part W1 of FIG. 8 may correspond to the distance from the vehicle to the preceding vehicle 9. However, in the histograms for each of the horizontal lines HL in the road surface detection process, the frequency of this distance value is low, so this distance value is unlikely to become the representative distance. As a result, this distance value is not adopted in the road surface detection process, and is therefore removed in the distance image shown in FIG.

[0051] Thus, in the distance image (FIG. 9) showing the multiple distance values ​​employed in the road surface detection process, noise in the distance values ​​is reduced compared to the distance image PZ32 shown in FIG.

[0052] Next, the three-dimensional object detection processing unit 34 detects a three-dimensional object based on the left image PL2, the right image PR2, and the distance image PZ32. The three-dimensional object detection processing unit 34 also supplies information on a plurality of distance values ​​adopted in this three-dimensional object detection processing, among the plurality of distance values ​​included in the distance image PZ32, to the distance value selection unit 35.

[0053] Fig. 10 shows a distance image showing the multiple distance values ​​adopted in the three-dimensional object detection process among the multiple distance values ​​included in the distance image PZ32. As shown in Fig. 10, each of the multiple distance values ​​adopted in the three-dimensional object detection process is located in a portion corresponding to these three-dimensional objects. In other words, each of the multiple distance values ​​indicates the distance from the vehicle to the three-dimensional object located above the road surface.

[0054] The three-dimensional object detection processing unit 34 detects a three-dimensional object by grouping a plurality of points that are close to each other in three-dimensional space above the road surface. Distance values ​​related to mismatches near a three-dimensional object may deviate from distance values ​​around it. Therefore, the three-dimensional object detection processing unit 34 can remove distance values ​​related to mismatches on the side of the vehicle, walls, etc.

[0055] In this distance image, the distance value caused by the virtual image of the specular reflection is also removed, as shown in part W3. That is, as described above, the distance value in part W1 of FIG. 8 may correspond to the distance from the vehicle to the preceding vehicle 9. However, this image itself is generated on the road surface. Therefore, the position in the three-dimensional space obtained based on this image is below the road surface. The three-dimensional object detection processing unit 34 detects a three-dimensional object based on the image above the road surface. As a result, this distance value is not adopted in the three-dimensional object detection process, and is therefore removed in the distance image shown in FIG. 10.

[0056] Thus, in the distance image (FIG. 10) showing a plurality of distance values ​​employed in the three-dimensional object detection process, noise in the distance values ​​is reduced compared to the distance image PZ32 shown in FIG.

[0057] The distance value selection unit 35 selects a plurality of distance values ​​to be supplied to the learning processing unit 37 from among the plurality of distance values ​​included in the distance image PZ32 supplied from the grouping processing unit 32. The distance value selection unit 35 can select, for example, a plurality of distance values ​​used in the road surface detection process from among the plurality of distance values ​​included in the distance image PZ32 as a plurality of distance values ​​to be supplied to the learning processing unit 37. The distance value selection unit 35 can also select, for example, a plurality of distance values ​​used in the three-dimensional object detection process from among the plurality of distance values ​​included in the distance image PZ32 as a plurality of distance values ​​to be supplied to the learning processing unit 37. The distance value selection unit 35 can also select, for example, a plurality of distance values ​​used in the three-dimensional object detection process and the road surface detection process from among the plurality of distance values ​​included in the distance image PZ32 as a plurality of distance values ​​to be supplied to the learning processing unit 37. Then, the distance value selection unit 35 supplies the learning processing unit 37 with the distance image PZ35 including the selected plurality of distance values. In this way, the learning processing unit 37 is supplied with the distance image PZ35 in which the noise of the distance values ​​is reduced.

[0058] The image selection unit 36 ​​supplies the captured image P2, which is one of the left image PL2 and the right image PR2, to the learning processing unit 37. The learning processing unit 37 then performs machine learning processing using a neural network based on the captured image P2 and the distance image PZ35 to generate a learning model M. The captured image P2 is supplied to the learning processing unit 37, and the distance image PZ35 is supplied as an expected value. The learning processing unit 37 is supplied with the distance image PZ35 in which noise in the distance values ​​has been reduced, so that a learning model M with high accuracy can be generated.

[0059] Next, the distance image PZ14 generated by the distance image generating unit 14 of the exterior environment recognition system 10 using the learning model M generated in this manner will be described.

[0060] Fig. 11 shows an example of a captured image generated by the stereo camera 11 in the vehicle exterior environment recognition system 10. In Fig. 11, the road surface is wet due to rain, for example, and specular reflection occurs on the road surface. Part W4 shows the image of a utility pole reflected by the road surface.

[0061] 12 and 13 show an example of a distance image PZ14 generated by the distance image generating unit 14 using a learning model M based on the captured image shown in FIG. 11. FIG. 12 shows a case where the learning model M is generated in the machine learning device 20 based on all of the distance values ​​included in the distance image PZ32. FIG. 13 shows a case where the learning model M is generated in the machine learning device 20 based on the distance values ​​used in the three-dimensional object detection process and the road surface detection process, among the distance values ​​included in the distance image PZ32. In FIGS. 12 and 13, the shading indicates the distance value. Light shading indicates a small distance value, and dark shading indicates a large distance value.

[0062] In the example of Fig. 12, as shown in part W5, the effect of a virtual image caused by specular reflection causes a disturbance in the distance value. The distance to the road surface on which the utility pole is reflected is close, but the distance to the actual utility pole is far, so the distance value in part W5 is large as shown in Fig. 12. In this way, distance image generator 14 outputs the distance value as is based on the input captured image.

[0063] 12, the learning model M is generated in the machine learning device 20 based on all of the distance values ​​included in the distance image PZ32. That is, the learning model M is trained using, for example, a captured image including an image portion of specular reflection and a distance image including an erroneous distance value due to specular reflection (e.g., FIG. 8). Therefore, as shown in FIG. 11, when the input captured image includes an image portion of specular reflection such as portion W4, the distance image generating unit 14 outputs a distance value corresponding to that image portion as shown in FIG. 12.

[0064] On the other hand, in the example of FIG. 13, the disturbance of the distance value as seen in FIG. 12 does not occur. In the example of FIG. 13, the learning model M is generated in the machine learning device 20 based on a plurality of distance values ​​used in the three-dimensional object detection process and the road surface detection process among a plurality of distance values ​​included in the distance image PZ32. That is, the learning model M is learned using, for example, an image including specular reflection and a distance image (for example, FIG. 9, 10) that does not include an erroneous distance value due to specular reflection. That is, the erroneous distance value due to specular reflection is not used in the machine learning process. The machine learning process is performed using stereo images PIC2 in various situations such as various weather conditions and various time periods. These stereo images PIC2 include, for example, images without specular reflection. Therefore, even if the input captured image (FIG. 11) includes an image portion with specular reflection such as the portion W4, the distance image generating unit 14 can reflect the learning under such various conditions and output the distance value in the case where there is no specular reflection as shown in FIG. 13.

[0065] In this way, the machine learning device 20 includes a road surface detection processing unit 33 that detects a road surface included in the first captured image (stereo image PIC2) based on the first captured image (stereo image PIC2) and a first distance image (distance image PZ32) corresponding to the first captured image (stereo image PIC2), a distance value selection unit 35 that selects one or more distance values ​​to be processed among the multiple distance values ​​included in the first distance image (distance image PZ32) based on the processing result of the road surface detection processing unit 33, and a learning processing unit 37 that performs machine learning processing based on the first captured image (stereo image PIC2) and one or more distance values ​​to generate a learning model M in which a second captured image is input and a second distance image corresponding to the second captured image is output. As a result, the machine learning device 20 can perform machine learning processing based on one or more distance values ​​selected based on the processing result of the road surface detection processing unit 33 among the multiple distance values ​​included in the distance image PZ32. For example, the distance value selection unit 35 of the machine learning device 20 can select the distance value (FIG. 9) employed in the road surface detection process as a distance value equal to or greater than 1, and can select the distance value (FIG. 10) employed in the three-dimensional object detection process for detecting a three-dimensional object on the road surface as a distance value equal to or greater than 1. This allows the machine learning device 20 to generate a learning model M that generates a highly accurate distance image.

[0066] When generating such a learning model M, machine learning may be performed using a distance image obtained by using, for example, a Lidar (Light detection and ranging) device and a captured image. However, since an image sensor that generates a captured image and a Lidar device that generates a distance image have different characteristics, for example, it may happen that a distance value is obtained in the distance image although it is not captured in the captured image. When such a contradiction occurs, it is difficult to perform machine learning processing.

[0067] 2, the machine learning device 20 generates distance images PZ24, PZ31, and PZ32 based on the stereo image PIC2, ​​making it difficult for the above-mentioned inconsistencies to occur, and thus facilitating machine learning processing. As a result, the machine learning device 20 can improve the accuracy of the learning model.

[0068] In addition, even when machine learning processing is performed using the distance image PZ24 generated based on the stereo image PIC2 generated by the stereo camera, as described above, mismatches occur or virtual images such as specular reflection occur, so the distance image PZ24 contains inaccurate distance values. Therefore, it is difficult to improve the accuracy of the learning model. In addition, although it is possible to select accurate distance values ​​and inaccurate distance values ​​from the distance image PZ24, it is unrealistic for a person to perform this selection, for example.

[0069] Meanwhile, the machine learning device 20 selects one or more distance values ​​to be processed from among the multiple distance values ​​included in the first distance image (distance image PZ32) based on the processing result of the road surface detection processing unit 33, and performs machine learning processing based on the first captured image (stereo image PIC2) and the one or more distance values. This allows the machine learning device 20 to reduce the influence of mismatches and specular reflections, and to select accurate distance values ​​without requiring manual annotation work. As a result, the machine learning device 20 can improve the accuracy of the learning model.

[0070] In the example shown in FIG. 2, the machine learning device 20 generates distance images PZ24, PZ31, and PZ32 by stereo matching. When stereo matching is performed in this manner, highly accurate distance values ​​can be obtained. However, since matching occurs locally, the density of distance values ​​may be low. Even in such cases, by using the learning model M generated by the machine learning device 20, highly accurate and dense distance values ​​can be obtained over the entire area.

[0071] In the machine learning device 20, the learning processing unit 37 calculates one or more distances in the entire image area of ​​the first captured image (stereo image PIC2) based on one or more distance values. value In this way, the learning processing unit 37 can perform machine learning processing for the image area to which the distance value is supplied from the distance value selection unit 35, and can not perform machine learning processing for the image area to which the distance value is not supplied from the distance value selection unit 35. As a result, for example, it is possible to not perform machine learning processing based on an erroneous distance value due to specular reflection, and therefore it is possible to improve the accuracy of the learning model.

[0072] [effect] As described above, in this embodiment, there is provided a road surface detection processing unit that detects the road surface contained in the first captured image based on the first captured image and a first distance image corresponding to the first captured image, a distance value selection unit that selects one or more distance values ​​to be processed from the multiple distance values ​​contained in the first distance image based on the processing result of the road surface detection processing unit, and a learning processing unit that performs machine learning processing based on the first captured image and the one or more distance values ​​to generate a learning model in which a second captured image is input and a second distance image corresponding to the second captured image is output, thereby making it possible to generate a learning model that generates distance images with high accuracy.

[0073] In this embodiment, one or more distance values ​​are used to calculate one or more distances in the entire image area of ​​the first captured image. value Since machine learning processing is performed on the image region corresponding to the image, the accuracy of the learning model can be improved.

[0074] [Variation 1] In the above embodiment, the machine learning device 20 performs the machine learning process based on the distance image PZ24 generated based on the stereo image PIC2, ​​but the present invention is not limited to this. Below, the present modified example will be described in detail with some examples.

[0075] 14 shows an example of the configuration of a machine learning device 40 according to this modification. The machine learning device 40 is configured to perform machine learning processing based on a range image obtained by a Lidar device. The machine learning device 40 includes a storage unit 41 and a processing unit 42.

[0076] The storage unit 41 stores image data DT3 and distance image data DT4. In this example, the image data DT3 is image data of a plurality of captured images PIC3. Each of the captured images PIC3 is a monocular image, generated by a monocular camera, and stored in the storage unit 41. The distance image data DT4 is image data of a plurality of distance images PZ4. The plurality of distance images PZ4 respectively correspond to the plurality of captured images PIC3. In this example, the distance images PZ4 are generated by a Lidar device and stored in the storage unit 41.

[0077] The processing unit 42 includes a data acquisition unit 43 and an image processing unit 45 .

[0078] The data acquisition section 43 is configured to acquire a plurality of captured images PIC3 and a plurality of distance images PZ4 from the storage section 41, and to supply the captured images PIC3 and distance images PZ4 corresponding to each other to the image processing section 45 in sequence.

[0079] The image processing unit 45 is configured to perform a predetermined image processing based on the captured image PIC3 and the distance image PZ4, thereby generating a learning model M. The image processing unit 45 has an image edge detection unit 51, a grouping processing unit 52, a road surface detection processing unit 53, a three-dimensional object detection processing unit 54, a distance value selection unit 55, and a learning processing unit 57. The image edge detection unit 51, the grouping processing unit 52, the road surface detection processing unit 53, the three-dimensional object detection processing unit 54, the distance value selection unit 55, and the learning processing unit 57 correspond to the image edge detection unit 31, the grouping processing unit 32, the road surface detection processing unit 33, the three-dimensional object detection processing unit 34, the distance value selection unit 35, and the learning processing unit 37, respectively, according to the above embodiment.

[0080] The learning processing unit 57 is configured to perform machine learning processing using a neural network based on the captured image PIC3 and the distance image PZ35, thereby generating a learning model M. The captured image PIC3 is supplied to the learning processing unit 57, and the distance image PZ35 is supplied as an expected value. The learning processing unit 57 performs machine learning processing based on these images, thereby generating a learning model M in which the captured image is input and a distance image is output. Here, the captured image PIC3 corresponds to a specific example of a "first captured image" in this disclosure.

[0081] For example, the distance image generation unit 14 of the exterior environment recognition system 10 shown in Figure 1 can use the learning model M generated by such a machine learning device 40 to generate a distance image PZ14 based on a captured image, which is one of the left image PL1 and the right image PR1.

[0082] 15 shows an example of the configuration of another machine learning device 60 according to this modification. The machine learning device 60 is configured to perform machine learning processing based on a distance image obtained by a motion stereo technique. The machine learning device 60 includes a storage unit 61 and a processing unit 62.

[0083] The storage unit 61 stores image data DT3. In this example, the image data DT3 is image data of a series of captured images PIC3. Each of the captured images PIC3 is a monocular image generated by a monocular camera and stored in the storage unit 61.

[0084] The processing unit 62 has an image data acquisition unit 63 , a distance image generation unit 64 , and an image processing unit 65 .

[0085] The image data acquisition section 63 is configured to acquire a series of captured images PIC3 from the storage section 61, and supply the captured images PIC3 to the distance image generation section 64 in sequence.

[0086] The distance image generating unit 64 is configured to generate a distance image PZ24 by a motion stereo technique based on two captured images PIC3 that are adjacent to each other on the time axis out of a series of a plurality of captured images PIC3.

[0087] The image processing unit 65 is configured to perform a predetermined image processing based on the captured image PIC3 and the distance image PZ24, thereby generating a learning model M. The image processing unit 65 has an image edge detection unit 71, a grouping processing unit 72, a road surface detection processing unit 73, a three-dimensional object detection processing unit 74, a distance value selection unit 75, and a learning processing unit 77. The image edge detection unit 71, the grouping processing unit 72, the road surface detection processing unit 73, the three-dimensional object detection processing unit 74, the distance value selection unit 75, and the learning processing unit 77 correspond to the image edge detection unit 31, the grouping processing unit 32, the road surface detection processing unit 33, the three-dimensional object detection processing unit 34, the distance value selection unit 35, and the learning processing unit 37, respectively, according to the above embodiment.

[0088] The learning processing unit 77 is configured to perform machine learning processing using a neural network based on the captured image PIC3 and the distance image PZ35, thereby generating a learning model M. The captured image PIC3 is supplied to the learning processing unit 77, and the distance image PZ35 is supplied as an expected value. The learning processing unit 77 performs machine learning processing based on these images, thereby generating a learning model M to which the captured image is input and a distance image is output.

[0089] For example, the distance image generation unit 14 of the exterior environment recognition system 10 shown in Figure 1 can use the learning model M generated by such a machine learning device 60 to generate a distance image PZ14 based on a captured image, which is one of the left image PL1 and the right image PR1.

[0090] [Variation 2] In the above embodiment, the learning model M is configured to input a captured image and output a distance image, but the input image is not limited to this, and for example, a stereo image may be input. In addition, in the case of motion stereo, two captured images adjacent to each other on the time axis may be input. The case where a stereo image is input will be described in detail below.

[0091] 16 shows an example of the configuration of a machine learning device 20B according to this modification. The machine learning device 20B includes a processing unit 22B. The processing unit 22B includes an image processing unit 25B. The image processing unit 25B includes an image edge detection unit 31, a grouping processing unit 32, a road surface detection processing unit 33, a three-dimensional object detection processing unit 34, a distance value selection unit 35, and a learning processing unit 37B.

[0092] The learning processing unit 37B is configured to perform machine learning processing using a neural network based on the stereo image PIC2 and the distance image PZ35, thereby generating a learning model M. The learning processing unit 37B is supplied with the stereo image PIC2 and the distance image PZ35 as an expected value. The learning processing unit 37B performs machine learning processing based on these images, thereby generating a learning model M in which the stereo image is input and a distance image is output.

[0093] For example, the distance image generating unit 14 of the vehicle exterior environment recognition system 10 can generate a distance image PZ14 based on the stereo image PIC, using the learning model M generated by such a machine learning device 20B.

[0094] Although the present technology has been described above by giving embodiments and some modified examples, the present technology is not limited to these embodiments and the like, and various modifications are possible.

[0095] For example, in the above-mentioned embodiment, the image processing unit 25 is provided with an image edge detection unit 31, a grouping processing unit 32, a road surface detection processing unit 33, and a three-dimensional object detection processing unit 34, but this is not limited to this, and for example, some of these may be omitted or other blocks may be added.

[0096] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0097] The present technology can be configured as follows.

[0098] (1) a road surface detection processing unit that detects a road surface included in a first captured image based on a first captured image and a first distance image corresponding to the first captured image; a distance value selection unit that selects, based on a processing result of the road surface detection processing unit, one or more distance values ​​to be processed from among a plurality of distance values ​​included in the first distance image; a learning processing unit that performs machine learning processing based on the first captured image and the one or more distance values ​​to generate a learning model in which a second captured image is input and a second distance image corresponding to the second captured image is output; A machine learning device equipped with (2) The distance value selection unit selects, as the one or more distance values, a distance value adopted in the detection process in the road surface detection processing unit, from among a plurality of distance values ​​included in the first distance image. The machine learning device according to claim 1. (3) The one or more distance values ​​include a distance value to the road surface included in the first captured image. The machine learning device according to claim 2. (4) a three-dimensional object detection processing unit that detects a three-dimensional object located above the road surface included in the first captured image, The distance value selection unit selects, as the one or more distance values, a distance value adopted in the detection process in the three-dimensional object detection processing unit, from among a plurality of distance values ​​included in the first distance image. The machine learning device according to any one of claims 1 to 3. (5) The one or more distance values ​​include a distance value to a three-dimensional object located above the road surface included in the first captured image. The machine learning device according to claim 4. (6) The learning processing unit performs the machine learning processing on an image region corresponding to the one or more distances in an entire image region of the first captured image, based on the one or more distance values. The machine learning device according to claim 1. (7) One or more processors one or more memories communicatively coupled to the one or more processors; Equipped with The one or more processors: performing a road surface detection process for detecting a road surface included in a first captured image based on a first captured image and a first distance image corresponding to the first captured image; selecting one or more distance values ​​to be processed from among a plurality of distance values ​​included in the first distance image based on a processing result of the road surface detection processing; performing machine learning processing based on the first captured image and the one or more distance values ​​to generate a learning model in which a second captured image is input and a second distance image corresponding to the second captured image is output; Do Machine learning device. [Explanation of symbols]

[0099] 10...Exterior environment recognition system, 11...Stereo camera, 11L...Left camera, 11R...Right camera, 12...Processing unit, 13,14...Distance image generation unit, 15...Exterior environment recognition unit, 20,20B,40,60...Machine learning device, 21,41,61...Memory unit, 22,22B,42,62...Processing unit, 23,63...Image data acquisition unit, 24,64...Distance image generation unit, 25,25B,45,65...Image processing unit, 31,51,71...Image edge detection unit, 32,52,72...Grouping processing unit, 33,53,73...Road surface detection processing unit, 34,54,74...Three-dimensional object detection processing unit, 35 , 55,75...distance value selection unit, 36...image selection unit, 37,37B,57,77...learning processing unit, 43...data acquisition unit, 100...vehicle, A1...compression processing, A2...convolution processing, B1...upsampling processing, B2...convolution processing, DT,DT3...image data, DT4...distance image data, M...learning model, P2...captured image, PIC,PIC1,PIC2...stereo image, PIC3...captured image, PL1,PL2...left image, PR1,PR2...right image, PZ4,PZ13,PZ14,PZ24,PZ31,PZ32,PZ35...distance image, RA...area to be calculated.

Claims

1. A processing unit that generates a first distance image including a plurality of distance values ​​obtained by matching an image portion in the left image having an edge strength equal to or greater than a predetermined strength with an image portion in the right image having an edge strength equal to or greater than a predetermined strength, of a left image and a right image included in a first captured image which is a stereo image, and generates a second distance image by grouping a plurality of points whose distances in three-dimensional space are equal to or less than a predetermined distance from each other based on the left image, the right image, and the first distance image; a road surface detection processing unit that detects a road surface included in the first captured image based on a plurality of distance values ​​included in the first captured image and the second distance image; a three-dimensional object detection processing unit that detects a three-dimensional object located above the road surface included in the first captured image based on a plurality of distance values ​​included in the first captured image and the second distance image; a distance value selection unit that selects, from among a plurality of distance values ​​included in the second distance image, one or more distance values ​​that have been adopted in at least one of the detection processing in the road surface detection processing unit and the detection processing in the three-dimensional object detection processing unit; a learning processing unit that performs machine learning processing using an input image that is a clear image of the left image and the right image included in the first captured image, and the one or more distance values ​​selected by the distance value selection unit, to generate a learning model in which a second captured image is input and a third distance image corresponding to the second captured image is output; A machine learning device equipped with

2. The one or more distance values ​​include a distance value to the road surface included in the first captured image. The machine learning device according to claim 1 .

3. The one or more distance values ​​include a distance value to a three-dimensional object located above the road surface included in the first captured image. The machine learning device according to claim 1 .

4. The learning processing unit performs the machine learning processing on an image region corresponding to the one or more distances in an entire image region of the input image based on the one or more distance values. The machine learning device according to claim 1 .

Images