Image Processing Device

Abstract

To provide an image processing apparatus capable of reducing an amount of computation.SOLUTION: An image processing apparatus 1 comprises: a feature extraction section 43 which is a first extraction section for extracting a first feature quantity included in a captured image; an object identification section 45 which is a first object identification section for identifying an object on the basis of the first feature quantity; a region setting section 32 which sets an image region in the captured image; a feature extraction section 33 which is a second extraction section for extracting a second feature quantity included in an image in the image region; a feature quantity selection section 22 which selects a part of the first feature quantity extracted by the feature extraction section 43 on the basis of information on the image region set by the region setting section 32; and a second object identification section 35 which identifies the object on the basis of the feature quantity selected by the feature quantity selection section 22 from the second feature quantity and the first feature quantity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device that identifies an object based on an image captured by an imaging device. [Background technology]

[0002] Images captured by an imaging device include images of various objects. Patent Document 1 discloses an image recognition device that distinguishes objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-165008 A Summary of the Invention [Problem to be solved by the invention]

[0004] In an image processing device that identifies an object, it is desirable to reduce the amount of calculation while improving the identification accuracy, and further reduction in the amount of calculation is anticipated.

[0005] It is desirable to provide an image processing device that can reduce the amount of calculations. [Means for solving the problem]

[0006] A first image processing device according to an embodiment of the present disclosure includes a first extraction unit, a first object identification unit, a distance image generation unit, a region setting unit, a second extraction unit, a selection unit, and a second object identification unit. The first extraction unit is configured to extract a first feature amount included in a captured image. The first object identification unit is configured to identify an object based on the first feature amount. The region setting unit is configured to set an image region in the captured image. The second extraction unit is configured to extract a second feature amount included in an image of the image region. The selection unit is configured to select a part of the first feature amount extracted by the first extraction unit based on information about the image region set by the region setting unit. The second object identification unit is configured to identify an object based on the second feature amount and a feature amount selected by the selection unit from among the first feature amounts. The captured image includes a left image and a right image. The distance image generation unit is configured to generate a distance image based on the left image and the right image. The region setting unit selects, based on the distance image, By identifying pixels in the range image that are located close to each other and have approximately the same disparity value, The selecting unit selects, from among the first feature amounts, a feature amount of a region that corresponds to the image region, based on a position of the image region in the captured image. A second image processing device according to an embodiment of the present disclosure includes a first extraction unit, a first object identification unit, a distance image generation unit, a region setting unit, a second extraction unit, a selection unit, and a second object identification unit. The first extraction unit is configured to extract a first feature amount included in a captured image. The first object identification unit is configured to identify an object based on the first feature amount. The region setting unit is configured to set an image region in the captured image. The second extraction unit is configured to extract a second feature amount included in an image of the image region. The selection unit is configured to select a part of the first feature amount extracted by the first extraction unit based on information about the image region set by the region setting unit. The second object identification unit is configured to identify an object based on the second feature amount and a feature amount selected by the selection unit from among the first feature amounts. The captured image includes a left image and a right image. The distance image generation unit is configured to generate a distance image based on the left image and the right image. The region setting unit selects, based on the distance image, By identifying pixels in the range image that are located close to each other and have approximately the same disparity value,The image region is set. The selection unit selects, from among the first feature amounts, a feature amount of a region corresponding to the image region, based on the magnitude of the disparity value in the image region. A third image processing device according to an embodiment of the present disclosure includes a first extraction unit, a first object identification unit, an area setting unit, a second extraction unit, a selection unit, and a second object identification unit. The first extraction unit is configured to extract a first feature amount included in a captured image. The first object identification unit is configured to identify an object based on the first feature amount. The area setting unit is configured to set an image area in the captured image. The second extraction unit is configured to extract a second feature amount included in an image of the image area. The selection unit is configured to select a part of the first feature amount extracted by the first extraction unit based on information about the image area set by the area setting unit. The second object identification unit is configured to identify an object based on the second feature amount and the feature amount selected by the selection unit from the first feature amount. The host device is mounted on a moving body. The area setting unit selects, based on travel information of the moving body, By predicting the direction and distance the moving object will travel, the area in the captured image through which the moving object will travel is identified, and based on the identification result, The selecting unit selects, from among the first feature amounts, a feature amount of a region that corresponds to the image region, based on a position of the image region in the captured image. Effect of the Invention

[0007] According to one embodiment of the present disclosure A first image processing device, a second image processing device, and a third image processing device According to the image processing device, the amount of calculation can be reduced. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an image processing device according to an embodiment of the present disclosure. [Diagram 2] 2 is an explanatory diagram illustrating an example of a left image and a right image generated by the stereo camera illustrated in FIG. 1. [Diagram 3] 2 is an explanatory diagram illustrating an example of an image region set by a region setting unit shown in FIG. 1. [Figure 4]2 is an explanatory diagram illustrating an example of a configuration of a neural network in the feature extraction unit illustrated in FIG. 1. [Figure 5A] 2 is an explanatory diagram illustrating an example of an operation of a feature extraction unit illustrated in FIG. 1; [Figure 5B] 2 is an explanatory diagram illustrating an example of an operation of a feature extraction unit illustrated in FIG. 1; [Figure 6] FIG. 13 is a block diagram illustrating an example of the configuration of an image processing device according to a modified example. [Figure 7] FIG. 13 is a block diagram illustrating an example of the configuration of an image processing device according to another modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment> [Configuration example] 1 shows an example of the configuration of an image processing device (image processing device 1) according to an embodiment. The image processing device 1 includes a stereo camera 11 and a processing unit 20. The image processing device 1 is mounted on a vehicle 10 such as an automobile.

[0011] The stereo camera 11 is configured to capture an image in front of the vehicle 10, thereby generating a pair of images (a left image PL and a right image PR) having a parallax between them. The stereo camera 11 has a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. The left camera 11L and the right camera 11R are disposed, for example, in the vehicle 10, near the upper part of the windshield of the vehicle 10, and spaced apart by a predetermined distance in the width direction of the vehicle 10. The left camera 11L and the right camera 11R perform imaging operations in synchronization with each other. The left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL includes a plurality of pixel values, and the right image PR includes a plurality of pixel values. The left image PL and the right image PR constitute a stereo image PIC.

[0012] Fig. 2 shows an example of a stereo image PIC, Fig. 3(A) shows an example of a left image PL, and Fig. 3(B) shows an example of a right image PR. In this example, another vehicle (a preceding vehicle 90) is traveling ahead of the vehicle 10 on the road on which the vehicle 10 is traveling. The left camera 11L captures an image of the preceding vehicle 90 to generate the left image PL, and the right camera 11R captures an image of the preceding vehicle 90 to generate the right image PR.

[0013] The stereo camera 11 generates a stereo image PIC including such a left image PL and a right image PR. The stereo camera 11 performs an imaging operation at a predetermined frame rate (for example, 60 [fps]) to generate a series of stereo images PIC.

[0014] The processing unit 20 (FIG. 1) is configured to identify an object ahead of the vehicle 10 based on the stereo image PIC supplied from the stereo camera 11. In the vehicle 10, for example, based on information about the object identified by the processing unit 20, the vehicle 10 can perform, for example, driving control of the vehicle 10, or display information about the identified object on a console monitor. The processing unit 20 is configured, for example, by a CPU (Central Processing Unit) that executes a program, a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores the program. The processing unit 20 has an image memory 21, identification units 30 and 40, a feature amount selection unit 22, a buffer 23, and a synthesis unit 24.

[0015] The image memory 21 is configured to temporarily store the left image PL and the right image PR included in the stereo image PIC. The image memory 21 then supplies the stored left image PL and right image PR to the recognition units 30 and 40 in sequence.

[0016] The identification unit 30 is configured to set one or more image regions R based on the left image PL and the right image PR read from the image memory 21, and identify an object in each of the one or more image regions R. Specifically, as shown below, the identification unit 30 is configured to identify an object in the image of the image region R, which is a rectangular region, by using, for example, a trained model of a DNN (Deep Neural Network), and assign a category indicating what the object is to each of the image regions R. The identification unit 30 has a distance image generation unit 31, a region setting unit 32, a feature extraction unit 33, a feature combination unit 34, and an object identification unit 35.

[0017] The distance image generating unit 31 is configured to generate a distance image PZ by performing predetermined image processing, including stereo matching processing and filtering processing, based on the left image PL and the right image PR read from the image memory 21. The distance image PZ includes a plurality of pixel values. In this example, each of the plurality of pixel values ​​is a parallax value. In other words, each of the plurality of pixel values ​​corresponds to a distance to a point corresponding to each pixel in a three-dimensional real space. Note that this is not limited to this, and for example, each of the plurality of pixel values ​​may be a distance value indicating a distance to a point corresponding to each pixel in a three-dimensional real space.

[0018] The region setting unit 32 is configured to set one or more image regions R based on the distance image PZ. Specifically, the region setting unit 32 identifies a plurality of pixels that are located close to each other in the image and have approximately the same parallax value based on the distance image PZ, and sets a rectangular region including the plurality of pixels as the image region R. That is, when an object is present, the pixels of the region corresponding to the object in the distance image PZ are located close to each other and have approximately the same parallax value. Thus, the region setting unit 32 sets the image region R in this manner to surround the object.

[0019] 3 shows an example of an image P, which is one of the left image PL and the right image PR. In this example, an image region R is set for each of the two vehicles. Note that, in this example, the region setting unit 32 sets the image region R for the vehicle, but is not limited thereto, and for example, the image region R can also be set for a person, a guardrail, a wall, etc. Then, the region setting unit 32 supplies information about the one or more image regions R that have been set to the feature amount extraction unit 33 and the feature amount selection unit 22.

[0020] The feature extraction unit 33 (FIG. 1) is configured to extract feature amounts FA included in an image P, which is, for example, one of the left image PL and the right image PR, based on each image of one or a plurality of image regions R in the image P. The feature extraction unit 33 is configured to extract feature amounts FA using a trained model of DNN.

[0021] FIG. 4 is a schematic diagram showing an example of the configuration of the neural network 100A in the feature extraction unit 33. The neural network 100A includes a plurality (N) of convolutional layers LC (convolutional layers LC1 to LC N ) and multiple (N) pooling layers LP (pooling layers LP1 to LP N ) The convolutional layers LC and the pooling layers LP are arranged alternately.

[0022] The feature extraction unit 33 first inputs an image of the image region R to the neural network 100A. Based on the input image, the feature extraction unit 33 performs convolution processing in the convolution layer LC1 and pooling processing in the pooling layer LP1.

[0023] 5A and 5B show an example of the convolution process by the convolution layer LC1 and the pooling process by the pooling layer LP1.

[0024] First, the feature extraction unit 33 performs a convolution process based on the image data 101. Specifically, the feature extraction unit 33 sets an area W1 of a predetermined size (3 pixels x 3 pixels in this example) in the image data 101, and calculates pixel values ​​in the image data 102 by weighting and adding nine pixel values ​​in the area W1 with nine filter values ​​in the filter 103 as weighting coefficients. In this example, the nine filter values ​​in the filter 103 are each set to "1 / 9", but this is not limited to this. These nine filter values ​​are generated by performing machine learning in advance. The feature extraction unit 33 sequentially sets the area W1 in the image data 101 while shifting it by one pixel, and calculates pixel values ​​by performing weighting and adding in each of the multiple areas W1 that have been set. In this manner, the feature extraction unit 33 generates the image data 102.

[0025] For example, in the example of Fig. 5A, the feature extraction unit 33 uses a plurality of (four in this example) filters 103 that are different from each other to generate a plurality of (four in this example) image data 102. In this case, a plurality of pixel values ​​in each of the plurality of image data 102 is the feature FA1. Also, for example, in the example of Fig. 5B, the feature extraction unit 33 uses a single filter 103 to generate a single image data 102. In this case, a plurality of pixel values ​​in the single image data 102 is the feature FA1.

[0026] Then, the feature extraction unit 33 performs a pooling process based on the image data 102 generated by the convolution process. Specifically, the feature extraction unit 33 sets an area W2 of a predetermined size (2 pixels x 2 pixels in this example) in the image data 102, selects the largest pixel value among the four pixel values ​​in this area W2, and sets the pixel value as the pixel value in the image data 104. The feature extraction unit 33 sequentially sets the areas W2 in the image data 102, and selects the largest pixel value in each of the multiple areas W2 that have been set. In this manner, the feature extraction unit 33 generates the image data 104.

[0027] For example, in the example of Fig. 5A, the feature extraction unit 33 generates a plurality of (four in this example) image data 104 based on a plurality of (four in this example) image data 102. In this case, a plurality of pixel values ​​in each of the plurality of image data 104 are the feature amount FA2. Also, in the example of Fig. 5B, the feature extraction unit 33 generates one image data 104 based on one image data 102. In this case, a plurality of pixel values ​​in the one image data 104 are the feature amount FA2.

[0028] In this example, by performing the convolution process and the pooling process once in this manner, the number of pixels in the image data is reduced to about 1 / 4. The feature extraction unit 33 repeats the convolution process and the pooling process multiple times (N times) as shown in FIG. 4. The feature extraction unit 33 performs the convolution process N times, for example, using filters 103 different from each other. The number of filters 103 in each of the N convolution processes is set individually. When the image of the image region R of the number of filters 103 is a vehicle, the image data includes the characteristics of the vehicle, for example, the outline of the vehicle becomes clearer, by performing the convolution process multiple times in this manner. The pixel values ​​of the image data generated by the feature extraction unit 33 performing the convolution process first are the feature FA1, the pixel values ​​of the image data generated by performing the pooling process thereafter are the feature FA2, the pixel values ​​of the image data generated by performing the convolution process thereafter are the feature FA3, and the pixel values ​​of the image data generated by performing the pooling process thereafter are the feature FA4. The same applies thereafter. In this way, the feature extraction unit 33 extracts a plurality of feature amounts FA1, FA2, FA3, FA4, ... as feature amounts FA. Note that in this example, all the processing results of the convolution processing and pooling processing are set as feature amounts FA, but this is not limited thereto, and the processing results of some of the convolution processing and pooling processing may be set as feature amounts FA. Specifically, only the processing results of the last Nth convolution processing and pooling processing may be set as feature amounts FA.

[0029] The feature combining unit 34 (FIG. 1) is configured to combine the feature FA extracted by the feature extracting unit 33 with the feature supplied from the discriminating unit 40 via the feature selecting unit 22 and the buffer 23.

[0030] The object identification unit 35 is configured to identify an object in each of one or more image regions R based on the feature amounts combined by the feature amount combination unit 34. That is, for example, if the image of the image region R shows a vehicle, the feature amount includes the feature of the vehicle, and if the image of the image region R shows a person, the feature amount includes the feature of the person, so that the object identification unit 35 can identify the object in the image region R based on this feature amount. Then, the object identification unit 35 assigns a category indicating what the object is to each of the image regions R. Specifically, if the object in the image of the image region R is a vehicle, the object identification unit 35 assigns a category indicating a vehicle to the image region R, and if the object in the image of the image region R is a person, the object identification unit 35 assigns a category indicating a person to the image region R.

[0031] The identification unit 40 is configured to identify an object in an image P, which is, for example, one of the left image PL and the right image PR, read from the image memory 21. Specifically, like the identification unit 30, the identification unit 40 identifies an object in the image P, which is an entire image, by using, for example, a trained model of DNN. The identification unit 40 performs so-called semantic segmentation processing to assign, to each pixel of the image P, a label indicating to which object the pixel belongs. The identification unit 40 has a feature extraction unit 43 and an object identification unit 45.

[0032] The feature extraction unit 43 is configured to extract a feature FB included in an image P, which is one of the left image PL and the right image PR, for example. The feature extraction unit 43 is configured to extract the feature FB using a trained model of a DNN, similar to the feature extraction unit 33 in the classification unit 30. The neural network 100B in the feature extraction unit 43 is configured to extract the feature FB using a trained model of a DNN, similar to the neural network 100A (FIG. 4) in the feature extraction unit 33, which includes multiple (M) convolutional layers LC (convolutional layers LC1 to LC M ) and multiple (M) pooling layers LP (pooling layers LP1 to LP M ) and

[0033] The feature extraction unit 43 first reduces the image P, which is the entire image, by, for example, half in the horizontal and vertical directions, and inputs the reduced image to the neural network 100B. Then, the feature extraction unit 43 repeats the convolution process and the pooling process multiple times (M times) in the same manner as the feature extraction unit 33 (FIG. 4). In this manner, the feature extraction unit 33 extracts multiple feature amounts FB1, FB2, FB3, FB4, ... as the feature amounts FB. Note that in this example, the processing results of all the convolution processes and the pooling processes are set as the feature amounts FB, but this is not limiting, and the processing results of some of the convolution processes and the pooling processes may be set as the feature amounts FB.

[0034] The object identification unit 45 is configured to identify an object in the image P based on the feature amount FB. Then, based on the identification result, the object identification unit 45 assigns, to each pixel of the image P, a label indicating to which image of an object the pixel belongs. Specifically, when a pixel belongs to an image of a vehicle, the object identification unit 45 assigns, to the pixel, a label indicating that the pixel belongs to an image of a vehicle, when a pixel belongs to an image of a person, the object identification unit 45 assigns, to the pixel, a label indicating that the pixel belongs to an image of a person, and when a pixel belongs to an image of the ground such as a road, the object identification unit 45 assigns, to the pixel, a label indicating that the pixel belongs to an image of the ground.

[0035] The feature amount selection unit 22 is configured to select, from the feature amounts FB, feature amounts to be supplied to the classification unit 30, based on information on one or more image regions R supplied from the region setting unit 32. Specifically, the feature amount selection unit 22 selects, from the feature amounts FB, feature amounts to be supplied to the classification unit 30, based on the positions of one or more image regions R in the image P, the parallax values ​​in one or more image regions R in the distance image PZ, and the like.

[0036] The buffer 23 is configured to temporarily store the features supplied from the feature selection unit 22. Then, the buffer 23 is configured to supply the stored features to a feature combination unit 34 of the classification unit 30.

[0037] The synthesis unit 24 is configured to synthesize the processing result of the object identification unit 35 in the identification unit 30 and the processing result of the object identification unit 45 in the identification unit 40, and output the synthesized processing result as an identification result RES.

[0038] With this configuration, in the image processing device 1, the classification unit 40 classifies an object based on the image P, which is the entire image, and the classification unit 30 classifies the object based on the image of one or more image regions R set by the region setting unit 32. At that time, in the image processing device 1, the feature selection unit 22 supplies a part of the feature values ​​FB in the classification unit 40 to the classification unit 30 via the buffer 23. As a result, in the image processing device 1, the classification unit 30 can reuse a part of the feature values ​​FB extracted by the classification unit 40, thereby reducing the amount of calculation, and as a result, it is possible to perform the classification process by effectively using limited calculation resources.

[0039] Here, the feature extraction unit 43 corresponds to a specific example of a "first extraction unit" in the present disclosure. The feature FB corresponds to a specific example of a "first feature" in the present disclosure. The object identification unit 45 corresponds to a specific example of a "first object identification unit" in the present disclosure. The area setting unit 32 corresponds to a specific example of a "area setting unit" in the present disclosure. The image area R corresponds to a specific example of an "image area" in the present disclosure. The feature extraction unit 33 corresponds to a specific example of a "second extraction unit" in the present disclosure. The feature FA corresponds to a specific example of a "second feature" in the present disclosure. The feature selection unit 22 corresponds to a specific example of a "selection unit" in the present disclosure. The object identification unit 35 corresponds to a specific example of a "second object identification unit" in the present disclosure.

[0040] [Actions and Actions] Next, the operation and function of the image processing device 1 of this embodiment will be described.

[0041] (Overview of overall operation) First, an overview of the overall operation of the image processing device 1 will be described with reference to FIG. 1. The stereo camera 11 captures an image in front of the vehicle 10 to generate a stereo image PIC including a left image PL and a right image PR. In the processing unit 20, the image memory 21 temporarily stores the left image PL and the right image PR included in the stereo image PIC. The identification unit 40 identifies an object in an image P, which is one of the left image PL and the right image PR, read from the image memory 21. Specifically, the identification unit 40 identifies an object in the image P, which is a whole image, using, for example, a trained model of DNN. Then, the identification unit 40 assigns a label to each pixel of the image P indicating to which object the pixel belongs by performing so-called semantic segmentation processing. The identification unit 30 sets one or more image regions R based on the left image PL and the right image PR read from the image memory 21, and identifies objects in each image of the one or more image regions R. Specifically, the identification unit 30 uses the trained model of the DNN to identify objects in an image of an image region R, which is a rectangular region, and assigns a category indicating what the object is to each image region R. The synthesis unit 24 synthesizes the processing result of the object identification unit 35 in the identification unit 30 and the processing result of the object identification unit 45 in the identification unit 40, and outputs the synthesized processing result as the identification result RES.

[0042] (Details of operation) Next, the operation of the processing unit 20 (FIG. 1) will be described in detail.

[0043] In the classification unit 40 of the processing unit 20, the feature extraction unit 43 extracts a feature FB included in an image P, which is one of the left image PL and the right image PR, for example. Specifically, the feature extraction unit 43 reduces the image P, which is the entire image, inputs the reduced image to the neural network 100B, and repeats the convolution process and the pooling process multiple times (M times) to extract multiple feature amounts FB1, FB2, FB3, FB4, ... as feature amounts FB. The object classification unit 45 classifies objects in the image P based on these feature amounts FB. Then, the object classification unit 45 performs a so-called semantic segmentation process to assign a label to each pixel of the image P indicating to which object the pixel belongs.

[0044] In the identification unit 30, the distance image generation unit 31 performs predetermined image processing, including stereo matching and filtering, based on the left image PL and the right image PR read from the image memory 21, to generate a distance image PZ. The region setting unit 32 sets one or more image regions R based on the distance image PZ. Specifically, the region setting unit 32 identifies a plurality of pixels that are located close to each other in the image and have approximately the same parallax value based on the distance image PZ, and sets a region including the plurality of pixels as the image region R. The feature extraction unit 33 extracts a feature FA included in the image P based on each image of one or more image regions R. Specifically, the feature extraction unit 33 inputs the image of the image region R to the neural network 100A, and repeats the convolution process and the pooling process multiple times (N times) to extract a plurality of feature amounts FA1, FA2, FA3, FA4, ... as the feature amounts FA.

[0045] The feature amount selection unit 22 selects a feature amount to be supplied to the classification unit 30 from among the feature amounts FB extracted by the feature amount extraction unit 43, based on information on one or more image regions R supplied from the region setting unit 32. Specifically, the feature amount selection unit 22 selects a feature amount to be supplied to the classification unit 30 from among the feature amounts FB, based on the positions of one or more image regions R in the image P, the parallax values ​​in one or more image regions R in the distance image PZ, and the like. For example, the feature amount selection unit 22 selects a feature amount of a region corresponding to the image region R from among the feature amounts FB, based on the position of the image region R. That is, since the feature amount FB is image data as shown in FIGS. 4, 5A, and 5B, the feature amount selection unit 22 can select a feature amount of a region corresponding to the image region R. Also, for example, the feature amount selection unit 22 can select a feature amount in which the feature of an object in the image of the image region R is prominently expressed from among the multiple feature amounts FB1, FB2, FB3, FB4, ..., based on the parallax value in the image region R. Specifically, for example, when the disparity value is small, the distance to the object is far, and therefore performing a large number of convolution and pooling processes may result in the image becoming distorted. Therefore, it is possible to avoid selecting the last features among the multiple features FB1, FB2, FB3, FB4, ....

[0046] The buffer 23 temporarily stores the features supplied from the feature selection unit 22. Then, the buffer 23 supplies the stored features to a feature combination unit 34 of the classification unit 30.

[0047] In the classification unit 30, the feature combination unit 34 combines the feature FA extracted by the feature extraction unit 33 with the feature supplied from the classification unit 40 via the feature selection unit 22 and the buffer 23. The object classification unit 35 identifies an object in each of one or more image regions R based on the feature combined by the feature combination unit 34. Then, the object classification unit 35 assigns a category indicating what the object is to each image region R.

[0048] Then, the synthesis unit 24 synthesizes the processing result of the object identification unit 35 in the identification unit 30 and the processing result of the object identification unit 45 in the identification unit 40, and outputs the synthesized processing result as the identification result RES.

[0049] In this way, in the image processing device 1, the feature extraction unit 43 extracts the feature FB included in the image P, and the feature extraction unit 33 extracts the feature FA included in the image of the image region R. As a result, the object can be identified based on the image P, which is the entire image, and also based on the image of the image region R, so that the amount of calculation can be reduced while improving the identification accuracy. That is, for example, if only identifying the object based on the image P, which is the entire image, it is difficult to analyze the entire image in detail and in real time because of limited calculation resources, and there is a risk of the identification accuracy being reduced. In particular, if the image P, which is the entire image, is reduced in order to reduce the amount of calculation, as in the feature extraction unit 43, and the convolution process and pooling process are performed multiple times based on the reduced image, for example, the image of a distant object is crushed, making it difficult to identify such an object. On the other hand, in the image processing device 1, the object is identified based on the image P, which is the entire image, and also based on the image of one or more image regions R. As a result, in the image processing device 1, for example, by setting the image region R for a distant object, the distant object can be individually identified, and therefore the amount of calculation can be reduced while improving the identification accuracy.

[0050] Moreover, in the image processing device 1, the feature extraction unit 43 extracts the feature FB contained in the image P, and the feature extraction unit 33 extracts the feature FA contained in the image of the image region R, and the object identification unit 35 identifies an object in this image region R based on a part of the feature FB and the feature FA. As a result, in the image processing device 1, the object identification unit 35 can identify an object by using a part of the feature FB in addition to the feature FA, so that it is possible to reduce the amount of calculation while improving the identification accuracy. As a result, the image processing device 1 can identify an object by effectively using limited calculation resources.

[0051] Furthermore, in the image processing device 1, the feature selection unit 22 selects a part of the feature FB based on information about the image region R set by the region setting unit 32, so that it is possible to selectively supply to the classification unit 30 feature amounts that can improve the classification accuracy of the classification unit 30, and to avoid supplying feature amounts that do not contribute to improving the classification accuracy to the classification unit 30. This makes it possible to reduce the amount of calculations performed by the classification unit 30 in the image processing device 1.

[0052] [effect] As described above, in this embodiment, the feature amounts contained in the image P are extracted, and the feature amounts contained in the image of the image region are also extracted, so that it is possible to reduce the amount of calculation while improving the classification accuracy.

[0053] In this embodiment, a feature FB contained in image P is extracted, and a feature FA contained in the image of the image region is extracted. Objects in this image region are identified based on a portion of the feature FB and the feature FA, thereby making it possible to reduce the amount of calculation while improving the identification accuracy.

[0054] In this embodiment, a portion of the feature values ​​FB is selected based on information about the image region set by the region setting unit, so that the amount of calculation can be reduced.

[0055] [Variation 1] In the above embodiment, the object identification section 45 of the identification unit 40 identifies an object based on the feature amount FB extracted by the feature amount extraction section 43, but this is not limited to this. Instead, the object identification section may identify an object based on, for example, a feature amount of the feature amount FB other than the feature amount supplied to the identification unit 30. This modification will be described in detail below.

[0056] FIG. 6 shows an example of the configuration of an image processing device 1A according to this modified example. The image processing device 1A includes a processing unit 20A. The processing unit 20A includes a recognition unit 40A. The recognition unit 40A includes a feature quantity selection unit 44A. The feature quantity selection unit 44A is configured to select a feature quantity to be supplied to the recognition unit 30 from among the feature quantities FB, based on information on one or more image regions R supplied from the region setting unit 32, similar to the feature quantity selection unit 22 according to the above embodiment. In addition, the feature quantity selection unit 44A is configured to supply the feature quantities other than the feature quantity to be supplied to the recognition unit 30 from among the feature quantities FB, to the object recognition unit 45. As a result, the object recognition unit 45 recognizes an object based on the feature quantities other than the feature quantity to be supplied to the recognition unit 30 from among the feature quantities FB. In other words, the object recognition unit 45 does not perform processing based on the same feature quantity as the feature quantity to be supplied to the recognition unit 30. As a result, the amount of calculation in the object recognition unit 45 can be reduced.

[0057] [Variation 2] In the above embodiment, the region setting unit 32 sets one or more image regions R based on the distance image PZ, but this is not limited to this. Instead, the region setting unit may set one or more image regions R based on, for example, travel information of the vehicle 10. This modified example will be described in detail below.

[0058] 7 shows an example of the configuration of an image processing device 1B according to this modified example. The image processing device 1B includes a camera 11B and a processing unit 20B.

[0059] Camera 11B includes a lens and an image sensor, and is configured to generate image P. That is, while a stereo camera is used in the above embodiment, a monocular camera is used in this modification.

[0060] The processing unit 20B has a recognition unit 30B. The recognition unit 30B has a travel information acquisition unit 31B and an area setting unit 32B.

[0061] The travel information acquisition unit 31B is configured to acquire travel information of the vehicle 10 from, for example, a vehicle control device of the vehicle 10 via a controller area network (CAN). The travel information includes, for example, information such as the travel speed and yaw rate of the vehicle 10.

[0062] The region setting unit 32B is configured to set one or more image regions R by predicting the course of the vehicle 10 based on the driving information acquired by the driving information acquisition unit 31B. Specifically, the region setting unit 32B predicts in which direction and how far the vehicle 10 will travel based on the driving information of the vehicle 10, thereby specifying the region in the image P in which the vehicle 10 will travel, and sets one or more image regions R in the vicinity of the specified region. That is, for example, near the region in the image P in which the vehicle 10 travels, there is a high need to identify objects in order to avoid collision with the vehicle 10, while in a region away from the region in which the vehicle travels, there is a low risk of collision with the vehicle 10, and therefore there is a low need to identify objects. Therefore, the region setting unit 32B is configured to set one or more image regions R in the vicinity of the region in the image P in which the vehicle 10 travels.

[0063] Even with this configuration, the same effects as those of the above embodiment can be obtained.

[0064] Although the present technology has been described above by giving embodiments and modifications, the present technology is not limited to these embodiments and the like, and various modifications are possible.

[0065] For example, in the above embodiment, the stereo camera 11 and the camera 11B are configured to capture an image of the area in front of the vehicle 10. However, this is not limited to this. For example, the stereo camera 11 and the camera 11B may capture an image of the area to the side or rear of the vehicle 10.

[0066] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained. [Explanation of symbols]

[0067] 1, 1A, 1B... image processing device, 11... stereo camera, 11B... camera, 11L... left camera, 11R... right camera, 20, 20A, 20B... processing unit, 21... image memory, 22, 44A... feature selection unit, 23... buffer, 24... synthesis unit, 30, 30B... recognition unit, 31... distance image generation unit, 31B... driving information acquisition unit, 32, 32B... area setting unit, 33... feature extraction unit, 34... feature combination unit , 35...object recognition unit, 40, 40A...recognition unit, 43...feature extraction unit, 45...object recognition unit, 100A, 100B...neural network, 101, 102, 104...image data, 103...filter, FA, FB...feature, LC...convolution layer, LP...pooling layer, P...image, PIC...stereo image, PL...left image, PR...right image, PZ...distance image, R...image region, RES...recognition result.

Claims

1. a first extraction unit that extracts a first feature amount included in a captured image; a first object identification unit that identifies an object based on the first feature amount; A distance image generating unit; a region setting unit that sets an image region in the captured image; a second extraction unit that extracts a second feature amount included in an image of the image region; a selection unit that selects a part of the first feature amount extracted by the first extraction unit based on information about the image region set by the region setting unit; a second object identification unit that identifies an object based on the second feature amount and the feature amount selected by the selection unit from the first feature amount; Equipped with the captured image includes a left image and a right image, the distance image generating unit generates a distance image based on the left image and the right image; the region setting unit sets the image region by identifying a plurality of pixels that are located close to each other in the distance image and have approximately the same disparity value, based on the distance image; The selection unit selects, from the first feature amounts, a feature amount of a region corresponding to the image region, based on a position of the image region in the captured image. Image processing device.

2. a first extraction unit that extracts a first feature amount included in a captured image; a first object identification unit that identifies an object based on the first feature amount; A distance image generating unit; a region setting unit that sets an image region in the captured image; a second extraction unit that extracts a second feature amount included in an image of the image region; a selection unit that selects a part of the first feature amount extracted by the first extraction unit based on information about the image region set by the region setting unit; a second object identification unit that identifies an object based on the second feature amount and the feature amount selected by the selection unit from the first feature amount; Equipped with the captured image includes a left image and a right image, the distance image generating unit generates a distance image based on the left image and the right image; the region setting unit sets the image region by identifying a plurality of pixels that are located close to each other in the distance image and have approximately the same disparity value, based on the distance image; The selection unit selects, from the first feature amounts, a feature amount of a region corresponding to the image region, based on a magnitude of a disparity value in the image region. Image processing device.

3. a first extraction unit that extracts a first feature amount included in a captured image; a first object identification unit that identifies an object based on the first feature amount; a region setting unit that sets an image region in the captured image; a second extraction unit that extracts a second feature amount included in an image of the image region; a selection unit that selects a part of the first feature amount extracted by the first extraction unit based on information about the image region set by the region setting unit; a second object identification unit that identifies an object based on the second feature amount and the feature amount selected by the selection unit from the first feature amount; Equipped with The device is mounted on a moving object, the region setting unit predicts, based on travel information of the moving object, in which direction and how far the moving object will travel, thereby specifying a region in the captured image in which the moving object will travel, and sets the image region based on the result of the specification; The selection unit selects, from the first feature amounts, a feature amount of a region corresponding to the image region, based on a position of the image region in the captured image. Image processing device.

4. the first extraction unit extracts a plurality of partial feature amounts by repeatedly performing a convolution operation a plurality of times on the basis of the captured image, and outputs the plurality of partial feature amounts as the first feature amount; The selection unit selects one or more partial feature amounts from the plurality of partial feature amounts in the first feature amount based on information about the image region. The image processing device according to claim 1 .

Images