Image processing device and vehicle

Abstract

To provide an image processing apparatus or the like, configured to increase the speed of estimation processing speed of an object position.SOLUTION: An image processing apparatus includes: an area setting unit which sets one or more image areas in a captured image; an extraction unit which extracts two-dimensional layout feature quantities included in the image areas of the captured image; and a position estimation unit which performs correlation operation using convolutional operation on the feature quantities, in image areas of the captured image obtained in each of frame periods adjacent along a time axis, to estimate a position of an object to be tracked. The position estimation unit uses Fourier transform on the feature quantities in the convolution operation, and executes the Fourier transform and the convolution operation on the two-dimensional layout feature quantities per dimension.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an image processing device that estimates the position of an object based on a captured image, and a vehicle equipped with such an image processing device. [Background technology]

[0002] Images captured by an imaging device include images of various objects. For example, Patent Document 1 discloses an image processing device that estimates the position of an object based on such captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-123141 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in such an image processing device, there is a demand for speeding up the process of estimating the object position. It is desirable to provide an image processing device capable of speeding up the process of estimating the object position, and a vehicle equipped with such an image processing device. [Means for solving the problem]

[0005] An image processing device according to an embodiment of the present disclosure includes an area setting unit that sets one or more image areas in a captured image, an extraction unit that extracts feature amounts of a two-dimensional arrangement included in the image areas of the captured image, and a position estimation unit that estimates the position of a tracked object by performing a correlation calculation using a convolution calculation on the feature amounts between image areas of the captured images obtained in previous and subsequent frame periods along a time axis. When performing the convolution calculation, the position estimation unit uses a Fourier transform on the feature amounts, and performs the Fourier transform and the convolution calculation on the feature amounts arranged in the two-dimensional arrangement, one dimension at a time.

[0006] A vehicle according to one embodiment of the present disclosure includes an image processing device according to the embodiment of the present disclosure, and a vehicle control unit that controls the vehicle using the estimated position of a tracked object obtained from the position estimation unit. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram illustrating an example of a schematic configuration of a vehicle according to an embodiment of the present disclosure. [Diagram 2] 2 is a top view showing a schematic external configuration example of the vehicle shown in FIG. 1. [Diagram 3] 2 is a schematic diagram illustrating an example of a left image and a right image generated by the stereo camera illustrated in FIG. 1. [Figure 4] 2 is a schematic diagram illustrating an example of an image region set by a region setting unit illustrated in FIG. 1. [Diagram 5] FIG. 1 is a schematic diagram for explaining an overview of object tracking using correlation calculation. [Figure 6] 5A and 5B are schematic diagrams illustrating an example of a correlation calculation process according to the embodiment. [Figure 7] 11 is a flowchart illustrating an example of a process including a correlation calculation according to an embodiment. [Figure 8] 11A and 11B are schematic diagrams illustrating examples of correlation calculation processes according to a comparative example and an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order. 1. Embodiment (Example of Setting Image Region Using Distance Information or Machine Learning) 2. Variations

[0009] <1. Preferred embodiment> [composition] Fig. 1 is a block diagram showing an example of a schematic configuration of a vehicle (vehicle 10) according to an embodiment of the present disclosure. Fig. 2 is a schematic top view showing an example of the external configuration of the vehicle 10 shown in Fig. 1.

[0010] As shown in Fig. 1, the vehicle 10 includes a stereo camera 11, an image processing device 12, and a vehicle control unit 13. Note that the driving power source (engine, motor, etc.) of the vehicle 10 is omitted in Fig. 1. The vehicle 10 is, for example, an electrically-powered vehicle such as a hybrid electric vehicle (HEV) or an electric vehicle (EV), or a gasoline-powered vehicle.

[0011] (A. Stereo Camera 11) The stereo camera 11 is a camera that captures an image of the area in front of the vehicle 10 to generate a pair of images (a left image PL and a right image PR) having a parallax therebetween, as shown in Fig. 2. As shown in Figs. 1 and 2, the stereo camera 11 has a left camera 11L and a right camera 11R.

[0012] The left camera 11L and the right camera 11R each include, for example, a lens and an image sensor. The left camera 11L and the right camera 11R are arranged near the upper part of the windshield 19 of the vehicle 10, for example, as shown in FIG. 2, at a predetermined distance apart along the width direction of the vehicle 10. The left camera 11L and the right camera 11R are adapted to perform imaging operations in synchronization with each other. Specifically, as shown in FIG. 1, the left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL includes a plurality of pixel values, and the right image PR includes a plurality of pixel values. The left image PL and the right image PR constitute a stereo image PIC, as shown in FIG. 1.

[0013] FIG. 3 shows an example of such a stereo image PIC. Specifically, FIG. 3(A) shows an example of a left image PL, and FIG. 3(B) shows an example of a right image PR. Note that x and y shown in FIG. 3 represent the x-axis and y-axis, respectively. In this example, another vehicle (a preceding vehicle 90) is traveling ahead of the vehicle 10 on the road on which the vehicle 10 is traveling. The left camera 11L captures an image of the preceding vehicle 90 to generate the left image PL, and the right camera 11R captures an image of the preceding vehicle 90 to generate the right image PR.

[0014] The stereo camera 11 generates a stereo image PIC including such a left image PL and a right image PR. The stereo camera 11 also generates a series of stereo images PIC by performing an imaging operation at a predetermined frame rate (e.g., 60 [fps]).

[0015] (B. Image processing device 12) The image processing device 12 is a device that performs various image processing (such as tracking processing of an object ahead of the vehicle 10) based on the stereo image PIC supplied from the stereo camera 11. As shown in FIG. 1, the image processing device 12 has an image memory 121, a distance information generating unit 122, an area setting unit 123, a feature extraction unit 124, and a position estimation unit 125.

[0016] Such an image processing device 12 includes, for example, one or more processors (CPU: Central Processing Unit) that execute programs, and one or more memories communicably connected to these processors. In addition, such memories include, for example, a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores programs.

[0017] The feature amount extraction unit 124 described above corresponds to a specific example of an "extraction unit" in the present disclosure.

[0018] (Image memory 121) 1, the image memory 121 is a memory for temporarily storing the left image PL and the right image PR included in the stereo image PIC. The image memory 121 sequentially supplies at least one of the left image PL and the right image PR thus stored as a captured image P to the distance information generating unit 122 and the feature amount extracting unit 124 (see FIG. 1).

[0019] (Distance information generation unit 122) The distance information generating unit 122 generates distance information Iz by performing predetermined image processing including stereo matching processing and filtering processing based on the captured image P (here, the left image PL and the right image PR) read out from the image memory 121 (see FIG. 1). Specifically, the distance information generating unit 122 generates a distance image including a plurality of pixel values ​​based on the left image PL and the right image PR. In this example, each of the plurality of pixel values ​​is a parallax value. In other words, each of the plurality of pixel values ​​corresponds to a distance to a point corresponding to each pixel in a three-dimensional real space. Note that this example is not limited, and for example, each of the plurality of pixel values ​​may be a distance value indicating a distance to a point corresponding to each pixel in a three-dimensional real space. In this way, the distance information generating unit 122 generates distance information Iz, which is information indicating a distance to a point corresponding to each pixel.

[0020] (Area setting unit 123) The region setting unit 123 sets one or more image regions R in the captured image P based on the distance information Iz supplied from the distance information generating unit 122. Specifically, the region setting unit 123 specifies a plurality of pixels that are located close to each other in the captured image P and have approximately the same parallax value based on the distance information Iz, and sets a rectangular region including the plurality of pixels as the image region R. That is, if there is an object in the captured image P, the pixels in the region corresponding to the object are located close to each other and have approximately the same parallax value. Therefore, by setting the image region R in this manner, the region setting unit 123 sets the image region R so as to surround the object.

[0021] Fig. 4 is a schematic diagram showing an example of an image region R set by the region setting unit 123. In the example shown in Fig. 4, an image region R is set for each of two vehicles in a captured image P (here, one of the left image PL and the right image PR). Note that, in this example, the region setting unit 123 sets the image region R for the vehicles, but is not limited to this example, and may also set the image region R for, for example, a person, a guardrail, a wall, etc.

[0022] Information on one or more image regions R thus set by the region setting section 123 is supplied to a feature extraction section 124 as shown in FIG.

[0023] 1, the region setting unit 123 sets the image region R using the distance information Iz, but is not limited to this example. That is, the region setting unit 123 may identify an object in the captured image P using a trained model such as a DNN (Deep Neural Network) and set the image region R, which is a rectangular region, by outputting the coordinates of the identified object. That is, the region setting unit 123 may set the image region R using machine learning.

[0024] (Feature extraction unit 124) The feature extraction unit 124 extracts feature F contained in one or more image regions R in a captured image P (here, one of the left image PL and the right image PR) (see FIG. 1). The feature F is composed of pixel values ​​of a plurality of pixels arranged in a matrix (two-dimensional arrangement), as will be described in detail later (FIG. 6, etc.). Examples of such feature F include RGB (Red, Green, Blue) feature and HOB (Histograms of Oriented Gradients) feature.

[0025] The feature extraction unit 124 is configured to extract such feature F by using, for example, a trained model of DNN. In this case, for example, each of the neural networks in the feature extraction unit 124 has a plurality of convolution layers and a plurality of pooling layers.

[0026] The feature amount F thus extracted by the feature amount extraction unit 124 is supplied to a position estimation unit 125 (see FIG. 1).

[0027] (Position estimation unit 125) The position estimation unit 125 estimates the position of the tracked object by performing a correlation calculation using a convolution operation on the above-mentioned feature amount F between image regions R of captured images P obtained in previous and subsequent frame periods along the time axis. In other words, the position estimation unit 125 performs object tracking by estimating the position of the tracked object using the feature amount F for each captured image P in each frame period.

[0028] In addition, such object tracking is performed using an object tracking method that uses a CNN (Convolutional Neural Network), such as KCF (Kernelized Correlation Filter), template matching, or Siamese Network.

[0029] Fig. 5 is a schematic diagram showing an overview of object tracking using such correlation calculation. Note that t shown in Fig. 5 represents the time axis, and this also applies to the following descriptions.

[0030] For example, as shown in Fig. 5(A) and Fig. 5(B), in object tracking using correlation calculation, how the position (position coordinate x0) of the tracked object has moved in each captured image P0, P1, ... Pn at each timing t0, t1, ..., tn along the time axis t is estimated (see the amount of movement Δx). Specifically, in such object tracking, the position where the correlation value is the largest (indicates the maximum value) among the captured images P0, P1, ... Pn is estimated as the position coordinate x0 of the tracked object, and is updated as needed.

[0031] As an example, when tracking an object using the above-mentioned KCF, various calculations are performed as follows. That is, first, machine learning of the coefficient w is performed so that the value shown in the following formula (1) is minimized. Then, if the value of this coefficient w can be optimized, it becomes a formula that outputs the position coordinates of the tracked object in the input image. Note that, when optimizing the coefficient w in this way, for example, the kernel trick described later is used.

[0032]

number

[0033] 6 is a schematic diagram showing a correlation calculation process example according to the present embodiment. Specifically, the correlation calculation process example uses a feature amount F extracted from an image area R of a captured image Pn obtained in a current frame period (see FIG. 6(A)) and a feature amount Fn+1 obtained from an image area R of a captured image Pn+1 obtained in the immediately following frame period (one frame later: see FIG. 6(B)).

[0034] In each of the above feature quantities Fn and Fn+1, 16 pixels PX (4 pixels in the vertical direction corresponding to the y-axis direction × 4 pixels in the horizontal direction corresponding to the x-axis direction) are arranged in a matrix (two-dimensional arrangement) (see Fig. 6(A) and Fig. 6(B)). Fig. 6 also shows a correlation map Mc obtained from a correlation calculation by a convolution calculation using a Fourier transform, which will be described below.

[0035] In this embodiment, the position estimation unit 125 uses a Fourier transform on the feature amount F when performing a convolution operation on the feature amount F described above. The position estimation unit 125 also uses, for example, the KCF described above to perform a correlation operation using such a Fourier transform. Furthermore, when this KCF is used, as shown in Fig. 5, for example, a Fourier transform is used to convert a feature amount indicating a color space (such as the RGB feature amount described above) into a frequency space, thereby speeding up the processing.

[0036] Here, when the above-mentioned kernel trick is used in such KCF, the function F(x) for performing the Fourier transform is defined by the following formula (2). Moreover, when the above-mentioned convolution operation is expressed as (f*g), it is defined by the following formula (3).

[0037]

number

[0038] Then, the convolution operation between the Fourier transformed values ​​is expressed by the following formula (4). That is, as can be seen from the last formula in formula (4), the convolution operation between the Fourier transformed values ​​is equal to the product of the Fourier transformed values. Specifically, in the example of FIG. 6, when the convolution operation is performed on the feature amounts Fn and Fn+1 transformed into the frequency space using the Fourier transform, the operation is performed by taking the product of these feature amounts Fn and Fn+1 (matrices). By using KCF in this way, the amount of calculation is reduced and the processing speed is increased.

[0039]

number

[0040] In this embodiment, the position estimation unit 125 performs such Fourier transform and convolution calculation on the two-dimensionally arranged feature amount F, one dimension at a time (for example, see the dashed arrows shown in the feature amounts Fn and Fn+1 in Fig. 6(A) and Fig. 6(B)). Note that detailed processing examples in such a position estimation unit will be described later (Figs. 7 and 8).

[0041] (C. Vehicle control unit 13) The vehicle control unit 13 performs various vehicle controls on the vehicle 10 using the result of the position estimation (estimated position Pe of the tracked object) supplied from the position estimation unit 125 (see FIG. 1). Specifically, the vehicle control unit 13 performs, for example, driving control of the vehicle 10 and operation control of various members in the vehicle 10 based on the information on the estimated position Pe.

[0042] The vehicle control unit 13 includes, for example, one or more processors (CPUs) that execute programs and one or more memories communicably connected to these processors, similar to the image processing device 12. Also, similar to the image processing device 12, such memories include, for example, a RAM that temporarily stores processing data, and a ROM that stores programs.

[0043] [Operation and Effects] Next, the operation, functions, and effects of this embodiment will be described in detail.

[0044] (A. Position Estimation Processing, etc., of the Present Embodiment) First, an example of the process (position estimation process, etc.) including the correlation calculation in this embodiment will be described with reference to Figs. 1 to 6 as well as Figs. 7 and 8. Fig. 7 shows a flow chart of such a process example including correlation calculation. Fig. 8 shows a schematic comparison (see symbols CF indicated by dashed lines) of a comparative example (Fig. 8(A)) and an example of this embodiment (Fig. 8(B)).

[0045] 7, first, the stereo camera 11 captures an image of the area ahead of the vehicle 10 to generate a stereo image PIC (a left image PL and a right image PR) (step S100). Specifically, a stereo image PIC (a left image PL and a right image PR) such as that shown in FIG. 3 is generated.

[0046] Next, the image memory 21 in the image processing device 12 temporarily stores the stereo image PIC (left image PL and right image PR) thus generated as a captured image P (step S101). Then, the region setting unit 123 determines whether or not an image region R in the captured image P has been set (step S102). If it is determined that such an image region R has been set (step S102: Y), the process proceeds to step S104, which will be described later.

[0047] On the other hand, if it is determined that such an image region R has not been set (step S102: N), then the region setting unit 123 uses the above-mentioned method (distance information Iz or machine learning, etc.) to set one or more image regions R in the captured image P (step S103). Then, next, in step S104, the feature extraction unit 124 extracts the above-mentioned feature F (two-dimensional arrangement) included in the image region R of the captured image P.

[0048] Next, the position estimation unit 125 estimates the position of the tracked object by performing a correlation calculation using a convolution calculation on the feature amount F between the image regions R of the above-mentioned captured images Pn and Pn+1 (see FIG. 6) (step S105). Specifically, the position estimation unit 125 performs the above-mentioned Fourier transform and convolution calculation on the two-dimensionally arranged feature amount F one dimension at a time to estimate the position of such a tracked object.

[0049] More specifically, in the embodiment of Fig. 8(B) corresponding to the example of Fig. 6, the position estimation unit 125 performs position estimation as follows. That is, the position estimation unit 125 first performs one-dimensional processing of the Fourier transform and convolution operation as described above along the x-axis direction (row direction) of the feature amount Fn (see the dashed arrow). Then, the position estimation unit 125 adds the results of such one-dimensional processing along the x-axis direction for multiple sets (four sets in this example) along the y-axis direction (column direction) to each other, thereby performing Fourier transform and convolution operation on the two-dimensionally arranged feature amount Fn.

[0050] Here, the x-axis direction (row direction) corresponds to a specific example of a "first direction" in the present disclosure, and the y-axis direction (column direction) corresponds to a specific example of a "second direction" in the present disclosure.

[0051] If it is determined in step S105 that it is difficult to estimate the position of the tracked object, then the position estimation ends, and the series of processing examples shown in FIG. 7 also ends.

[0052] Next, the vehicle control unit 13 performs various vehicle controls (such as the driving control of the vehicle 10 and the operation control of various components) on the vehicle 10 using the position estimation result obtained in step S105 (the estimated position Pe of the tracked object described above) (step S106).

[0053] This is the end of the series of processing examples shown in FIG.

[0054] (B. Actions and Effects) In this manner, in this embodiment, the following processing is performed when estimating the position of a tracked object by performing correlation calculation using convolution calculation on extracted feature amounts F (two-dimensional arrangement) between image regions R of captured images P obtained in adjacent frame periods along the time axis. That is, when performing convolution calculation on feature amounts F, Fourier transform is used on feature amounts F, and these Fourier transforms and convolution calculations are each performed one-dimensionally on feature amounts F arranged in two dimensions. As a result, in this embodiment, the amount of calculation processing (calculation amount) is reduced compared to, for example, the case where these Fourier transforms and convolution calculations are each performed two-dimensionally (corresponding to the case of the comparative example below).

[0055] Specifically, in the comparative example shown in FIG. 8A, when considering the use of a Discrete Fourier Transform (DFT), the amount of calculation is as follows. That is, first, the order of the amount of calculation for one-dimensional DFT of pixels PX arranged in a two-dimensional array of (N×N) is O(N 2 ) As described above, in this comparative example, the Fourier transform is performed two-dimensionally (along both the x-axis (row) direction and the y-axis (column) direction: see the dashed arrows), so the order of the computational complexity in the case of a two-dimensional DFT is O(N 2 ×N 2 )=O(N 4 In other words, in the example of FIG. 8A, since the pixels PX are arranged in a two-dimensional (4×4) arrangement, the order of the amount of calculation required for the two-dimensional DFT is O(4 2 ×42 )=O(4 4 )

[0056] In contrast, in the embodiment shown in Figure 8(B), when considering the case where DFT is used in a similar manner, the amount of calculation is as follows. That is, in this embodiment, the results of the one-dimensional DFT (see the dashed arrow) executed along the x-axis are added to each other for multiple sets (four sets) along the y-axis direction, so the order of the total amount of calculation is O(N 2 ×N)=O(N 3 )=O(4 3 ) In other words, in this embodiment, the amount of calculations required for the DFT of the feature quantity Fn is reduced while maintaining the calculation accuracy, as compared with the comparative example described above. Note that such a reduction in the amount of calculations is not limited to the above-described DFT, but is also applicable to other types of Fourier transforms, such as the fast Fourier transform (FFT).

[0057] As a result, in this embodiment, it is possible to speed up the object position estimation process (shorten the processing time).

[0058] In addition, in this embodiment, the results of one-dimensional processing along the x-axis direction are added together for multiple sets along the y-axis direction to execute Fourier transform and convolution calculations on the two-dimensionally arranged feature amount F, as follows: In other words, by using such a specific calculation method, it is possible to actually achieve the reduction in the amount of calculations as described above.

[0059] Furthermore, in this embodiment, when estimating the position of a tracked object, a correlation calculation is performed using the above-mentioned KCF, and a feature value F indicating a color space is converted into a frequency space by a Fourier transform, which makes it possible to further increase the speed (further shorten the processing time).

[0060] In addition, in this embodiment, the image area R in the captured image P is set using the above-mentioned distance information Iz or machine learning, so that such an image area R can be easily set, which also makes it possible to achieve further speedup (further shortening of processing time).

[0061] <2. Modifications> Although the present disclosure has been described above by way of the embodiment, the present disclosure is not limited to this embodiment and various modifications are possible.

[0062] For example, the configuration (type, shape, arrangement, number, etc.) of each member in the vehicle 10 and the image processing device 12 is not limited to that described in the above embodiment. That is, the configuration of each of these members may be of other types, shapes, arrangements, numbers, etc. Furthermore, the values, ranges, magnitude relationships, etc. of the various parameters described in the above embodiment are not limited to those described in the above embodiment, and may be other values, ranges, magnitude relationships, etc.

[0063] Specifically, for example, in the above embodiment, the stereo camera 11 is configured to capture an image in front of the vehicle 10, but the present invention is not limited to this configuration, and for example, the stereo camera 11 may be configured to capture an image of the side or rear of the vehicle 10. Also, in the above embodiment, an example in which the stereo camera 11 is used has been described, but the present invention is not limited to this example, and for example, a monocular camera may be used to perform the various processes described in the above embodiment.

[0064] Also, for example, in the above embodiment, the various processes performed in the vehicle 10 and the image processing device 12 are described using specific examples, but are not limited to these specific examples. That is, these various processes may be performed using other methods. Specifically, for example, the one-dimensional processes of the Fourier transform and convolution calculation described above are not limited to the methods described in the above embodiment (one-dimensional processes along the x-axis direction), and may be one-dimensional processes along the y-axis direction or other directions (diagonal directions, etc.). Also, in the above embodiment, an example of a processing method using KCF is described, but is not limited to this example, and other processing methods may be used.

[0065] Furthermore, the series of processes described in the above embodiments may be performed by hardware (circuits) or software (programs). When performed by software, the software is composed of a group of programs for causing a computer to execute each function. Each program may be, for example, pre-installed in the computer and used, or may be installed in the computer from a network or a recording medium and used.

[0066] Further, in the above embodiment, an example in which the image processing device 12 is provided in a vehicle has been described; however, this is not limiting, and such an image processing device 12 may be provided, for example, in a moving body other than a vehicle, or in a device other than a moving body.

[0067] Furthermore, the various examples described above may be applied in any combination.

[0068] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0069] The present disclosure can also be configured as follows. (1) a region setting unit that sets one or more image regions in a captured image; an extraction unit that extracts a feature amount of a two-dimensional arrangement included in the image region of the captured image; a position estimation unit that estimates a position of a tracked object by performing a correlation calculation using a convolution calculation on the feature amount between the image regions of the captured images obtained in the previous and next frame periods along a time axis; Equipped with The position estimation unit is When performing the convolution operation, a Fourier transform is used for the feature amount, The Fourier transform and the convolution operation are respectively performed on the feature amounts arranged in the two-dimensional array, one dimension at a time. Image processing device. (2) The position estimation unit is performing one-dimensional processing of the Fourier transform and the convolution operation along a first direction of the feature amount, The results of the one-dimensional processing along the first direction are added to each other in a plurality of sets along a second direction different from the first direction, Executing the Fourier transform and the convolution operation on the feature quantities of the two-dimensional arrangement. The image processing device according to (1) above. (3) The position estimation unit is The correlation calculation is performed using a KCF (Kernelized Correlation Filter), The feature quantity indicating the color space is transformed into a frequency space by the Fourier transform. The image processing device according to (1) or (2) above. (4) The region setting unit sets the image region by using distance information generated based on a left image and a right image as the captured image obtained from a stereo camera, or by using machine learning. An image processing device according to any one of (1) to (3) above. (5) An image processing device according to any one of (1) to (4) above; a vehicle control unit that performs vehicle control using the estimated position of the tracked object provided by the position estimation unit; A vehicle equipped with. (6) One or more processors one or more memories communicatively coupled to the one or more processors; Equipped with The one or more processors: Setting one or more image regions in a captured image; extracting a feature amount of a two-dimensional arrangement included in the image region of the captured image; A correlation calculation is performed using a convolution calculation on the feature amount between the image regions of the captured images obtained in the previous and next frame periods along a time axis, thereby estimating the position of the tracked object. In addition to carrying out the above, When performing the convolution operation, a Fourier transform is used for the feature amount, The Fourier transform and the convolution operation are respectively performed on the feature amounts arranged in the two-dimensional array, one dimension at a time. Image processing device. [Explanation of symbols]

[0070] 10...vehicle, 11...stereo camera, 11L...left camera, 11R...right camera, 12...image processing device, 121...image memory, 122...distance information generation unit, 123...area setting unit, 124...feature extraction unit, 125...position estimation unit, 13...vehicle control unit, 19...windshield, 90...preceding vehicle, PL...left image, PR...right image, PIC...stereo image, P, P0 to Pn, Pn+1...captured image, R...image area, Iz...distance information, F, Fn, Fn+1...feature, Mc...correlation map, Pe...estimated position, x0...position coordinate, PX...pixel, t...time axis, t0 to tn...timing.

Claims

1. a region setting unit that sets one or more image regions in a captured image; an extraction unit that extracts a feature amount of a two-dimensional arrangement included in the image region of the captured image; a position estimation unit that estimates a position of a tracked object by performing a correlation calculation using a convolution calculation on the feature amount between the image regions of the captured images obtained in the previous and next frame periods along a time axis; Equipped with The position estimation unit is When performing the convolution operation, a Fourier transform is used for the feature amount, The Fourier transform and the convolution operation are respectively performed on the feature amounts arranged in the two-dimensional array, one dimension at a time. Image processing device.

2. The position estimation unit is performing one-dimensional processing of the Fourier transform and the convolution operation along a first direction of the feature amount, The results of the one-dimensional processing along the first direction are added to each other in a plurality of sets along a second direction different from the first direction, Executing the Fourier transform and the convolution operation on the feature quantities of the two-dimensional arrangement. The image processing device according to claim 1 .

3. The position estimation unit is The correlation calculation is performed using a KCF (Kernelized Correlation Filter), The feature quantity indicating the color space is transformed into a frequency space by the Fourier transform.

3. The image processing device according to claim 1 or 2.

4. The region setting unit sets the image region by using distance information generated based on a left image and a right image as the captured image obtained from a stereo camera, or by using machine learning.

4. The image processing device according to claim 1,

5. An image processing device according to any one of claims 1 to 4, a vehicle control unit that performs vehicle control using the estimated position of the tracked object provided by the position estimation unit; A vehicle equipped with.

Images