Image processing apparatus and vehicle

Abstract

To provide an image processing apparatus and the like capable of reducing a processing time while suppressing a reduction of an object tracking accuracy due to an external factor.SOLUTION: An image processing apparatus according to an aspect of one embodiment of the present disclosure includes: a region setting unit that sets one or a plurality of image regions in an imaged image obtained from a stereo camera; a first extraction unit that extracts a first feature quantity indicating a sparse feature included in the image region of the imaged image; a second extraction unit that extracts a second feature quantity indicating a dense feature that is sharper than the sparse feature included in the image region of a priority image obtained from a priority camera; a first object tracking unit that performs determination whether or not object tracking using the first feature quantity can be performed for each of a left image and a right image; a second object tracking unit that performs object tracking using the second feature quantity in the priority image; and a switching unit that switches settings of the priority camera and the priority image when it is determined that the object tracking using the first feature quantity in the priority image cannot be performed in the first object tracking unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus that performs object tracking based on a captured image, and a vehicle equipped with such an image processing apparatus.

Background Art

[0002] The captured image obtained by the imaging device includes images of various objects. For example, Patent Document 1 discloses an image processing apparatus that performs object tracking based on such a captured image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in such an image processing apparatus, it is required to suppress a decrease in object tracking accuracy due to external factors and to shorten the processing time. It is desirable to provide an image processing apparatus capable of shortening the processing time while suppressing a decrease in object tracking accuracy due to external factors, and a vehicle equipped with such an image processing apparatus.

Means for Solving the Problems

[0005] An image processing apparatus according to an embodiment of the present disclosure includes, in a captured image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image, a region setting unit that sets one or more image regions, a first extraction unit that extracts a first feature amount indicating sparse features included in the image region of the captured image, a second extraction unit that extracts a second feature amount indicating dense features that are clearer than the sparse features included in the image region of the left image or the right image obtained from the left camera or the right camera set as the priority camera, a first object tracking unit that determines whether object tracking using the first feature amount is possible in each of the left image and the right image, a second object tracking unit that performs object tracking using the second feature amount in the priority image, and when it is determined in the first object tracking unit that object tracking using the first feature amount in the priority image is impossible, Among the stereo cameras, switch and set the camera that has not been set as the priority camera to the priority camera, and switch and set the captured image obtained from the camera that has not been set as the priority camera to the priority image a switching unit, and is provided with.

[0006] A vehicle according to an embodiment of the present disclosure includes the image processing apparatus according to an embodiment of the present disclosure, and a vehicle control unit that performs vehicle control using the result of object tracking using the second feature amount obtained from the second object tracking unit.

Brief Description of Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7A

Figure 7B

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order. 1. Embodiment (Example of setting an image area using distance information or machine learning) 2. Modification example

[0009] <1. Embodiment> [Configuration] FIG. 1 shows a schematic configuration example of a vehicle (vehicle 10) according to an embodiment of the present disclosure in a block diagram. FIG. 2 schematically shows an external configuration example of the vehicle 10 shown in FIG. 1 in a top view.

[0010] As shown in FIG. 1, the vehicle 10 includes a stereo camera 11, an image processing device 12, and a vehicle control unit 13. In FIG. 1, illustration of a driving power source (such as an engine or a motor) of the vehicle 10 is omitted. This vehicle 10 is composed of, for example, an electric vehicle such as a hybrid electric vehicle (HEV) or an electric vehicle (EV: Electric Vehicle), or a gasoline vehicle.

[0011] (A. Stereo camera 11) As shown in FIG. 2, for example, the stereo camera 11 is a camera that generates a set of images (left image PL and right image PR) having a parallax with each other by imaging the front of the vehicle 10. As shown in FIGS. 1 and 2, this stereo camera 11 has a left camera 11L and a right camera 11R.

[0012] The left camera 11L and the right camera 11R each include, for example, a lens and an image sensor. As shown in FIG. 2, for example, the left camera 11L and the right camera 11R are arranged at a predetermined distance along the width direction of the vehicle 10 near the upper part of the windshield 19 in the vehicle 10. These left camera 11L and right camera 11R are adapted to perform an imaging operation in synchronization with each other. Specifically, as shown in FIG. 1, the left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL includes a plurality of pixel values, and the right image PR includes a plurality of pixel values. These left image PL and right image PR constitute a stereo image PIC as shown in FIG. 1.

[0013] FIG. 3 shows an example of such a stereo image PIC. Specifically, FIG. 3(A) shows an example of the left image PL, and FIG. 3(B) shows an example of the right image PR. Note that x and y shown in FIG. 3 represent the x-axis and the y-axis, respectively. In this example, another vehicle (preceding vehicle 90) is traveling in front of the vehicle 10 on the road on which the vehicle 10 is traveling. The left camera 11L generates a left image PL by imaging the preceding vehicle 90, and the right camera 11R generates a right image PR by imaging the preceding vehicle 90.

[0014] The stereo camera 11 is adapted to generate a stereo image PIC including such left image PL and right image PR. Further, the stereo camera 11 is adapted to generate a series of stereo images PIC by performing an imaging operation at a predetermined frame rate (for example, 60 [fps]).

[0015] (B. Image processing device 12) The image processing device 12 is a device that performs various image processes (such as tracking processes of objects in front of the vehicle 10) based on the stereo image PIC supplied from the stereo camera 11. As shown in FIG. 1, this image processing device 12 includes an image memory 121, a distance information generation unit 122, a region setting unit 123, feature amount extraction units 124A and 124B, object tracking units 125A and 125B, a switching unit 126, and a movement estimation unit 127.

[0016] Such an image processing device 12 is configured to include, for example, one or more processors (CPU: Central Processing Unit) that execute a program, and one or more memories that are communicably connected to these processors. Further, such a memory is constituted by, for example, a RAM (Random Access Memory) that temporarily stores processing data, a ROM (Read Only Memory) that stores a program, and the like.

[0017] Note that the above-described feature amount extraction unit 124A corresponds to a specific example of the "first extraction unit" in the present disclosure, and the feature amount extraction unit 124B corresponds to a specific example of the "second extraction unit" in the present disclosure. Also, the object tracking unit 125A corresponds to a specific example of the "first object tracking unit" in the present disclosure, and the object tracking unit 125B corresponds to a specific example of the "second object tracking unit" in the present disclosure.

[0018] (Image Memory 121) As shown in FIG. 1, the image memory 121 is a memory that temporarily stores the left image PL and the right image PR included in the stereo image PIC. Further, the image memory 21 sequentially supplies at least one of the left image PL and the right image PR stored in this manner to the distance information generation unit 122 and the feature amount extraction units 124A and 124B as the captured image P (see FIG. 1).

[0019] (Distance Information Generation Unit 122) The distance information generation unit 122 generates distance information Iz by performing predetermined image processing including stereo matching processing, filtering processing, etc. based on the captured images P read from the image memory 121 (here, the left image PL and the right image PR) (see FIG. 1). Specifically, the distance information generation unit 122 generates a distance image including a plurality of pixel values based on these left image PL and right image PR. Each of the plurality of pixel values is, in this example, a disparity value. In other words, each of the plurality of pixel values corresponds to the distance to the point corresponding to each pixel in the three-dimensional real space. Note that this is not limited to this example, and for example, each of the plurality of pixel values may be a distance value indicating the distance to the point corresponding to each pixel in the three-dimensional real space. In this way, the distance information generation unit 122 is configured to generate distance information Iz, which is information indicating the distance to the point corresponding to each pixel.

[0020] (Region setting unit 123) The region setting unit 123 sets one or more image regions R in the captured image P based on the distance information Iz supplied from the distance information generation unit 122. Specifically, the region setting unit 123 identifies a plurality of pixels that are located close to each other and have substantially the same disparity value in the captured image P based on the distance information Iz, and sets a rectangular region including the plurality of pixels as the image region R. That is, when there is an object in the captured image P, the pixels in the region corresponding to the object are located close to each other and have substantially the same disparity value. Therefore, the region setting unit 123 is configured to set the image region R so as to surround the object by setting the image region R in this way.

[0021] FIG. 4 schematically shows an example of the image region R set by the region setting unit 123. In the example shown in this FIG. 4, in the captured image P (here, one of the left image PL and the right image PR), image regions R are set for two vehicles respectively. Note that the region setting unit 123 sets the image region R for a vehicle in this example, but this is not limited to this example, and for example, the image region R may also be set for a person, a guardrail, a wall, etc.

[0022] The information about the one or more image regions R set by the region setting unit 123 in this way is supplied to the feature extraction units 124A and 124B, respectively, as shown in FIG. 1.

[0023] In the example shown in FIG. 1, the region setting unit 123 sets the image region R using the distance information Iz, but this is not limited to this example. That is, the region setting unit 123 may identify an object in the captured image P using a trained model such as a DNN (Deep Neural Network), etc., and set the image region R, which is a rectangular region, by outputting the coordinates of the identified object. That is, the region setting unit 123 may set the image region R using machine learning.

[0024] (Feature extraction units 124A, 124B) The feature extraction unit 124A extracts sparse feature amounts FA (feature amounts indicating sparse features) included in one or more image regions R in the captured image P (here, one of the left image PL and the right image PR) (see FIG. 1).

[0025] The feature extraction unit 124B extracts dense feature amounts FB (feature amounts indicating dense features) included in one or more image regions R in the captured image P (here, one of the left image PL and the right image PR) (see FIG. 1). Specifically, the feature extraction unit 124B extracts the dense feature amounts FB included in the image region R in the left image PL or the right image PR set as the priority image Pp. Further, the feature extraction unit 124B extracts the dense feature amounts FB by diverting the sparse feature amounts FA extracted in the feature extraction unit, although details will be described later (see FIG. 1).

[0026] Here, the above-mentioned priority image Pp is the captured image P obtained from the camera (left camera 11L or right camera 11R) set as the priority camera 11p in the stereo camera 11. Such a priority camera 11p and priority image Pp are each set in advance before use, and setting changes (setting switching) can be made at any time by the switching process described later in the switching unit 126.

[0027] Also, the sparse feature in the above-mentioned sparse feature amount FA is a rough feature compared to the dense feature in the dense feature amount FB. Conversely, the dense feature in the dense feature amount FB is a distinct feature compared to the sparse feature in the sparse feature amount FA. Although details will be described later, in object tracking using the dense feature amount FB, although high-precision object tracking is possible, the processing time becomes long or the processing cost becomes large. Therefore, in the present embodiment, as will be described later, by properly using these two types of feature amounts, both object tracking accuracy and processing time (processing cost) are made compatible.

[0028] Note that the sparse feature amount FA corresponds to a specific example of the "first feature amount" in the present disclosure, and the dense feature amount FB corresponds to a specific example of the "second feature amount" in the present disclosure.

[0029] Here, FIG. 5A schematically shows an example of the sparse feature amount FA, and FIG. 5B schematically shows an example of the dense feature amount FB.

[0030] First, in the sparse feature amount FA shown in FIG. 5A, four pixels PX are arranged in a matrix (2 vertically × 2 horizontally), and the pixel values at each pixel PX are pixel values PXa, PXb, PXc, and PXd. On the other hand, in the dense feature amount FB shown in FIG. 5B, 16 pixels PX are arranged in a matrix (4 vertically × 4 horizontally). Note that the difference in the number of pixels PX between such a sparse feature amount FA and a dense feature amount FB is caused by, for example, the difference in the size of the filter (number of parameters) applied during the extraction process using the DNN learned model described later and the difference in the size of the stride (application range of the convolution operation). Also, in this dense feature amount FB, among these 16 pixels PX, some (4) of the pixels PX have the above-described pixel values PXa, PXb, PXc, and PXd. That is, in this dense feature amount FB, as described above, the sparse feature amount FA (pixel values therein) is reused. Note that among this dense feature amount FB, for the remaining pixels PX (in the example of FIG. 5B, the remaining 12 pixels PX excluding the pixels PX showing the pixel values PXa, PXb, PXc, and PXd), the pixel values are newly acquired.

[0031] The feature amount extraction units 124A and 124B are each configured to extract a sparse feature amount FA or a dense feature amount FB, for example, using a learned model of a DNN. In that case, for example, the neural networks in the feature amount extraction units 124A and 124B each have a plurality of convolution layers and a plurality of pooling layers.

[0032] The sparse feature amount FA extracted by the feature amount extraction unit 124A in this way is supplied to the object tracking unit 125A and the feature amount extraction unit 124B, respectively (see FIG. 1). Also, the dense feature amount FB extracted by the feature amount extraction unit 124B is supplied to the object tracking unit 125B (see FIG. 1).

[0033] (Object tracking units 125A, 125B) The object tracking unit 125A determines, for each of the left image PL and the right image PR, whether object tracking using the sparse feature amount FA is feasible, and supplies the determination result JA of such determination to the switching unit 126 (see FIG. 1). The determination as to whether such object tracking is feasible is made, for example, based on whether the maximum value in the correlation value between the image of the object (object to be tracked) acquired in the immediately preceding frame period and the image of the object acquired in the current frame period is less than a predetermined threshold value. Alternatively, by recording the coordinates on the image of the object to be tracked, the movement status and the coordinates after movement of the object to be tracked are estimated, and the determination as to whether object tracking is feasible is made based on the degree of deviation between the estimated value of the coordinates after movement and the updated value of the actual coordinates.

[0034] The object tracking unit 125B performs object tracking using the dense feature amount FB in the above-described priority image Pp, and supplies the result of such object tracking (tracking position TB of the object to be tracked) to the movement estimation unit 127 and the vehicle control unit 13, respectively (see FIG. 1).

[0035] These object tracking units 125A and 125B perform object tracking by estimating the position of the object to be tracked for each captured image P in each frame period using the sparse feature amount FA or the dense feature amount FB. Specifically, the position (coordinates) where the above-described correlation value is the largest (indicating the maximum value) is estimated as the position of the object to be tracked and updated as needed. Further, such object tracking is performed using, for example, an object tracking method using a CNN (Convolutional Neural Network) such as KCF (Kernelized Correlation Filter), template matching, or Siamese Network.

[0036] (Switching unit 126) When it is determined in the object tracking unit 125A that object tracking using the sparse feature amount FA in the priority image Pp is impossible, the switching unit 126 switches the settings of the priority camera 11p and the priority image Pp, respectively. Specifically, the switching unit 126 performs such setting switching processing using the determination result JA supplied from the object tracking unit 125A (see FIG. 1). Further, such switching processing of the settings of the priority camera 11p and the priority image Pp is performed by supplying a switching signal Ss from the switching unit 126 to the feature amount extraction unit 124B, the object tracking unit 125B, and the movement estimation unit 127, respectively (see FIG. 1). Note that the details of such setting switching processing will be described later (FIGS. 6, 7A, 7B). Also, when performing such setting switching processing, the switching unit 126 compensates for the parallax amount between the left image PL and the right image PR using, for example, various general methods.

[0037] (Movement Estimation Unit 127) Based on the object tracking result (tracking position TB of the object to be tracked) supplied from the object tracking unit 125B and the distance information Iz supplied from the distance information generation unit 122, the movement estimation unit 127 estimates the three-dimensional movement operation of the object to be tracked (see FIG. 1). Specifically, the movement estimation unit 127 estimates the speed VB (moving speed) during such three-dimensional movement operation and supplies it to the vehicle control unit 13 (see FIG. 1). Also, based on the tracking position TB and the distance information Iz, the movement estimation unit 127 estimates the three-dimensional coordinates of the object to be tracked, and obtains the difference in the three-dimensional coordinates between the current frame period and the immediately preceding frame period, thereby estimating the above-described three-dimensional speed VB.

[0038] Further, when the switching unit 126 performs the above-described switching process (the switching process for setting the priority camera 11p and the priority image Pp), the movement estimation unit 127 performs, for example, the following correction process. That is, when the switching signal Ss is supplied from the switching unit 126, the movement estimation unit 127 corrects, using, for example, linear approximation or the like, the movement error during the three-dimensional movement operation of the tracking target object that occurs due to the switching of the setting of the priority image Pp. Specifically, the movement estimation unit 127 estimates the movement operation of the tracking target object by approximating, using a linear function, a quadratic function, a cubic function, or the like, with the data point group before the setting switch (the three-dimensional coordinates of the tracking target object) and the data point group after the setting switch. As a result, the movement error during the three-dimensional movement operation of the tracking target object that occurs due to the switching of the setting of the priority image Pp is corrected. Note that, in the curve fitting during the above-described approximation, a plurality of methods such as the least squares method are used.

[0039] (C. Vehicle control unit 13) The vehicle control unit 13 performs various vehicle controls in the vehicle 10 by using the result of object tracking (the tracking position TB of the tracking target object) supplied from the object tracking unit 125B and the speed VB (the three-dimensional movement speed of the tracking target object) supplied from the movement estimation unit 127 (see FIG. 1). Specifically, the vehicle control unit 13 performs, for example, the travel control of the vehicle 10 and the operation control of various members in the vehicle 10 based on the information of these tracking position TB and speed VB.

[0040] Such a vehicle control unit 13 includes, similarly to the image processing device 12, for example, one or more processors (CPUs) that execute a program and one or more memories that are communicably connected to these processors. Also, such a memory is similarly configured by, for example, a RAM that temporarily stores processing data and a ROM that stores a program, as in the image processing device 12.

[0041] [Operation and action / effect] Next, the operations, actions, and effects in the present embodiment will be described in detail.

[0042] (A. Regarding the decrease in object tracking accuracy due to external factors, etc.) First, with reference to FIGS. 6(A) and 6(B), the decrease in object tracking accuracy due to external factors, etc. will be described. FIG. 6(A) shows an example of a stereo image PIC1 (left image PL1 and right image PR1) in a certain frame period. Further, FIG. 6(B) shows an example of a stereo image PIC2 (left image PL2 and right image PR2) in a frame period after this stereo image PIC1 (refer to the dashed arrow t).

[0043] Generally, during the running of a vehicle, for example, as in the right image PR2 shown in FIG. 6(B), imaging by a camera may be hindered due to obstacles 8 such as wipers or water droplets (external factors that cannot be avoided during running). Specifically, in the example of the right image PR2 in FIG. 6(B), the images included in a part of the plurality of image regions R (the image region R indicated by the dashed line) are hidden by this obstacle 8, and tracking of the object (tracking target object) located within that image region R may become difficult. As a result, if left as it is, the tracking accuracy of this tracking target object will decrease.

[0044] Also, in an in-vehicle CPU, generally, since the required processing time is very short, speeding up the processing (shortening the processing time) during object tracking is desired. Especially in the present embodiment, as described above, in object tracking using a dense feature amount FB, although high-precision object tracking is possible, the processing time becomes long or the processing cost becomes large, so shortening such a processing time is important.

[0045] From these facts, it can be said that when performing object tracking based on an imaging image by a camera, it is desirable to shorten the processing time (reduce the processing cost) while suppressing the decrease in object tracking accuracy due to the above-described external factors.

[0046] (B. Switching Process, etc. of the Present Embodiment) Therefore, in the vehicle 10 of the present embodiment, the image processing apparatus 12 is configured to perform each process described below (such as the switching process of setting the priority camera 11p and the priority image Pp described above).

[0047] Hereinafter, with reference to FIGS. 7A and 7B in addition to FIGS. 1 to 6, an example of the above-described switching process, etc. in the present embodiment will be described in detail. FIGS. 7A and 7B respectively show an example of the above-described switching process, etc. in the present embodiment in the form of a flowchart. In the following, an example of such a switching process, etc. will be described while referring to the examples of the stereo images PIC1 and PIC2 shown in FIGS. 6(A) and 6(B) described above.

[0048] In the processing example shown in FIGS. 7A and 7B, first, the stereo camera 11 images the front of the vehicle 10 to generate a stereo image PIC (left image PL and right image PR) (step S100 in FIG. 7A). Specifically, for example, stereo images PIC1 (left image PL1 and right image PR1), stereo images PIC2 (left image PL2 and right image PR2), etc. as shown in FIGS. 6(A) and 6(B) are generated.

[0049] Next, the image memory 21 in the image processing apparatus 12 temporarily stores the stereo image PIC (left image PL and right image PR) generated in this way as the captured image P (step S101). Subsequently, in the region setting unit 123, it is determined whether or not the image region R in the captured image P has been set (step S102). Here, if it is determined that such an image region R has been set (step S102: Y), the process proceeds to step S104 described later.

[0050] On the other hand, when it is determined that such an image region R has not been set (step S102: N), next, the region setting unit 123 sets one or more image regions R in the captured image P using the above-described method (such as distance information Iz or machine learning) (step S103). Then, in step S104 next, the feature amount extraction unit 124A extracts the sparse feature amount FA described above included in the image region R of the captured image P.

[0051] Subsequently, the object tracking unit 125A determines whether object tracking using the sparse feature amount FA is possible in the above-described priority image Pp (left image PL or right image PR) (step S105). Here, when it is determined that object tracking using the sparse feature amount FA is possible in the priority image Pp (step S105: Y), the process proceeds to step S108 (FIG. 7B) described later.

[0052] On the other hand, when it is determined that object tracking using the sparse feature amount FA is impossible in the priority image Pp (step S105: N), the following occurs. That is, in this case, subsequently, the object tracking unit 125A determines whether object tracking using the sparse feature amount FA is possible in the other captured image P (the image different from the priority image Pp among the left image PL and the right image PR) (step S106 in FIG. 7B). Here, when it is determined that object tracking using the sparse feature amount FA is also impossible in such the other captured image P (step S106: N), the object tracking ends, and the series of processing examples shown in FIGS. 7A and 7B also ends. That is, when it is determined that object tracking using the sparse feature amount FA is impossible in both the left image PL and the right image PR in the object tracking unit 125A, the processing in step S109 described later (object tracking using the dense feature amount FB in the object tracking unit 125B) will not be performed.

[0053] On the other hand, in the case where it is determined that object tracking using sparse feature amount FA is executable in the above-described other captured image P (step S106: Y), the following occurs. That is, in this case, next, the switching unit 126 performs switching processing of the setting of the above-described priority camera 11p and priority image Pp (step S107). Thereafter, the process proceeds to step S108 described later.

[0054] Here, explaining with the examples of FIGS. 6(A) and 6(B) described above, the above-described switching process is specifically performed as follows. That is, in this example, first, in the stereo image PIC1 shown in FIG. 6(A), among the left image PL1 and the right image PR1, the right image PR1 is set as the priority image Pp. That is, as the priority camera 11p, the right camera 11R among the left camera 11L and the right camera 11R is set.

[0055] Here, in the stereo image PIC2 obtained in a certain subsequent frame period, for example, as shown in FIG. 6(B), in the right image PR2 set as the priority image Pp, due to the above-described external factor (obstacle 8), the following occurs. That is, in the right image PR2 as this priority image Pp, the images included in a part (image region R indicated by the broken line) of the plurality of image regions R are hidden by this obstacle 8, and tracking of the object (tracking target object) located within that image region R may become difficult.

[0056] Therefore, in this case, the switching unit 126 performs switching processing of the setting of the above-described priority camera 11p and priority image Pp. Specifically, in this case, the switching unit 126 switches the setting of the priority camera 11p from the right camera 11R to the left camera 11L and switches the setting of the priority image Pp from the right image PR2 to the left image PL2 (refer to the broken-line arrow SW in FIG. 6(B)). In the left image PL2 after such switching of the setting of the priority image Pp, for example, as shown in FIG. 6(B), unlike the right image PR2, no external factor (obstacle 8) has occurred. For this reason, the possibility that tracking of the tracking target object becomes difficult as described above is avoided.

[0057] Subsequently, in the above-described step S108 (FIG. 7B), the feature quantity extraction unit 124B extracts the above-described dense feature quantity FB included in the image area R of the priority image Pp set at that time. Next, the object tracking unit 125B performs object tracking (step S109) using the dense feature quantity FB extracted in step S108 in the priority image Pp set at that time.

[0058] Next, the movement estimation unit 127 estimates the speed VB during the three-dimensional movement operation of the object to be tracked based on the tracking position TB of the object to be tracked described above and the distance information Iz (step S110). Also, when the setting switching of the above-described priority image Pp (the switching process in step S107) is performed at this time, the movement estimation unit 127 performs the above-described correction process. That is, in that case, the movement estimation unit 127 corrects the movement error during the three-dimensional movement operation of the object to be tracked that occurs due to the setting switching of the priority image Pp.

[0059] Subsequently, the vehicle control unit 13 uses the tracking position TB of the object to be tracked described above and the speed VB estimated in step S110 to perform various vehicle controls in the vehicle 10 (the above-described running control of the vehicle 10, the operation control of various members, etc.) (step S111).

[0060] Thus, the series of processing examples shown in FIGS. 7A and 7B is completed.

[0061] (C. Operation and Effect) In this way, in the present embodiment, in the image region R of the priority image Pp among the captured images P, dense feature amounts FB (clearer than the sparse feature amounts FA) are extracted, and object tracking is performed using the dense feature amounts FB in the priority image Pp. As a result, the accuracy of object tracking is higher than in the case of object tracking using the sparse feature amounts FA. Also, since object tracking is performed using the priority image Pp set as one of the left image PL and the right image PR, the burden of the tracking process is reduced compared to the case of object tracking using both of these images (the left image PL and the right image PR).

[0062] Further, in the present embodiment, when it is determined that object tracking using the sparse feature amounts FA is impossible, the settings of the priority camera 11p and the priority image Pp are switched respectively. Thereby, for example, even when the above-described external factors occur in the priority camera 11p or the priority image Pp, it is possible to avoid the object tracking being hindered due to the external factors during the object tracking using the priority image Pp. Also, when object tracking using the sparse feature amounts FA is possible, since object tracking using the above-described dense feature amounts FB can also be surely and easily performed, the processing burden during object tracking using the dense feature amounts FB is reduced also in this respect.

[0063] From these points, in the present embodiment, it is possible to reduce the processing time (reduce the processing cost) while suppressing a decrease in object tracking accuracy due to external factors.

[0064] Further, in the present embodiment, the sparse feature amounts FA extracted by the feature amount extraction unit 124A are diverted to extract the dense feature amounts FB in the feature amount extraction unit 124B, so the following is achieved. That is, the dense feature amounts FB can be easily extracted, and it is possible to further shorten the processing time.

[0065] Furthermore, in the present embodiment, when the setting switching of the above-described priority image Pp is performed, the movement error during the three-dimensional movement operation of the tracking target object caused by such setting switching of the priority image Pp is corrected, so the following is obtained. That is, by correcting the movement error caused by such setting switching of the priority image Pp, it becomes possible to improve the estimation accuracy of the speed VB and the like in the movement estimation unit 127.

[0066] In addition, in the present embodiment, since the image region R in the captured image P is set using the above-described distance information Iz or machine learning, such an image region R can be easily set, and it becomes possible to further shorten the processing time.

[0067] Also, in the present embodiment, when it is determined that object tracking using the sparse feature amount FA cannot be performed in both the left image PL and the right image PR, object tracking using the dense feature amount FB in the object tracking unit 125B is not performed, so the following is obtained. That is, when object tracking using the sparse feature amount FA cannot be performed, the probability that object tracking using the dense feature amount FB cannot be performed is high. Therefore, when it is impossible to perform in both images, by ending the object tracking, an increase in the useless processing burden can be suppressed. As a result, it becomes possible to further shorten the processing time.

[0068] <2. Modification Example> As described above, the present disclosure has been described with reference to the embodiments, but the present disclosure is not limited to these embodiments, and various modifications are possible.

[0069] For example, the configurations (form, shape, arrangement, number, etc.) of the respective members in the vehicle 10 and the image processing apparatus 12 are not limited to those described in the above embodiment. That is, the configurations of these respective members may be other forms, shapes, arrangements, numbers, etc. Also, the values, ranges, magnitude relationships, etc. of the various parameters described in the above embodiment are not limited to those described in the above embodiment, and may be other values, ranges, magnitude relationships, etc.

[0070] Specifically, for example, in the above embodiment, the stereo camera 11 was configured to image the front of the vehicle 10, but the configuration is not limited to this. For example, the stereo camera 11 may be configured to image the side or rear of the vehicle 10.

[0071] Also, for example, in the above embodiment, an example in the case where the image processing device 12 is provided with the movement estimation unit 127 was described, but this example is not limited thereto. For example, the movement estimation unit 127 may not be provided in the image processing device 12.

[0072] Furthermore, in the above embodiment, various processes performed in the vehicle 10 and the image processing device 12 were described with specific examples, but these specific examples are not limiting. That is, these various processes may be performed using other methods. Specifically, for example, for the switching process of the priority camera 11p and the priority image Pp described above, the method described in the above embodiment is not limiting. Also, for example, in the above embodiment, an example in the case of extracting the dense feature amount FB by diverting the sparse feature amount FA was described, but this example is not limited thereto. For example, the dense feature amount FB may be extracted without diverting the sparse feature amount FA.

[0073] In addition, the series of processes described in the above embodiment may be performed by hardware (circuit) or may be performed by software (program). When performed by software, the software is composed of a group of programs for causing a computer to execute each function. Each program may be, for example, pre-installed in the above computer and used, or may be installed from a network or a recording medium into the above computer and used.

[0074] Also, in the above embodiment, an example in the case where the image processing device 12 is provided in the vehicle was described, but this example is not limiting. Such an image processing device 12 may be provided, for example, in a moving body other than a vehicle or in a device other than a moving body.

[0075] Furthermore, the various examples described so far may be applied in any combination.

[0076] Note that the effects described in this specification are merely illustrative and not limiting, and there may be other effects.

[0077] In addition, the present disclosure can also adopt the following configurations. (1) In an imaging image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image, a region setting unit that sets one or more image regions, A first extraction unit that extracts a first feature amount indicating sparse features included in the image region of the imaging image, A second extraction unit that extracts a second feature amount indicating dense features that are clearer than the sparse features, included in the image region in the left image or the right image as a priority image, obtained from the left camera or the right camera set as the priority camera, A first object tracking unit that determines, in each of the left image and the right image, whether object tracking using the first feature amount is feasible, A second object tracking unit that performs object tracking using the second feature amount in the priority image, A switching unit that switches the setting of the priority camera and the priority image respectively when it is determined in the first object tracking unit that object tracking using the first feature amount in the priority image is infeasible An image processing apparatus comprising the same. (2) The second extraction unit extracts the second feature amount by diverting the first feature amount extracted by the first extraction unit The image processing apparatus according to (1) above. (3) Based on the tracking position of the object to be tracked obtained by object tracking using the second feature amount and the distance information generated based on the left image and the right image, a movement estimation unit that estimates a three-dimensional movement operation of the object to be tracked is further provided. The movement estimation unit When the priority image is set and switched by the switching unit, corrects the movement error during the three-dimensional movement operation of the object to be tracked that occurs due to the setting and switching of the priority image. The image processing apparatus according to the above (1) or (2). (4) The area setting unit sets the image area based on the distance information generated based on the left image and the right image, or using machine learning. The image processing apparatus according to any one of the above (1) to (3). (5) In the first object tracking unit, when it is determined that object tracking using the first feature amount is impossible in both the left image and the right image, object tracking using the second feature amount in the second object tracking unit is not performed. The image processing apparatus according to any one of the above (1) to (4). (6) The image processing apparatus according to any one of the above (1) to (5), and a vehicle control unit that performs vehicle control using the result of object tracking using the second feature amount obtained from the second object tracking unit. A vehicle provided with the same. (7) One or more processors and One or more memories communicably connected to the one or more processors, and is provided with The one or more processors set one or more image areas in a captured image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image, and Extracting a first feature amount indicating sparse features included in the image region of the captured image; Extracting a second feature amount indicating dense features that are clearer than the sparse features included in the image region in the left image or the right image as the priority image obtained from the left camera or the right camera set as the priority camera; Determining, in the left image and the right image respectively, whether object tracking using the first feature amount is feasible; Performing object tracking using the second feature amount in the priority image; When it is determined that object tracking using the first feature amount in the priority image is infeasible, switching the priority camera and the setting of the priority image respectively; Performing An image processing apparatus.

Explanation of Signs

[0078] 10…Vehicle, 11…Stereo camera, 11L…Left camera, 11R…Right camera, 11p…Priority camera, 12…Image processing apparatus, 121…Image memory, 122…Distance information generation unit, 123…Region setting unit, 124A, 124B…Feature amount extraction units, 125A, 125B…Object tracking units, 126…Switching unit, 127…Movement estimation unit, 13…Vehicle control unit, 19…Front windshield, 8…Obstacle, 90…Leading vehicle, PL, PL1, PL2…Left images, PR, PR1, PR2…Right images, PIC, PIC1, PIC2…Stereo images, P…Captured image, Pp…Priority image, R…Image region, Iz…Distance information, FA…Sparse feature amount, FB…Dense feature amount, JA…Judgment result, Ss…Switching signal, TB…Tracking position, VB…Speed, PX…Pixel, PXa~PXd…Pixel values.

Claims

1. In a captured image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image, a region setting unit that sets one or more image regions, a first extraction unit that extracts a first feature amount indicating sparse features included in the image region of the captured image, a second extraction unit that extracts a second feature amount indicating dense features that are clearer than the sparse features, included in the image region in the left image or the right image as a priority image, obtained from the left camera or the right camera set as the priority camera, a first object tracking unit that determines, in each of the left image and the right image, whether object tracking using the first feature amount is possible, a second object tracking unit that performs object tracking using the second feature amount in the priority image, a switching unit that, when it is determined in the first object tracking unit that object tracking using the first feature amount in the priority image is impossible, switches and sets, among the stereo cameras, the camera not set as the priority camera to the priority camera, and switches and sets the captured image obtained from the camera not set as the priority camera to the priority image An image processing apparatus comprising the above.

2. The second extraction unit extracts the second feature amount by diverting the first feature amount extracted by the first extraction unit. The image processing apparatus according to claim 1.

3. Further provided is a movement estimation unit that estimates a three-dimensional movement operation of the tracking target object based on the tracking position of the tracking target object obtained by object tracking using the second feature amount and the distance information generated based on the left image and the right image, The movement estimation unit, when the setting of the priority image is switched by the switching unit, corrects the movement error during the three-dimensional movement operation of the tracking target object caused by the switching of the setting of the priority image. The image processing apparatus according to claim 1 or claim 2.

4. The region setting unit sets the image region based on the distance information generated based on the left image and the right image, or using machine learning. The image processing apparatus according to any one of claims 1 to 3.

5. In the first object tracking unit, when it is determined that object tracking using the first feature amount cannot be executed in both the left image and the right image, object tracking using the second feature amount in the second object tracking unit is disabled. The image processing apparatus according to any one of claims 1 to 4. **Claim 6** An image processing apparatus according to any one of claims 1 to 5, and a vehicle control unit that performs vehicle control using the result of object tracking using the second feature amount obtained from the second object tracking unit A vehicle comprising the same.

Images