Detection device, vehicle control apparatus, detection method, and program

Abstract

To detect a moving object from an image at high speed.SOLUTION: Respective image feature amounts of multiple frames are generated. On the basis of the image feature amounts, a cost volume representing, at each depth, the reliability of matching between the multiple frames is generated. A feature amount representing a distribution of depth direction values in the cost volume is generated. A moving body in the image is detected on the basis of the feature amount representing the distribution of depth direction values.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a detection device, a vehicle control device, a detection method, and a program, and more particularly to detecting a moving object from an image. [Background technology]

[0002] In recent years, technologies have been proposed that use learning models to recognize objects in images. In particular, technologies have been studied that detect all moving objects in an image so that objects of classes that have not been trained by ordinary object detectors can be detected. For example, Non-Patent Document 1 achieves highly accurate moving object detection by calculating optical flow and performing ego flow suppression processing using FlowNet-2, and inputting the results into a ResNet-based neural network. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] M. Siam et al. "Video Class Agnostic Segmentation Benchmark for Autonomous Driving", Computer Vision and Pattern Recognition 2021, arXiv:2103.11015. [Non-patent document 2] J. Watson et al. "The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth", Computer Vision and Pattern Recognition 2021, arXiv:2104.14540. [Non-patent document 3] K. He et al. "Deep Residual Learning for Image Recognition", Computer Vision and Pattern Recognition 2015, arXiv:1512.03385. [Non-patent document 4] G. Xu et al. "Iterative Geometry Encoding Volume for Stereo Matching", Computer Vision and Pattern Recognition 2023, arXiv:2303.06615. [Non-Patent Document 5] T. Cheng et al. "Sparse Instance Activation for Real-Time Instance Segmentation", Computer Vision and Pattern Recognition 2022, arXiv:2203.12827. Summary of the Invention [Problem to be solved by the invention]

[0004] High-precision moving object detectors currently proposed, such as those described in Non-Patent Document 1, use large-scale models. On the other hand, when detecting moving objects in real time, such as in an autonomous driving system or an emergency avoidance system for a vehicle, it is desirable to use a lightweight model that is capable of high-speed operation while maintaining relatively high accuracy.

[0005] An object of the present invention is to quickly detect a moving object from an image. [Means for solving the problem]

[0006] A detection device according to an embodiment of the present invention has the following configuration: a first generating means for generating a feature amount of each image in a plurality of frames; a second generating means for generating a cost volume indicating the likelihood of matching between a plurality of frames for each depth based on the feature amount of the image; a third generating means for generating a feature quantity indicating a distribution of values ​​in a depth direction in the cost volume; a detection means for detecting a moving object in the image based on the feature amount indicating the distribution of values ​​in the depth direction; Equipped with. [Effects of the Invention]

[0007] It is possible to quickly detect moving objects from images. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a hardware configuration diagram of a detection device according to an embodiment. [Figure 2] FIG. 2 is a functional configuration diagram of a detection device according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a moving object detection model used in an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of the configuration of a three-dimensional convolutional network used in an embodiment. [Figure 5] 1 is a flowchart of a detection method according to an embodiment. [Figure 6] FIG. 1 is a functional configuration diagram of a vehicle according to an embodiment. [Figure 7] FIG. 10 is a diagram showing the relationship between the number of blocks in a network and detection accuracy. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] The configuration of the detection device 100 according to one embodiment of the present invention will be described in more detail below. The detection device 100 can detect moving objects based on captured images. The type of moving object is not particularly limited. For example, the moving object may include at least one of a vehicle, a person, an animal such as a dog or a cat, and a motorcycle.

[0011] The detection device 100 can be realized using a computer. Examples of the computer include a general-purpose desktop computer, a server, and an embedded device. For example, the functions of each processing unit of the detection device 100 shown in FIG. 2 can be realized by a computer. However, at least some of the processing units may be realized by dedicated hardware. Furthermore, the detection device may be configured by a plurality of information processing devices connected via a network, for example. For example, the functions of the detection device may be provided as a cloud service.

[0012] FIG. 1 is a diagram showing the basic configuration of a computer. In FIG. 1, a processor 101 is, for example, a CPU, and controls the operation of the entire computer. A memory 102 is, for example, a RAM, and temporarily stores programs, data, etc. A computer-readable storage medium 103 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. long-term. In this embodiment, a program that realizes the function of each unit, which is stored in the storage medium 103, is read into the memory 102. Then, the processor 101 operates in accordance with the program on the memory 102, thereby realizing the function of each unit.

[0013] 1, input interface 104 is an interface for acquiring information from an external device. Output interface 105 is an interface for outputting information to an external device. Bus 106 connects the above-mentioned components and enables data exchange.

[0014] Next, an example of the functional configuration of the detection device 100 according to an embodiment will be described with reference to Fig. 2. The detection device 100 includes a first generation unit 230, a second generation unit 240, a third generation unit 250, and a detection unit 260. The detection device 100 may further include an image acquisition unit 210, a determination unit 220, and an output unit 270.

[0015] The image acquisition unit 210 acquires images in a plurality of frames. For example, the determination unit acquires an image I in a frame at time t, which is a target for detecting a moving object. t and image I in the frame at time t-1 before time t. t-1 The plurality of frames may be frames at different times. For example, the plurality of frames may be consecutive frames.

[0016] In this embodiment, each image acquired by the image acquisition unit 210 is captured by the same imaging device, although the position of the imaging device may be different when each image is captured.

[0017] The determination unit 220 determines a change in posture of the imaging device that captured images between multiple frames. The determination unit 220 can determine a change in posture of the imaging device based on each image acquired by the image acquisition unit 210. The specific method is not particularly limited. For example, the determination unit 220 can determine a change in posture using a pose network 310 (described later). The determination unit 220 may also determine a change in posture using another method that utilizes matching between images. However, it is not essential for the determination unit 220 to determine a change in posture of the imaging device. For example, an imaging device may be installed in a vehicle. In this case, a control unit of the vehicle can determine a change in posture of the imaging device based on information indicating the movement of the vehicle, such as acceleration information acquired by a sensor. In such a case, the determination unit 220 may acquire information indicating a change in posture of the imaging device determined by another device. In this embodiment, a change in posture of the imaging device refers to at least one of a change in position (e.g., a translation vector) and a change in posture (e.g., a rotation matrix) of the imaging device.

[0018] The first generating unit 230 generates feature quantities for each image in a plurality of frames. The method for generating the feature quantities is not particularly limited. For example, the first generating unit 230 can generate feature quantities for each image using a neural network. As a specific example, the first generating unit 230 can generate feature quantities for each image by inputting each image into a Feature Network 320, which will be described later. The first generating unit 230 generates feature quantities for each image by inputting each image into a Feature Network 320, which will be described later, using common parameters. t and Image I t-1 In the following, the image I generated by the first generation unit 230 is t The feature map F 1 t In addition, the image I generated by the first generation unit 230 is called t-1 The feature map F 1 t-1 It is called.

[0019] The first generating unit 230 may generate multiple types of feature amounts of an image for further processing. For example, the first generating unit 230 generates a feature amount of an image I t As the feature of 1 t The feature map F is on a different scale from 2 t and feature map F 3 t can be further generated.

[0020] The second generation unit 240 generates a cost volume that indicates the likelihood of matching between multiple frames for each depth based on the image features. The cost volume indicates matching between two images or between feature maps for two images. More specifically, the cost volume indicates the accuracy of matching when matching is performed according to each depth value for each position in the spatial direction (vertical and horizontal directions of the image). In other words, the cost volume indicates the cost (or likelihood) for each depth value for each position. For example, for a specific position, the depth value corresponding to the smallest cost indicates the most likely depth to the subject appearing at that specific position.

[0021] In this embodiment, the second generator 240 generates a feature map F 1 t-1 and feature map F 1 t For example, the second generating unit 240 generates a cost volume based on the difference between the depth range d max ~d min One of the discrete depth values ​​contained in (d i ) and the ego-motion between each frame to obtain the source feature map (F 1 t-1 ) to the target feature map (F 1 t ) above. The ego-motion corresponds to the change in posture of the image capture device determined by the determination unit 220. Specifically, the second generation unit 240 warps the depth value d i A feature map F on the plane 1 t-1 to the viewpoint of the image capture device in the frame at time t-1 based on the ego-motion between the frames. Furthermore, the second generation unit 240 calculates the difference between the warped feature map and the target feature map. The second generation unit 240 performs this process within the depth range d max ~d minFor example, the second generating unit 240 can generate the cost volume using a warping process 330 and an l1 distance calculation process 340, which will be described later.

[0022] The depth range, i.e., the upper limit value dmax and the lower limit value dmin of the depth range, can be set by any method. The number of depth values ​​or the interval between depth values ​​can also be set by any method. These values ​​may be preset. For example, these values ​​can be set manually. In this embodiment, moving object detection is performed based on the distribution of costs in the depth direction, as described below. In this configuration, the depth range does not need to be set to include all depth values ​​up to each object within the imaging range. For example, it is not necessary to set the depth range to include the depth of the background. In one embodiment, the depth range can be set in advance based on the depth values ​​of moving objects in the image. For example, the depth range can be set to include each depth value up to most moving objects. The depth range can also be set depending on the use case. For example, when using an imaging device mounted on a vehicle to perform moving object detection for driving assistance purposes, the depth range can be set to include the depth of objects located at a distance that may cause traffic disruption. This configuration can improve the moving object detection speed by reducing the size of the cost volume while maintaining the moving object detection accuracy.

[0023] For example, a depth image can be obtained by inputting a captured image into a depth network such as ManyDepth. A depth range can be set based on the depth value distribution in a region of the depth image where a moving object exists. For example, the depth range may be set to match the range of the depth value distribution. Furthermore, for example, the depth range may be set to match the range of the depth value distribution after excluding outliers. The process of setting such a depth range can be performed using a depth image obtained for a typical captured image. That is, moving objects can be detected in multiple frames according to the specific depth range set in this way.

[0024] The second generator 240 can generate a four-dimensional cost volume. In one embodiment, the feature map F generated by the first generator 230 1 t and F 1 t-1 has dimensions of C×H×W. The warped feature map has dimensions of D×C×H×W. Here, D represents the number of depth values, C represents the number of channels, H represents the number of elements in the vertical direction, and W represents the number of elements in the horizontal direction. In this case, the first generation unit 230 generates a warped feature map and a target feature map (F 1 I t ) for each depth value. As a result, the first generating unit 230 generates a four-dimensional cost volume having dimensions D×C×H×W.

[0025] The distribution of costs in the depth direction at a specific spatial position indicated by the cost volume generated by the second generation unit 240 in this manner is considered. If the subject captured at this position does not move between frames, the cost is considered to be smallest at the most probable depth to the subject captured at this position. Furthermore, as the depth value increases or decreases, the cost is considered to gradually increase. Thus, if the subject does not move between frames, the cost tends to change regularly in the depth direction. On the other hand, if the subject captured at this position moves between frames, the cost changes irregularly compared to surrounding positions. This is because the warping process takes ego-motion into account, but does not take into account the movement of moving objects. Therefore, if the cost volume in a specific region has an irregular cost distribution in the depth direction, there is a high possibility that a moving object exists in this region. Therefore, in this embodiment, processing is performed to capture irregular values ​​of the cost volume.

[0026] The third generation unit 250 generates a feature indicating the distribution of depth-wise values ​​in the cost volume. By referring to such a feature, irregular cost distribution in the depth direction can be captured. The third generation unit 250 performs a process of aggregating information in the depth direction for the cost volume, thereby extracting a feature indicating the movement of the subject between frames. To aggregate the information in the depth direction, the third generation unit 250 can perform a depth-wise convolution process for the cost volume. For example, the third generation unit 250 can generate a feature using a network 350, which will be described later.

[0027] As described above, in one embodiment, the cost volume is a four-dimensional cost volume having dimensions D×C×H×W. The third generation unit 250 can generate a Channel-wise Motion Feature that aggregates depth-wise information for such a four-dimensional cost volume. For example, the third generation unit 250 can aggregate depth-wise information by inputting the cost volume into a network such as a three-dimensional CNN. The Channel-wise Motion Feature can have dimensions C×H×W. However, the values ​​of C, H, and W do not need to be consistent between the four-dimensional cost volume and the Channel-wise Motion Feature.

[0028] The configuration of the network used by the third generation unit 250 can be selected as appropriate. On the other hand, the third generation unit 250 may perform convolution processing on the cost volume using a network including a down sampling block and an up sampling block. In particular, by using a network including multiple down sampling blocks and multiple up sampling blocks, the accuracy of detecting moving objects is improved. Furthermore, by using a network including two down sampling blocks and two up sampling blocks, both high accuracy and low calculation cost can be achieved.

[0029] Furthermore, the third generator 250 may use a 3D normalization network, which further includes a feature map F 1 t The feature map F is on a different scale from 2 t and feature map F 3 t may be input. As such a network, the network shown in Fig. 4, which will be described later, can be used.

[0030] The detection unit 260 detects moving objects in an image based on features indicating the distribution of depth values. The detection unit 260 can generate a mask indicating moving objects in an image, for example, using an instance segmentation technique. In the instance segmentation technique, a mask is generated based on the idea that the activity of a feature map represents an instance. As described above, the third generation unit 250 aggregates depth information, and the feature map generated by the third generation unit 250 aggregates three-dimensional information related to each instance. In other words, since this feature map represents features related to the activity of each instance, the instance segmentation technique can be applied. The detection unit 260 can generate a mask indicating moving objects in an image, for example, using a segmentation decoder 360 (described later).

[0031] The output unit 270 outputs the detection result of the moving object. For example, the output unit 270 can output a mask indicating the moving object in the image generated by the detection unit 260. The output destination of the detection result is not particularly limited. For example, the output unit 270 may display the detection result on a display via the output interface 105. The output unit 270 may also output the detection result to another processing device. For example, the output unit 270 may output the detection result to a controller 607 that controls the operation of the vehicle 600, as will be described later.

[0032] As described above, the first generating unit 230, the second generating unit 240, the third generating unit 250, and the detecting unit 260 can perform each process using a learning model. An example of the learning model will be described below with reference to FIG. 3.

[0033] Image I in the frame at time t-1 obtained by the imaging device t-1 and image I in frame at time t t is input to the Pose Network 310. Image I t-1 and Image I tis a three-channel (e.g., RGB) image. The Pose Network 310 estimates the ego-motion of the image capture device from time t-1 to t. That is, the Pose Network 310 calculates the relative pose T between the pose of the image capture device at time t and the pose of the image capture device at time t-1. t→t-1 The Pose Network 310 may be the one described in Non-Patent Document 2.

[0034] Also, the image t-1 is input to the Feature Network 320. Similarly, image I t are also input to the Feature Network 320. The Feature Network 320 extracts image features from the input images using common weight parameters. t-1 From the feature map F 1 t-1 In addition, the feature network 320 generates an image I t From the feature map F 1 t , F 2 t , and F 3 t Generate a feature map F 1 , F 2 , and F 3 The resolutions of the input images are 1 / 4, 1 / 8, and 1 / 16, respectively. The first four layers of ResNet50 described in Non-Patent Document 3 can be used as the feature network 320. In the example of Figure 3, image I t and images t-1 The resolution of the feature map F is H pixels vertically and W pixels horizontally. 1 t and feature map F 1 t-1 The number of elements is H / 4 elements vertically and W / 4 elements horizontally.

[0035] Feature map F 1 t-1In the warping process 330, the depth value d is calculated based on the relative pose estimated by the Pose Network 310. i Using the plane, the feature map F 1 t-1 to the viewpoint of the image capture device at time t. In the warping process 330, multiple depth values ​​d i The warping results are obtained using each of the depth values ​​d i is the depth range d max ~d min As described above, the depth range can be set based on the expected depth value distribution of moving objects. dmax may be, for example, 4.0, and dmin may be, for example, 0.1.

[0036] In this example, the feature map F 1 t-1 The warping process 330 for image I yields a warped feature 335 with D×C×H / 4×W / 4 elements, where D is the number of depth values ​​and C is the number of channels. H and W are the dimensions of image I as described above. t-1 The Warped Feature 335 is expressed by the following formula: F t-1→t (d i )=F t-1 <proj(d i ,T t→t-1 ,K)> where proj() is the projection function, and the feature map F 1 t-1 In, d i gives the 2D coordinates at the depth of . <> is the sampling operator. K is a 3x3 camera internal parameter.

[0037] The Warped Feature 335 obtained in this way and the feature map F 1 t In the l1 distance calculation process 340, F t-1→t(d i ) and F 1 t The absolute distance between the depth value d i This is done every time. 4DCV(d i ,c,x,y)=|F t-1→t (d i )-F 1 t (c,x,y)| The 4DCV calculated in this way is called a four-dimensional cost volume 345. x and y are coordinate values ​​in the spatial direction. c represents the channel dimension. By performing such distance calculations for each depth value, spatial coordinate value, and channel, the four-dimensional cost volume 345 is obtained. Such a four-dimensional cost volume 345 can be generated by a method similar to that described in Non-Patent Document 2.

[0038] The four-dimensional cost volume 345 thus obtained is input to a network 350 that generates features indicating the distribution of values ​​in the depth direction. The network 350 then outputs a channel-wise motion feature 355. The channel-wise motion feature 355 is an aggregate of three-dimensional information for each channel. More specifically, the network 350 is a three-dimensional normalization network similar to that described in Non-Patent Document 4. The network 350 receives the four-dimensional cost volume 345 and a feature map F 2 t and F 3 t In this way, a Channel-wise Motion Feature 355 having C×H / 4×W / 4 elements is obtained from the four-dimensional cost volume 345 having D×C×H / 4×W / 4 elements.

[0039] FIG. 4 shows an example of the configuration of network 350. Network 350 is a lightweight 3D convolutional network. As shown in FIG. 4, network 350 has two down sampling blocks and two up sampling blocks. Each down sampling block includes two 3D convolution operations of size 3×3×3. The up sampling block is made up of a block that performs one 3D transposed convolution operation of size 4×4×4, and a block that performs a transposed convolution operation of size 4×4×4 and two 3D convolution operations of size 3×3×3. In addition, in the intermediate layer, a feature map F 2 t and F 3 t This 3D normalization network uses guided cost volume excitation. That is, a feature map F that matches the scale of the cost volume is calculated. i , and use it as a weight by aligning it in the channel dimension direction. By using such feature map weights, the cost volume C i The guided cost volume excitation for is expressed as follows: w=σ(f(F i )) C' i =w◎C i Here, f represents two-dimensional point-wise convolution, σ represents the sigmoid function, and ◎ represents the Hadamard product. In the final layer of the 3D normalization network, the depth dimension is set to 1, resulting in a Channel-wise Motion Feature355 with C×H / 4×W / 4 elements.

[0040] Finally, the Channel-wise Motion Feature 355 is input to an instance segmentation decoder 360. In this way, a moving object identification mask 365 is obtained as a moving object detection result. The segmentation decoder 360 can be an IAM-based decoder as described in Non-Patent Document 5.

[0041] The method of this embodiment does not require computationally intensive processes such as optical flow calculation and ego flow suppression, as described in Non-Patent Document 1. Furthermore, the third generator 250 can use relatively lightweight processes, such as a process using a 3D convolutional neural network, to generate features indicating the distribution of values ​​in the depth direction. Therefore, the method of this embodiment allows for high-speed detection of moving objects from images.

[0042] Below are shown the evaluation results of the moving object detection accuracy and detection speed using the learning model shown in Figure 3. The table below also shows the evaluation results when the learning models described in Non-Patent Document 1 and Non-Patent Document 5 were used. The KITTI and CityScapes datasets were used for model training. The evaluation was based on the SQ, RQ, and CAQ metrics, as well as the processing frame rate (FPS). As shown in the table below, the method of this embodiment achieved detection accuracy comparable to that of the method in Non-Patent Document 1 and superior to that of the method in Non-Patent Document 5. Meanwhile, since the number of parameters could be significantly reduced compared to Non-Patent Document 1, the detection speed was significantly improved and a practical frame rate could be achieved. [Table 1]

[0043] Figure 7 shows the number of parameters and detection accuracy when the number of blocks in the network 350 is changed. In Figure 7, the number of blocks indicates the number of sets of down sampling blocks and up sampling blocks. For example, if the network 350 has two down sampling blocks and two up sampling blocks, the number of blocks is two. Accuracy is represented by the SQ, RQ, and CAQ metrics and is shown as a line graph. The number of model parameters is the number of parameters other than the Pose Network 310 used by the learning model shown in Figure 3 and is shown as a bar graph. Figure 7(A) shows the evaluation results when training was performed using the KITTI dataset. As shown, increasing the number of blocks from 1 to 2 significantly improved detection accuracy. On the other hand, increasing the number of blocks from 2 to 3 did not significantly improve detection accuracy, but the number of parameters increased. As shown, setting the number of blocks to 2 achieved both high accuracy and low computational cost. Figure 7(B) shows the evaluation results when training was performed using the CityScapes dataset. In this case, too, by setting the number of blocks to 2, it was possible to achieve both high accuracy and low calculation cost.

[0044] The model shown in FIG. 3 can be trained according to a normal method. For example, a set of images (training images) in a plurality of frames and a moving object detection result (teaching data) can be used. Then, the model can be trained based on the error between the model evaluation result for the training images and the teaching data. The backpropagation method can be used for training the model. Note that a model trained in advance may be used as a model for determining a change in posture of an imaging device, such as Pose Network 310.

[0045] Next, each step of the detection method according to one embodiment will be described with reference to the flowchart in Fig. 5. In S510, the image acquisition unit 210 acquires each image in a plurality of frames as described above. In S520, the determination unit 220 determines a change in the posture of the imaging device that captured the images between the plurality of frames as described above.

[0046] At S530, the first generation unit 230 generates feature quantities for each image in multiple frames as described above. At S540, the second generation unit 240 generates a cost volume as described above. At S550, the third generation unit 250 generates feature quantities indicating the distribution of depth-wise values ​​in the cost volume as described above. At S560, the detection unit 260 detects a moving object in the image based on the feature quantities indicating the distribution of depth-wise values. At S570, the output unit 270 outputs the moving object detection result.

[0047] In the above embodiment, two frames of image I t and Image I t-1 On the other hand, the cost volume may be generated based on three or more frames of images. For example, the second generator 240 generates a warped feature map and a feature map F for the image of the additional frame. 1 t The cost volume may be generated further based on the difference between

[0048] [Installation in a vehicle] The detection device 100 according to the above embodiment can be mounted on a vehicle. For example, the vehicle can have an automatic driving system or an emergency avoidance system. These systems can control the operation of the vehicle based on the detection result of the moving object by the detection device 100. An example of the configuration of such a vehicle will be described below with reference to FIG. 6.

[0049] In the following example, a case will be described in which the detection device 100 is incorporated into a vehicle 600. On the other hand, the vehicle 600 may have a control module or an information processing device having the functions of the detection device 100.

[0050] The sensor unit 601 includes a camera (image capture device) that outputs an image captured in front of the vehicle 600 (or further to the side and rear of the vehicle). The sensor unit 601 may also include various sensors that output the acceleration, position information, steering angle, etc. of the vehicle 600.

[0051] The communication unit 602 is a communication device including, for example, a communication circuit, and communicates with the information processing server 150, a surrounding transportation system, etc. via mobile communication standardized, for example, as LTE, LTE-Advanced, or 5G. The communication unit 602 can acquire, from an external information processing server, trained parameters of a training model used by the detection device 100, etc.

[0052] The operation unit 603 includes operation members such as buttons and touch panels attached inside the vehicle 600, as well as members that accept inputs for driving the vehicle 600, such as a steering wheel and brake pedal.

[0053] The power supply unit 604 includes a battery such as a lithium ion battery, and supplies power to each unit in the vehicle 600. The power unit 605 includes, for example, an engine or a motor that generates power to run the vehicle.

[0054] The notification unit 606 can notify the driver with a predetermined sound such as a warning sound, or can notify the driver of a warning by other methods. For example, the notification unit 606 can issue a warning sound when it detects the possibility of a collision with a target on the road. Such a notification operation is performed based on the detection result of targets around the vehicle 600.

[0055] The controller 607 is an electronic circuit that executes control of the vehicle 600, including driving assistance for the vehicle 600. For example, the controller 607 can control the driving (acceleration) of the vehicle 600 by controlling a power unit. The controller 607 can also control the braking (deceleration) of the vehicle 600 by controlling a hydraulic device. Furthermore, the controller 607 can control the steering of the vehicle 600 by controlling an electric power steering device.

[0056] The controller 607 can also provide driving assistance for the vehicle 600. Examples of driving assistance include acceleration / deceleration assistance, lane keeping assistance, and lane change assistance. The acceleration / deceleration assistance is a driving assistance (ACC: Adaptive Cruise Control) that controls the acceleration / deceleration of the vehicle 600 within a predetermined vehicle speed while maintaining a distance from a preceding vehicle by controlling a power unit and a hydraulic device. The lane keeping assistance is a driving assistance (LKAS: Lane Keeping Assist System) that keeps the vehicle 600 within the lane by controlling an electric power steering device. The lane change assistance is a driving assistance (ALC: Auto Lane Changing, ALCA: Active Lane Change Assist) that changes the driving lane of the vehicle 600 to an adjacent lane by controlling an electric power steering device. The driving assistance may include a collision mitigation brake, an ABS function, traction control, and / or attitude control of the vehicle 600 that assists in avoiding a collision with an object on the road (e.g., a pedestrian or another vehicle) by controlling a hydraulic device. These driving assistances are performed based on the detection results of targets around the vehicle 600. On the other hand, the controller 607 may control the power unit, the hydraulic system, and the electric power steering system so that the vehicle 600 performs automatic driving to head to a destination or follow another vehicle based on map information. Such automatic driving is also performed based on the detection results of targets around the vehicle 600.

[0057] The detection device 100 can detect moving objects around the vehicle 600 as described above, based on the captured image captured by the sensor unit 601. The detection result is sent to the controller 607 or the alarm unit 606. The controller 607 controls the vehicle based on the detected moving objects. For example, the controller 607 can perform driving assistance or automatic driving based on the detection result of the moving objects. In this way, the combination of the detection device 100 and the controller 607 can function as a vehicle control device that controls the vehicle 600. Furthermore, the alarm unit 606 can perform an alarm operation based on the detection result of the moving objects.

[0058] More specifically, the controller 607 may include a route calculation unit 608 and a vehicle control unit 609. The route calculation unit 608 may calculate a travel route for the vehicle 600 based on moving objects detected around the vehicle 600 as described above. For example, the route calculation unit 608 may calculate a travel route that avoids other vehicles or other moving objects. The route calculation unit 608 may also predict the behavior of the moving object based on the detection result of the moving object and calculate a travel route based on the predicted behavior of the moving object. In this way, the route calculation unit 608 may calculate an optimal route or a risk-avoidance route based on the detection result of the moving object. Then, the vehicle control unit 609 may control the vehicle 600 based on the calculated travel route. As described above, the vehicle control unit 609 may control the vehicle 600 to travel based on the travel route by controlling the power unit, hydraulic system, electric power steering system, etc.

[0059] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0060] <Summary of the embodiment> The above-described embodiments disclose at least the following detection device, vehicle control device, detection method, and program.

[0061] 1. The detection device of the above embodiment is a first generating means (230) for generating a feature amount of each image in a plurality of frames; a second generating means (240) for generating a cost volume (345) that indicates the likelihood of matching between each of the images in the plurality of frames for each depth; a third generating means (250) for generating a feature (355) indicating a distribution of values ​​in the depth direction in the cost volume (345); a detection means (260) for detecting a moving object in the image based on a feature (355) indicating a distribution of values ​​in the depth direction; Equipped with. According to this configuration, moving objects can be detected from images at high speed.

[0062] 2. In the above embodiment, The cost volume (345) is a four-dimensional cost volume having dimensions in the vertical and horizontal directions of the image, the depth direction, and the channel direction. According to this configuration, it becomes easier to collect more detailed information in the depth direction from the cost volume, thereby improving the accuracy of moving object detection.

[0063] 3. In the above embodiment, The image capturing apparatus further includes a determination means (220) for determining a change in posture of the image capturing device that captured the images between the plurality of frames. According to this configuration, the method of the above embodiment can be used without a sensor for detecting changes in the posture of the imaging device.

[0064] 4. In the above embodiment, The second generating means (240) generates the cost volume (345) based on the difference between the feature map of the image of the first frame and the feature map of the image of the second frame, which are aligned based on the pose change and their respective depths. According to this configuration, it becomes easier to collect more detailed information in the depth direction from the cost volume, thereby improving the accuracy of moving object detection.

[0065] 5. In the above embodiment, The depth range is preset based on the depth values ​​of moving objects in the image. According to this configuration, moving objects can be detected from images at higher speed.

[0066] 6. In the above embodiment, The third generating means (250) generates a feature (355) indicating a distribution of values ​​in the depth direction and a distribution of values ​​in the channel direction in the cost volume (345). According to this configuration, the feature amount can indicate the movement of the subject in more detail, thereby improving the accuracy of moving object detection.

[0067] 7. In the above embodiment, The third generating means (250) performs a depth-wise convolution process on the cost volume (345). According to this configuration, it becomes easier to collect more detailed information in the depth direction from the cost volume, thereby improving the accuracy of moving object detection.

[0068] 8. In the above embodiment, The third generating means (250) performs convolution processing on the cost volume (345) using a network (350) including a Down Sampling Block and an Up Sampling Block. According to this configuration, it becomes easier to collect more detailed information in the depth direction from the cost volume, thereby improving the accuracy of moving object detection.

[0069] 9. In the above embodiment, The network (350) includes two Down Sampling Blocks and two Up Sampling Blocks. This configuration makes it possible to achieve both high detection accuracy and low calculation costs.

[0070] 10. In the above embodiment, The moving object includes at least one of a vehicle, a person, an animal, and a two-wheeled vehicle. According to this configuration, it is possible to obtain moving object detection results that are suitable for the purpose of vehicle control.

[0071] 11. The vehicle control device of the above embodiment is The detection device (100) of the above embodiment; a travel path calculation unit (608) that calculates a travel path of the vehicle based on the detected moving object; a vehicle control unit (609) that controls the vehicle based on the calculated travel route; Equipped with. According to this configuration, moving object detection results with higher real-time performance can be used, thereby improving the accuracy of vehicle control.

[0072] 12. The detection method of the above embodiment includes: A detection method performed by the detection device (100), generating feature quantities for each of the images in the plurality of frames (S530); A step (S540) of generating a cost volume (345) that indicates the likelihood of matching between each image in a plurality of frames for each depth; A step (S550) of generating a feature (355) indicating a distribution of values ​​in the depth direction in the cost volume (345); a step (S560) of detecting a moving object in the image based on the feature (355) indicating the distribution of values ​​in the depth direction; Includes. According to this configuration, moving objects can be detected from images at high speed.

[0073] 13. The program of the above embodiment Computer, a first generating means for generating a feature quantity of each image in a plurality of frames; a second generation means for generating a cost volume indicating the likelihood of matching between a plurality of frames for each depth based on the feature amount of the image; a third generating means for generating a feature quantity indicating a distribution of values ​​in a depth direction in the cost volume; and The image processing device functions as a detection means for detecting a moving object in the image based on the feature amount indicating the distribution of values ​​in the depth direction. According to this configuration, moving objects can be detected from images at high speed.

[0074] Although the embodiments of the invention have been described above, the invention is not limited to the above-described embodiments, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0075] 210: Image acquisition unit, 220: Determination unit, 230: First generation unit, 240: Second generation unit, 250: Third generation unit, 260: Detection unit, 270: Output unit

Claims

1. a first generating means for generating a feature quantity of each image in a plurality of frames; a second generating means for generating a cost volume indicating the likelihood of matching between a plurality of frames for each depth based on the feature amount of the image; a third generating means for generating a feature quantity indicating a distribution of values ​​in a depth direction in the cost volume; a detection means for detecting a moving object in the image based on the feature amount indicating the distribution of values ​​in the depth direction; A detection device comprising:

2. The detection device according to claim 1 , wherein the cost volume is a four-dimensional cost volume having dimensions in the vertical and horizontal directions of an image, a depth direction, and a channel direction.

3. The detection device according to claim 1 , further comprising a determination unit for determining a change in posture of an image capturing device that captured the images between the plurality of frames.

4. The detection device according to claim 3 , wherein the second generation means generates the cost volume based on a difference between a feature map of an image of a first frame and a feature map of an image of a second frame, which are aligned based on the pose change and their respective depths.

5. The detection device according to claim 1 , wherein the depth range is preset based on depth values ​​of moving objects in the image.

6. The detection device according to claim 1 , wherein the third generating means generates a feature quantity indicating a distribution of values ​​in a depth direction and a distribution of values ​​in a channel direction in the cost volume.

7. The detection device according to claim 1 , wherein the third generating means performs a depth-direction convolution process on the cost volume.

8. The detection device according to claim 1 , wherein the third generation means performs convolution processing on the cost volume using a network including a Down Sampling Block and an Up Sampling Block.

9. The detection device according to claim 8 , wherein the network includes two Down Sampling Blocks and two Up Sampling Blocks.

10. The detection device according to claim 1 , wherein the moving object includes at least one of a vehicle, a person, an animal, and a two-wheeled vehicle.

11. The detection device according to claim 1 ; a travel path calculation unit that calculates a travel path of the vehicle based on the detected moving object; a vehicle control unit that controls the vehicle based on the calculated travel route; A vehicle control device comprising:

12. A detection method performed by a detection device, generating feature quantities for each of the images in a plurality of frames; A step of generating a cost volume that indicates the likelihood of matching between each image in a plurality of frames for each depth; generating a feature value indicating a distribution of values ​​in a depth direction in the cost volume; detecting a moving object in the image based on the feature amount indicating the distribution of values ​​in the depth direction; A detection method comprising:

13. Computer, a first generating means for generating a feature quantity of each image in a plurality of frames; a second generating means for generating a cost volume indicating the likelihood of matching between a plurality of frames for each depth based on the feature amount of the image; a third generating means for generating a feature quantity indicating a distribution of values ​​in a depth direction in the cost volume; and A program for causing the program to function as a detection means for detecting a moving object in the image based on a feature that indicates the distribution of values ​​in the depth direction.

Images