Information processing device, information processing method, and information processing program

Abstract

The information processing device is provided with: a plurality of stereo cameras arranged so that directions of their baseline lengths intersect with each other; a depth estimation unit that estimates the depth of an object included in a captured image based on the captured image captured by the plurality of stereo cameras; and an object detection unit that detects the object based on the depth estimated by the depth estimation unit and the reliability of the depth, the reliability of the depth being determined according to the angle of the direction of an edge line of the object relative to the direction of the baseline length of the plurality of stereo cameras.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. Background Art

[0002] There is known a technology that detects an object from an image captured by a stereo camera attached to an unmanned mobile body such as a drone.

[0003] For example, Patent Document 1 discloses a ranging system that includes multiple ranging systems including stereo cameras and uses the minimum value among multiple ranging values obtained by the respective ranging systems as the final ranging value. Furthermore, Patent Document 2 discloses a system that includes multiple stereo cameras and switches the stereo camera to be used based on the movement of the vehicle's windshield wipers.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: JP 2018-146457 A

[0007] Patent Document 2: JP 2018-32986 A Summary of the Invention

[0008] Technical issues

[0009] In object detection using stereo cameras, for example, the distance between the camera and the object is measured based on the parallax of the object as seen from the right and left cameras. However, when the object being measured extends in the direction of the stereo camera's baseline length, distance measurement becomes difficult.

[0010] In this regard, the present disclosure proposes an information processing device, an information processing method, and an information processing program that are capable of detecting an object with high accuracy.

[0011] Solution to the problem

[0012] According to the present disclosure, there is provided an information processing device including: a plurality of stereo cameras arranged so that directions of baseline lengths of the respective stereo cameras intersect with each other; a depth estimation unit that estimates the depth of an object included in a captured image based on the captured image captured by the plurality of stereo cameras; and an object detection unit that detects the object based on the depth estimated by the depth estimation unit and a reliability of the depth, the reliability of the depth being determined according to an angle of a direction of an edge line of the object relative to a direction of the baseline lengths of the plurality of stereo cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a view for explaining the configuration of a general stereo camera system.

[0014] Figure 2 is a diagram illustrating an example of a case where the direction of the base line length and the extending direction of an object are different from each other in depth estimation by a stereo camera.

[0015] Figure 3 is a diagram illustrating an example of a case where the direction of the base line length is the same as the extending direction of an object in depth estimation by a stereo camera.

[0016] Figure 4 is a view illustrating an example of a case where the depth cannot be estimated appropriately.

[0017] Figure 5 is a view illustrating an example of a case where the depth cannot be estimated appropriately.

[0018] Figure 6 is a block diagram illustrating the configuration of an information processing apparatus according to the first embodiment of the present disclosure.

[0019] Figure 7 is a diagram illustrating a configuration example of a stereo camera system provided in the information processing apparatus according to the first embodiment of the present disclosure.

[0020] Figure 8 is a diagram illustrating a configuration example of a stereo camera system provided in the information processing apparatus according to the first embodiment of the present disclosure.

[0021] Figure 9 is a diagram illustrating an overview of information processing performed by the information processing apparatus according to the first embodiment of the present disclosure.

[0022] Figure 10 is a view illustrating a state in which an edge image is generated from an RGB image in information processing performed by the information processing apparatus according to the first embodiment of the present disclosure.

[0023] Figure 11 is a diagram illustrating an example of an occupancy grid map in information processing performed by the information processing apparatus according to the first embodiment of the present disclosure.

[0024] Figure 12 is a diagram illustrating an example of an edge image in information processing performed by the information processing apparatus according to the first embodiment of the present disclosure.

[0025] Figure 13 is a diagram illustrating a first stereo camera in a stereo camera system provided in the information processing apparatus according to the first embodiment of the present disclosure.

[0026] Figure 14: is a view illustrating an example of a map voting probability (probability distribution) based on an edge direction in information processing performed by the information processing apparatus according to the first embodiment of the present disclosure, the map voting probability corresponding to the first stereo camera.

[0027] Figure 15 is a diagram illustrating a second stereo camera in the stereo camera system provided in the information processing apparatus according to the first embodiment of the present disclosure.

[0028] Figure 16 : is a view illustrating an example of a map voting probability (probability distribution) based on an edge direction in information processing performed by the information processing apparatus according to the first embodiment of the present disclosure, the map voting probability corresponding to the second stereo camera.

[0029] Figure 17 is a flowchart illustrating an example of information processing performed by the information processing apparatus according to the first embodiment of the present disclosure.

[0030] Figure 18 is a block diagram illustrating the configuration of an information processing apparatus according to a second embodiment of the present disclosure.

[0031] Figure 19 is a diagram illustrating a state in which the position and orientation of a stereo camera system are changed in information processing performed by the information processing apparatus according to the second embodiment of the present disclosure.

[0032] Figure 20 is a view illustrating an example of deformation of a map voting probability (probability distribution) in information processing performed by the information processing apparatus according to the second embodiment of the present disclosure, the deformation being based on an edge direction when the position and posture of a stereo camera system are changed.

[0033] Figure 21 is a view illustrating an example of deformation of a map voting probability (probability distribution) in information processing performed by the information processing apparatus according to the second embodiment of the present disclosure, the deformation being based on an edge direction when the position and posture of a stereo camera system are changed.

[0034] Figure 22 is a flowchart illustrating an example of information processing performed by the information processing apparatus according to the second embodiment of the present disclosure.

[0035] Figure 23 is a view illustrating an example of deformation of a map voting probability (probability distribution) in a deformation example of information processing performed by an information processing device according to the second embodiment of the present disclosure, the deformation being based on edge directions when the position and posture of a stereo camera system are changed.

[0036] Figure 24is a flowchart illustrating an example of a modification of information processing performed by the information processing apparatus according to the second embodiment of the present disclosure.

[0037] Figure 25 is an explanatory diagram illustrating a hardware configuration example of an example of a computer that realizes the functions of the information processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in the following corresponding embodiments, the same reference numerals are assigned to the same parts, and repeated descriptions will be omitted.

[0039] Note that the description will be given in the following order.

[0040] 1. Overview

[0041] 2. First embodiment

[0042] <2-1. Configuration>

[0043] <2-2. Overview of Processing>

[0044] <2-3. Processing Flow>

[0045] 3. Second embodiment

[0046] 4. Modification of the Second Embodiment

[0047] 《5. Hardware Configuration Example》

[0048] 6. Effect

[0049] 1. Overview

[0050] First, refer to Figures 1 to 5 , a description will be given of an overview of information processing performed by an information processing apparatus according to an embodiment of the present disclosure. Figure 1 The figure shows the configuration of a typical stereo camera system 110, which is attached to an unmanned mobile object such as a drone. This stereo camera system 110 is a binocular camera system with two imaging units (cameras) 110a and 110b. Stereo camera system 110 is attached to the lower portion of the unmanned mobile object, for example, via a support member 120.

[0051] In object detection using such a stereo camera system 110 as described above, the distance to the object (hereinafter, this distance will be referred to as "depth") is estimated based on the parallax of the object seen from the left and right imaging units 110a and 110b, for example, by using a method such as triangulation. In such depth estimation, when the direction of the baseline length indicating the distance between the center of the imaging unit 110a and the center of the imaging unit 110b and the extending direction of the object as the measurement target are not parallel to each other but intersect with each other, for example, Figure 2 As shown in , the depth can be estimated appropriately because it is easy to grasp the correlation between such an object reflected in the video of the right camera and such an object reflected in the video of the left camera.

[0052] Meanwhile, when the direction of the baseline length and the extending direction of the object as the measurement target are parallel or nearly parallel to each other, for example Figure 3 As shown in , the depth cannot be properly estimated because it is difficult to grasp the correlation between the object reflected in the video of the right camera and the object reflected in the video of the left camera. Therefore, for example, in the case of operating the stereo camera system 110 while attaching it to an unmanned mobile object so that the direction of its baseline length and the horizontal direction coincide with each other (see Figure 1 ), an unrecognized Figure 4 The wires shown in Section A, or identify Figure 5 Part B shows the top of the building protruding forward due to malfunction.

[0053] Therefore, the information processing device according to the present disclosure uses a trinocular camera system having three lenses arranged in a V shape, for example, to estimate depth using two stereo cameras, then performs edge processing on the captured images to detect the edges of objects. The information processing device according to the present disclosure then calculates a map voting probability for the occupancy grid map based on the reliability of the depth, which corresponds to the direction of the edge (hereinafter, this direction will be referred to as the "edge direction"), reflects the calculated map voting probability on the occupancy grid map, and thereby detects objects.

[0054] 2. First embodiment

[0055] <2-1. Configuration>

[0056] Next, refer to Figures 6 to 8, a description will be given of the configuration of the information processing device 1 according to the first embodiment. The information processing device 1 is a device that is installed on an unmanned mobile body such as a drone, and detects an object from an image captured by a trinocular camera system. Note that in this embodiment, a description will be given of an example in which the information processing device 1 is installed on an unmanned mobile body. However, in addition to this, the information processing device 1 may be installed on an autonomous mobile robot, a vehicle, a portable terminal, etc. used in a factory, a home, etc. As Figure 6 As shown in , the information processing apparatus 1 includes three imaging units (cameras) 10 a , 10 b , and 10 c , a control unit 20 , and a storage unit 30 .

[0057] Each of the imaging units 10a, 10b, and 10c has an optical system composed of a lens, an aperture, and the like, and includes an image sensor, and performs processing for capturing a subject. Each of the imaging units 10a, 10b, and 10c outputs an image captured thereby (hereinafter, this image will be referred to as a "captured image") to the control unit 20. Furthermore, the imaging units 10a, 10b, and 10c constitute a stereo camera system 10 including a first stereo camera 11a and a second stereo camera 11b.

[0058] For example, Figure 7 As shown in FIG, the stereo camera system 10 is a trinocular camera system comprising three imaging units 10a, 10b, and 10c. The stereo camera system 10 is attached to, for example, the lower portion of an unmanned mobile object, with a support member 12 interposed therebetween. The imaging units 10a, 10b, and 10c are arranged in a V-shape. Specifically, the first stereo camera 11a and the second stereo camera 11b are arranged such that the direction of the baseline length of the first stereo camera 11a and the direction of the baseline length of the second stereo camera 11b are perpendicular to each other.

[0059] The imaging units 10a and 10b constituting the first stereo camera 11a are arranged side by side in the horizontal direction. That is, the direction of the baseline length connecting the imaging units 10a and 10b coincides with the horizontal direction. Moreover, the imaging units 10b and 10c constituting the second stereo camera 11b are arranged side by side in the vertical direction. That is, the direction of the baseline length connecting the imaging units 10b and 10c coincides with the vertical direction. It is noted that the first stereo camera 11a and the second stereo camera 11b only need to be arranged so that the directions of the corresponding baseline lengths intersect with each other, and the directions of the corresponding baseline lengths may form an angle other than a right angle.

[0060] Herein, the direction of the base length of the first stereo camera 11a and the second stereo camera 11b does not necessarily coincide with the horizontal direction or the vertical direction. For example, in the information processing device 1, the base length of the first stereo camera 11a and the second stereo camera 11b may be aligned with the horizontal direction or the vertical direction. Figure 8 The stereo camera system 10A shown in FIG. Figure 7 . In the stereo camera system 10A, the imaging unit 10a and the imaging unit 10b constituting the first stereo camera 11a are arranged so as to be offset from each other in the horizontal or vertical direction. Similarly, the imaging unit 10b and the imaging unit 10c constituting the second stereo camera 11b are arranged so as to be offset from each other in the horizontal or vertical direction.

[0061] The control unit 20 can be implemented by various processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc.). The control unit 20 performs various processes on the captured images input from the imaging units 10a, 10b, and 10c. The control unit 20 includes a depth estimation unit 21, an edge detection unit 22, and an object detection unit 23.

[0062] Based on the captured images captured by the first stereo camera 11a and the second stereo camera 11b, the depth estimation unit 21 estimates the depth of the object included in the captured images. Based on the parallax of the object seen from the imaging units 10a and 10b and the parallax of the object seen from the imaging units 10b and 10c, the depth estimation unit 21 estimates the depth by using, for example, a known method such as triangulation.

[0063] Based on the captured images captured by the first stereo camera 11a and the second stereo camera 11b, the edge detection unit 22 detects the edge of an object included in the captured images. Note that "edge" refers to a continuous line indicating the boundary of an object. The edge detection unit 22 detects the edge of the object from the monocular image (RGB image) captured by any one of the imaging units 10a, 10b, and 10c, and generates an edge image (see the image processing unit described later). Figure 10 ).

[0064] The object detection unit 23 detects an object based on the depth information generated by the depth estimation unit 21 and the edge information generated by the edge detection unit 22. The object detection unit 23 detects an object based on the depth estimated by the depth estimation unit 21 and the reliability of the depth, which is determined by the angle of the object in the edge direction relative to the direction of the baseline length of the first stereo camera 11a and the second stereo camera 11b. Note that the details of the processing in the object detection unit 23 will be described later.

[0065] The storage unit 30 stores various information. The storage unit 30 stores, for example, programs for implementing the respective units of the control unit 20. In this case, the control unit 20 expands and executes the programs stored in the storage unit 30, thereby implementing the functions of the respective units. The storage unit 30 can be implemented, for example, by semiconductor memory elements such as random access memory (RAM), read-only memory (ROM), and flash memory, or by storage devices such as hard disks, solid-state drives, and optical disks. Furthermore, the storage unit 30 can be composed of a plurality of different memories, etc.

[0066] <2-2. Overview of Processing>

[0067] Next, refer to Figures 9 to 16 , a description will be given of an overview of the processing of the information processing apparatus 1 according to the present embodiment. Figure 9 As shown in , the information processing apparatus 1 implements an image capturing process Pr1, a depth estimating process Pr2, an edge detecting process Pr3, a map voting probability calculating process Pr4, and a map voting process Pr5 in this order.

[0068] (Image Capture Processing Prl)

[0069] In the image capturing process Pr1 , the imaging units 10 a , 10 b , and 10 c capture, for example, RGB images and output the captured RGB images to the depth estimating unit 21 and the edge detecting unit 22 .

[0070] (Depth estimation processing Pr2)

[0071] In the depth estimation process Pr2 , the depth estimation unit 21 estimates the depth of an object included in the RGB image and outputs the estimated depth information to the object detection unit 23 .

[0072] (Edge Detection Processing Pr3)

[0073] In the edge detection process Pr3, the edge detection unit 22 detects the edge of the object included in the RGB image. In the edge detection process Pr3, for example, Figure 10 As shown in , the edge detection unit 22 converts such an RGB image into an edge image and outputs edge information including information on the direction and position of the edge to the object detection unit 23.

[0074] (Map Voting Probability Calculation Process Pr4)

[0075] In the map voting probability calculation process Pr4, the object detection unit 23 calculates the map voting probability of the occupied grid map. Figure 11As shown in , the “occupancy grid map” refers to a map in which the space included in the RGB image is divided into a grid shape and viewed from above. The unmanned mobile object on which the information processing device 1 is mounted grasps whether there are obstacles on the moving path based on this occupancy grid map. Figure 11 In the figure, a densely hatched grid indicates "object exists", a blank grid indicates "object does not exist", and a lightly hatched grid indicates "unknown object exists".

[0076] The map voting probability is the probability that an object occupies each grid in the occupancy grid map. For example, Figure 11 The posterior probability of an event in a certain grid (Cell i) in the occupancy grid map shown in (the probability that an object occupies a certain grid) can be expressed by the following expression (1).

[0077] p(m i |z 1：t , x 1：t ) (1)

[0078] in,

[0079] z 1：t : Observation data

[0080] x 1：t : own position

[0081] Furthermore, the above expression (1) can be calculated using the following expression (2). Specifically, the "posterior probability of the event until time t" can be obtained by multiplying the "posterior probability of the event until one time point" by the "map voting probability based on the observation at the current time point." Specifically, the current occupancy grid map is obtained by multiplying this probability held by the occupancy grid map until one time point by the probability based on the current observation.

[0082] p(m i |z 1：t , x 1：t )=p(m i |z 1：t-1 , x 1：t-1 )xp(m i |z t , x t ) (2)

[0083] Among them, p(m i |z 1：t , x 1：t ): posterior probability of the event up to time t

[0084] p(m i |z 1：t-1 , x 1：t-1): the posterior probability of the event up to 1 time ago

[0085] p(m i |z t , x t ): The probability of voting for the map based on the observation at the current time point

[0086] In this embodiment, “p(m i |z t , x t )" is obtained by the following expression (3). That is, the "map voting probability based on the distance to the observation event (first probability distribution)" is multiplied by the "map voting probability based on the edge direction of the observation event (second probability distribution)" to obtain the "map voting probability based on the observation at the current time point". That is, the map voting probability is obtained three-dimensionally.

[0087] p(m i |z t , x t )=p(m i |L i，t )xp(m i |E i，t ) (3)

[0088] Among them, p(m i |L i，t ): Map voting probability based on distance to the observed event

[0089] p(m i |E i，t ): Map voting probability based on the edge direction of the observed event

[0090] In the map voting probability calculation process Pr4, the object detection unit 23 calculates a first probability distribution based on the depth estimated by the depth estimation unit 21 and a second probability distribution indicating the reliability of the depth. Herein, the first probability distribution refers to the "map voting probability based on the distance of the observation event" in the above expression (3), and the second probability distribution refers to the "map voting probability based on the edge direction of the observation event" in the above expression (3).

[0091] The “reliability of depth” refers to the probability distribution at the position where the edge of the object exists in the captured image, and the probability distribution is higher as the edge direction is closer to a right angle with respect to the direction of the baseline length.

[0092] For example, consider a case where an edge in the horizontal direction and an edge in the vertical direction are detected by the edge detection unit 22, such as Figure 12 In this case, as in Figure 13The second probability distribution corresponding to the first stereo camera 11a whose baseline length is in the horizontal direction becomes as follows Figure 14 The highest probability shown in the distribution overlaps with the edge in the vertical direction perpendicular to the baseline length. Figure 15 The second probability distribution corresponding to the second stereo camera 11b whose baseline length is in the vertical direction becomes as follows Figure 16 The highest probability shown in the distribution overlaps with the edge in the horizontal direction perpendicular to the baseline length.

[0093] As described above, in the map voting probability calculation process Pr4, such reliability indicating how reliable the depth is is approximated as, for example, Figure 14 and 16 Such a two-dimensional normal distribution is shown in , the depth is estimated from the captured images of the first stereo camera 11a and the second stereo camera 11b.

[0094] In the map voting probability calculation process Pr4, the object detection unit 23 calculates the map voting probability in the occupancy grid map based on the first probability distribution and the second probability distribution. That is, as shown in the above expression (3), the object detection unit 23 multiplies the first probability distribution (the map voting probability based on the distance to the object) and the second probability distribution (the map voting probability based on the edge direction of the object) by each other to calculate the map voting probability.

[0095] (Map voting process Pr5)

[0096] In the map voting process Pr5, the object detection unit 23 votes for each grid based on the calculated map voting probability, thereby creating an occupied grid map. The information processing device 1 detects an object based on the occupied grid map thus created.

[0097] <2-3. Processing Flow>

[0098] Next, refer to Figure 17 The following describes the processing flow of the information processing device 1 according to this embodiment. First, the control unit 20 acquires images captured by the first stereo camera 11a and the second stereo camera 11b (step S1). Subsequently, the depth estimation unit 21 of the control unit 20 estimates the depth of an object included in the captured image (step S2).

[0099] Subsequently, the control unit 20 determines whether the depth estimation unit 21 has been able to properly estimate the depth (step S3). In the case of determining that the depth cannot be properly estimated ("No" in step S3), the control unit 20 ends this process. At the same time, in the case of determining that the depth can be properly estimated ("Yes" in step S3), the edge detection unit 22 of the control unit 20 detects the edge of the object included in the captured image (step S4).

[0100] Subsequently, the control unit 20 determines whether the edge detection unit 22 can properly detect the edge (step S5). In the case of determining that the edge cannot be properly detected ("No" in step S5), the object detection unit 23 of the control unit 20 votes on the occupancy grid map (step S6) and ends this process. At the same time, in the case of determining that the edge has been properly detected ("Yes" in step S5), the object detection unit 23 calculates the map voting probability based on the above expressions (1) to (3) (step S7).

[0101] Subsequently, the object detection unit 23 determines whether there are multiple calculation results of the map voting probability (step S8). Note that "there are multiple calculation results of the map voting probability" means that, for example, Figure 13 The direction of the baseline length shown in the horizontal direction is the first stereo camera 11a corresponding to the map voting probability and Figure 15 The direction of the baseline length shown in FIG is the vertical direction of the second stereo camera 11 b corresponding to the map voting probability.

[0102] In the case where it is determined in step S8 that there are no calculation results of a plurality of map voting probabilities (No in step S8 ), the object detection unit 23 votes for the occupied grid map based on the calculated map voting probabilities (step S9 ), and ends this process.

[0103] If it is determined in step S8 that there are multiple map voting probabilities calculated ("Yes" in step S8), the object detection unit 23 multiplies the multiple map voting probabilities and adds them together (step S10). Subsequently, the object detection unit 23 votes for the occupied grid map based on the added map voting probabilities (step S11), and ends this process.

[0104] 3. Second embodiment

[0105] Next, refer to Figures 18 to 22 Next, a description will be given of an information processing device 1A according to a second embodiment. In addition to the corresponding components of the information processing device 1 described above, the information processing device 1A includes an inertial measurement unit 40. Furthermore, the control unit 20A of the information processing device 1A includes a position / attitude estimation unit 24 in addition to the corresponding components of the control unit 20 described above.

[0106] The inertial measurement unit 40 is composed of an inertial measurement unit (IMU) including, for example, a three-axis acceleration sensor, a three-axis gyro sensor, and the like, and outputs the acquired sensor information to the position / attitude estimation unit 24 of the control unit 20A. The position / attitude estimation unit 24 detects the position and attitude (e.g., orientation, inclination, etc.) of the unmanned mobile object on which the information processing device 1A is mounted based on the captured images captured by the imaging units 10a, 10b, and 10c and the sensor information input from the inertial measurement unit 40. Note that the method for detecting the position and attitude of the unmanned mobile object is not limited to the method using the above-mentioned IMU.

[0107] The object detection unit 23 in this embodiment takes the second probability distribution calculated in the previous frame (i.e., the map voting probability based on the edge direction of the object) as a key frame, and when the position and posture of the first stereo camera 11a and the second stereo camera 11b change in the current frame, the key frame is deformed (e.g., moved and rotated) based on the changes in the position and posture of the first stereo camera 11a and the second stereo camera 11b, thereby calculating the second probability distribution.

[0108] If the posture of the stereo camera system 10 is Figure 19 , the second probability distribution is recalculated when the second probability distribution is changed (for example, when the unmanned mobile object is flying), then the processing load of the information processing device 1A will increase. In this regard, the object detection unit 23 of the information processing device 1A pre-registers the second probability distribution calculated in the previous frame as a key frame. Then, when the posture of the stereo camera system 10 changes in the next frame (current frame), the object detection unit 23 rotates the corresponding key frame (second probability distribution) according to the change in the posture of the machine, as shown in FIG. Figure 20 As shown in , the posture of the present machine is estimated by the position / posture estimation unit 24, thereby calculating the second probability distribution of the current frame. Subsequently, the object detection unit 23 multiplies the calculated second probability distribution and the first probability distribution by each other, thereby calculating the map voting probability. Note that when the change in the position and posture of the present machine estimated by the position / posture estimation unit 24 is equal to or greater than a predetermined threshold, the object detection unit 23 recalculates the second probability distribution and re-registers the calculated second probability distribution as a key frame.

[0109] refer to Figure 21 , a description will be given below of an example of a deformation of the second probability distribution in which a change in the position and posture of the present machine is taken into account. For example, when the second probability distribution is approximated by a two-dimensional normal distribution of the periphery of the edge, as Figure 21As shown in , this normal distribution can be represented by values such as the x-mean, y-mean, horizontal edge dispersion, vertical edge dispersion, tilt, and the size of the overall distribution. These values change depending on the angle of the edge and the position and posture of the device. Note that, based on the distance to the edge and the parameters of imaging units 10a, 10b, and 10c, it is possible to estimate how the edge moves in the image.

[0110] For example, when the direction of the base line length of the first stereo camera 11a is the horizontal direction (see Figure 13 ),like Figure 21 As shown in , both the horizontal and vertical dispersions increase as the angle of the edge relative to the direction of the baseline length decreases. Moreover, as the angle of the edge relative to the direction of the baseline length decreases, the size of the entire distribution decreases.

[0111] The information processing device 1A performs such processing as described above, whereby the calculation frequency of the second probability distribution (map voting probability based on the edge direction of the object) can be reduced, and thus the processing load can be reduced.

[0112] Next, refer to Figure 22 Next, a description will be given of the processing flow of the information processing apparatus 1A according to this embodiment. First, the position / attitude estimation unit 24 of the control unit 20A estimates the position and attitude of the device (step S11). Subsequently, the control unit 20A determines whether the change in the position and attitude of the device estimated by the position / attitude estimation unit 24 is less than a predetermined threshold (step S12).

[0113] If the change in the position and posture of the machine estimated by the position / posture estimation unit 24 of the control unit 20A in step S12 is less than a predetermined threshold ("Yes" in step S12), the object detection unit 23 calculates the map voting probability caused by the change in the position and posture of the machine (step S13). In step S13, the object detection unit 23 deforms the pre-registered key frame according to the change in the position and posture of the machine, thereby calculating a second probability distribution. The object detection unit 23 then multiplies the calculated second probability distribution and the first probability distribution by each other to calculate the map voting probability. Subsequently, the object detection unit 23 votes for the occupancy grid map (step S14), and ends this process.

[0114] If the change in the position and posture of the device estimated by the position / posture estimation unit 24 of the control unit 20A in step S12 is equal to or greater than a predetermined threshold ("No" in step S12), the object detection unit 23 calculates the map voting probability due to the edge direction (step S15). In step S15, the object detection unit 23 again calculates the second probability distribution and multiplies the calculated second probability distribution by the first probability distribution to calculate the map voting probability. Subsequently, the object detection unit 23 votes for the occupancy grid map (step S16), re-registers the second probability distribution as a key frame (step S17), and ends this process.

[0115] 4. Modification of the Second Embodiment

[0116] Next, refer to Figure 23 and Figure 24 , a description will be given of a modification of the information processing apparatus 1A according to the second embodiment. The configuration of the information processing apparatus 1A according to this embodiment is the same as that of the second embodiment. Figure 18 The configuration in is similar, and its description will be omitted accordingly.

[0117] The object detection unit 23 in this embodiment compares the second probability distribution calculated by deforming the key frame with the second probability distribution calculated in the current frame, thereby filtering the second probability distribution. Figure 23 As shown in , the object detection unit 23 compares the second probability distribution calculated due to the change in the position and posture of the machine with the second probability distribution calculated due to the current edge direction, eliminating probability distributions that are extremely different from others and probability distributions that have a sense of incongruity, thereby improving the stability of object detection.

[0118] Next, refer to Figure 24 Next, a description will be given of the processing flow of the information processing apparatus 1A according to this embodiment. First, the position / attitude estimation unit 24 of the control unit 20A estimates the position and attitude of the device (step S21). Subsequently, the control unit 20A determines whether the change in the position and attitude of the device estimated by the position / attitude estimation unit 24 is less than a predetermined threshold (step S22).

[0119] If the change in the position and posture of the device estimated by the position / posture estimation unit 24 of the control unit 20A in step S22 is less than a predetermined threshold ("Yes" in step S22), the object detection unit 23 calculates the map voting probability caused by the change in the position and posture of the device (step S23). In step S23, the object detection unit 23 deforms the pre-registered keyframes according to the change in the position and posture of the device to calculate a second probability distribution. The object detection unit 23 then multiplies the calculated second probability distribution by the first probability distribution to calculate the map voting probability.

[0120] The object detection unit 23 then compares the second probability distribution calculated based on the change in the position and posture of the object with the second probability distribution calculated based on the current edge direction (step S24), and filters out probability distributions that are significantly different from the others (step S25). The object detection unit 23 then votes on the occupancy grid map (step S26), and ends this process.

[0121] If the change in the position and posture of the device estimated by the position / posture estimation unit 24 of the control unit 20A in step S22 is equal to or greater than a predetermined threshold ("No" in step S22), the object detection unit 23 calculates the map voting probability due to the edge direction (step S27). In step S27, the object detection unit 23 again calculates the second probability distribution and multiplies the calculated second probability distribution by the first probability distribution to calculate the map voting probability. Subsequently, the object detection unit 23 votes for the occupancy grid map (step S28), re-registers the second probability distribution as a key frame (step S29), and ends this process.

[0122] 《5. Hardware Configuration Example》

[0123] For example, by having Figure 25 The computer 1000 of the configuration shown in the figure realizes information equipment such as the information processing apparatus 1 and 1A according to the above-described embodiments. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The respective units of the computer 1000 are connected to each other via a bus 1050.

[0124] The CPU 1100 operates and controls corresponding units based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 expands the programs stored in the ROM 1300 or the HDD 1400 in the RAM 1200 and executes processing each corresponding to each of the various programs.

[0125] The ROM 1300 stores startup programs such as a basic input output system (BIOS) to be executed by the CPU 1100 when the computer 1000 is started, programs that depend on the hardware of the computer 1000 , and the like.

[0126] HDD 1400 is a computer-readable recording medium that non-temporarily records a program to be executed by CPU 1100, data to be used by the program, etc. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0127] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, via the communication interface 1500, the CPU 1100 receives data from another device and transmits data generated by the CPU 1100 to another device.

[0128] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000 to each other. For example, via the input / output interface 1600, the CPU 1100 receives data from input devices such as a keyboard and a mouse. Moreover, via the input / output interface 1600, the CPU 1100 transmits data to output devices such as a display, a speaker, and a printer. In addition, the input / output interface 1600 can be used as a medium interface for reading programs recorded in a predetermined recording medium. For example, the medium is an optical recording medium such as a digital versatile disk (DVD) and a phase-change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0129] For example, when the computer 1000 is used as the information processing apparatus 1 or 1A according to the embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded on the RAM 1200, thereby realizing the functions of the control unit 130, etc. Furthermore, the HDD 1400 stores the information processing program and data according to the present disclosure in the storage unit 30. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes it; however, as another example, the CPU 1100 may obtain these programs from another device via the external network 1550.

[0130] 6. Effect

[0131] Information processing devices 1 and 1A include a first stereo camera 11a and a second stereo camera 11b, a depth estimation unit 21, and an object detection unit 23. The first stereo camera 11a and the second stereo camera 11b are arranged so that the directions of their respective baseline lengths intersect. Furthermore, based on captured images captured by the first stereo camera 11a and the second stereo camera 11b, the depth estimation unit 21 estimates the depth of an object included in the captured image. Furthermore, the object detection unit 23 detects an object based on the depth estimated by the depth estimation unit 21 and the reliability of the depth, which is determined based on the angle of the object in the edge direction relative to the direction of the baseline length of the first stereo camera 11a and the second stereo camera 11b.

[0132] Therefore, by using the depth reliability, which is determined based on the angle of the object in the edge direction relative to the direction of the baseline length of the first stereo camera 11a and the second stereo camera 11b, the information processing devices 1 and 1A can detect objects with high accuracy, regardless of the edge direction. Moreover, the information processing devices 1 and 1A are applied to unmanned mobile objects such as drones, for example, so that horizontal lines such as power lines can be appropriately identified and the tops of buildings can be prevented from being identified as protruding forward. Moreover, according to the information processing devices 1 and 1A, existing depth estimation systems that use stereo cameras can be used, and accordingly, the accuracy of object detection can be improved at a low cost.

[0133] In the information processing devices 1 and 1A, the reliability of the depth is a probability distribution at a position where an edge of an object exists in a captured image, and a probability distribution in which the edge direction is more likely to be closer to a right angle with respect to the direction of the baseline length.

[0134] Therefore, the information processing devices 1 and 1A model the reliability of the depth to be determined based on the angle of the object in the edge direction relative to the direction of the baseline length of the first stereo camera 11a and the second stereo camera 11b into a probability distribution, thereby making it possible to detect the object with high accuracy.

[0135] In the information processing devices 1 and 1A, the object detection unit 23 calculates a first probability distribution (i.e., a map voting probability based on the distance to the object) based on the depth estimated by the depth estimation unit 21 and a second probability distribution indicating the reliability of the depth (i.e., a map voting probability based on the edge direction of the object). Based on the first and second probability distributions, the object detection unit 23 calculates a map voting probability indicating the probability that the object occupies each grid in the occupancy grid map, where the space included in the captured image is divided into a grid shape, and votes for each grid based on the map voting probability, thereby creating the occupancy grid map.

[0136] Therefore, the information processing devices 1 and 1A create an occupancy grid map, whereby the position of an object and the distance to the object can be grasped.

[0137] In the information processing devices 1 and 1A, the object detection unit 23 multiplies the first probability distribution and the second probability distribution by each other, thereby calculating the map voting probability.

[0138] Therefore, the information processing devices 1 and 1A can three-dimensionally obtain the map voting probability in consideration of the probability distribution based on the distance to the object and the probability distribution based on the edge direction of the object.

[0139] In the information processing device 1A, the object detection unit 23 registers the second probability distribution (i.e., the map voting probability based on the edge direction of the object) calculated in the previous frame as a key frame, and when the position and posture of the first stereo camera 11a and the second stereo camera 11b change in the current frame, the key frame is deformed based on the change in the position and posture of the first stereo camera 11a and the second stereo camera 11b, thereby calculating the second probability distribution.

[0140] Therefore, the information processing apparatus 1A can reduce the amount of calculation by using information on the movement and rotation of the own machine (the first stereo camera 11 a and the second stereo camera 11 b ), and accordingly reduce the processing load.

[0141] In the information processing device 1A, the object detection unit 23 compares the second probability distribution (map voting probability based on the edge direction of the object) calculated by deforming the key frame with the second probability distribution calculated in the current frame, thereby filtering the second probability distribution.

[0142] Therefore, the information processing device 1A filters out probability distributions that are extremely different from others and probability distributions that feel inconsistent, thereby enhancing the stability of object detection.

[0143] In the information processing devices 1 and 1A, the first stereo camera 11a and the second stereo camera 11b include the first stereo camera 11a and the second stereo camera 11b, and the first stereo camera 11a and the second stereo camera 11b are arranged so that the direction of the baseline length of the first stereo camera 11a and the direction of the baseline length of the second stereo camera 11b are perpendicular to each other.

[0144] Therefore, due to the fact that the first stereo camera 11a and the second stereo camera 11b are arranged so that the direction of the baseline length of the first stereo camera 11a and the direction of the baseline length of the second stereo camera 11b are perpendicular to each other, the information processing devices 1 and 1A can detect objects with high accuracy regardless of the edge direction.

[0145] An information processing method includes: estimating the depth of an object included in a captured image based on a captured image captured by a first stereo camera 11a and a second stereo camera 11b arranged so that directions of their respective baseline lengths intersect with each other; and detecting the object based on the depth and the reliability of the depth, the reliability of the depth being determined based on the angle of the object in an edge direction relative to the direction of the baseline lengths of the first stereo camera 11a and the second stereo camera 11b.

[0146] Therefore, by using the reliability of the depth determined based on the edge direction of an object, this information processing method can detect objects with high accuracy, regardless of the edge direction. Furthermore, this information processing method can be applied to unmanned mobile objects such as drones, enabling appropriate identification of horizontal lines such as power lines and preventing building tops from being recognized as protruding forward. Furthermore, this information processing method can be used with existing depth estimation systems that use stereo cameras, thereby improving object detection accuracy at a low cost.

[0147] The information processing program causes the computer to function as a depth estimation unit 21, an edge detection unit 22, and an object detection unit 23. The depth estimation unit 21 estimates the depth of an object included in a captured image captured by the first stereo camera 11a and the second stereo camera 11b, which are arranged so that the directions of their respective baseline lengths intersect. Furthermore, the object detection unit 23 detects the object based on the depth estimated by the depth estimation unit 21 and the reliability of the depth, which is determined based on the angle of the object in the edge direction relative to the direction of the baseline lengths of the first stereo camera 11a and the second stereo camera 11b.

[0148] Therefore, by using the reliability of the depth determined based on the edge direction of an object, the information processing program can detect objects with high accuracy regardless of the edge direction. Furthermore, this information processing program can be applied to unmanned mobile objects such as drones, enabling appropriate identification of horizontal lines such as power lines and preventing building tops from being recognized as protruding forward. Furthermore, this information processing program can be used with existing depth estimation systems that use stereo cameras, thereby improving object detection accuracy at a low cost.

[0149] Note that the effects described in this specification are merely examples and not limitations, and other effects may be exhibited.

[0150] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings; however, the technical scope of the present disclosure is not limited to such examples. Obviously, a person skilled in the art of the present disclosure can conceive of various modifications or changes within the scope of the technical concept described in the claims, and it should be understood that these also naturally fall within the technical scope of the present disclosure.

[0151] For example, the stereo camera system 10 of each of the above-mentioned information processing apparatuses 1 and 1A is a stereo camera system having three imaging units 10a, 10b, and 10c (see Figure 7 ) of the three-eye camera system; however, the number of imaging units may be four or more. Moreover, in each of the above-mentioned information processing devices 1 and 1A, the probability distribution indicating the reliability of the depth is represented by a two-dimensional normal distribution (see Figure 14 and Figure 16 ), but can be represented by distributions other than the two-dimensional normal distribution.

[0152] Furthermore, in the aforementioned information processing devices 1 and 1A, the edge of an object is detected by the edge detection unit 22, and the depth reliability is determined based on the angle of the object in the direction of the edge relative to the direction of the baseline length of the first stereo camera 11a and the second stereo camera 11b. However, the edge line of an object other than the edge may be detected, and the depth reliability may be determined based on the angle of the object in the direction of the edge line relative to the aforementioned direction of the baseline length. For example, the edge line of an object other than the edge may include a continuous line indicating a boundary between colors, patterns, etc. in the object.

[0153] In addition, in the above-mentioned information processing device 1, when there are a plurality of calculation results of the map voting probability, the plurality of map voting probabilities are multiplied by each other and added (see step S10 Figure 17 ); however, the occupancy grid maps may be voted based on the map voting probabilities with higher probabilities without adding the map voting probabilities to each other.

[0154] Furthermore, the information processing devices 1 and 1A described above can acquire reliable depth information regardless of the edge direction of the object used as the subject, and accordingly can be used for a wide range of purposes other than creating occupancy grid maps. For example, the information processing devices 1 and 1A use acquired data as key frames to check whether the data acquired thereafter (data acquired by a single stereo may be acceptable) has unnatural portions, or can be used to determine short-term collisions.

[0155] Note that the present technology can also adopt configurations as follows. (1)

[0157] An information processing device, comprising:

[0158] a plurality of stereo cameras arranged such that directions of baseline lengths of the respective stereo cameras intersect with each other;

[0159] a depth estimating unit that estimates a depth of an object included in the captured image based on the captured images captured by the plurality of stereo cameras; and

[0160] The object detection unit detects the object based on the depth estimated by the depth estimation unit and the reliability of the depth, the reliability of the depth being determined according to an angle of a direction of an edge line of the object relative to a direction of a baseline length of the plurality of stereo cameras. (2)

[0162] An information processing device according to (1), wherein the reliability of the depth is a probability distribution at a position where an edge line of an object in a captured image exists, and is a probability distribution in which the closer the direction of the edge line is to a right angle relative to the direction of the baseline length, the higher the probability. (3)

[0164] The information processing device according to (1) or (2), wherein

[0165] Object detection unit

[0166] calculating a first probability distribution based on the depth estimated by the depth estimation unit and a second probability distribution indicating reliability of the depth,

[0167] calculating a map voting probability indicating a probability that the object occupies each grid in an occupancy grid map, in which a space included in a captured image is divided into a grid shape, based on the first probability distribution and the second probability distribution, and

[0168] Each grid is voted based on the map voting probability to create an occupancy grid map. (4)

[0170] The information processing device according to (3), wherein the object detection unit multiplies the first probability distribution and the second probability distribution by each other to calculate the map voting probability. (5)

[0172] The information processing device according to any one of (1) to (4), wherein

[0173] Object detection unit

[0174] registering the second probability distribution calculated in the previous frame as a keyframe, and

[0175] When the positions and postures of the plurality of stereo cameras change in the current frame, the key frame is deformed based on the amount of change in the positions and postures of the plurality of stereo cameras to calculate a second probability distribution. (6)

[0177] The information processing device according to any one of (1) to (5), wherein

[0178] The object detection unit compares the second probability distribution calculated by deforming the key frame and the second probability distribution calculated in the current frame with each other to filter the second probability distribution. (7)

[0180] The information processing device according to any one of (1) to (6), wherein

[0181] The plurality of stereo cameras includes a first stereo camera and a second stereo camera, and

[0182] The first stereo camera and the second stereo camera are arranged such that a direction of a base line length of the first stereo camera and a direction of a base line length of the second stereo camera are perpendicular to each other. (8)

[0184] An information processing method, comprising:

[0185] estimating a depth of an object included in a captured image from captured images captured by a plurality of stereo cameras arranged such that directions of base line lengths of the respective stereo cameras intersect with each other; and

[0186] The object is detected based on the depth and the reliability of the depth, which is determined according to the angle of the direction of the edge line of the object relative to the direction of the baseline length of the plurality of stereo cameras. (9)

[0188] An information processing program that enables a computer to:

[0189] a depth estimating unit that estimates a depth of an object included in a captured image based on captured images captured by a plurality of stereo cameras arranged such that directions of base line lengths of the respective stereo cameras intersect with each other; and

[0190] The object detection unit detects the object based on the depth estimated by the depth estimation unit and the reliability of the depth, the reliability of the depth being determined according to an angle of a direction of an edge line of the object relative to a direction of a baseline length of the plurality of stereo cameras.

[0191] Reference Signs List

[0192] 1, 1A Information processing device

[0193] 10, 10A, 110 Stereo Camera System

[0194] 10a, 10b, 10c, 110a, 110b imaging unit

[0195] 11a The first stereo camera

[0196] 11b Second stereo camera

[0197] 12,120 support members

[0198] 20, 20A control unit

[0199] 21 Depth Estimation Unit

[0200] 22 edge detection unit

[0201] 23 Object Detection Unit

[0202] 24 Position / attitude estimation units

[0203] 30 storage units

[0204] 40 Inertial Measurement Unit

Claims

1. An information processing device, comprising: a plurality of stereo cameras arranged such that directions of baseline lengths of the respective stereo cameras intersect with each other; a depth estimation unit that estimates a depth of an object included in a captured image based on the captured image captured by the plurality of stereo cameras; as well as an object detection unit that detects the object based on the depth estimated by the depth estimation unit and a reliability of the depth, the reliability of the depth being determined according to an angle of a direction of an edge line of the object relative to a direction of a baseline length of the plurality of stereo cameras, wherein the reliability of the depth is a probability distribution of positions where edge lines of the object exist in the captured image, the probability distribution being higher the closer the direction of the edge line is to a right angle with respect to the direction of the baseline length, The object detection unit further: calculating a first probability distribution based on the depth estimated by the depth estimation unit and a second probability distribution indicating reliability of the depth, calculating a map voting probability indicating a probability that the object occupies each grid in an occupancy grid map, in which a space included in the captured image is divided into a grid shape, based on a first probability distribution and a second probability distribution; and Voting is performed on each grid based on the map voting probability to create the occupancy grid map. 2 . The information processing device according to claim 1 , wherein the object detection unit multiplies a first probability distribution and a second probability distribution by each other to calculate the map voting probability.

3. The information processing apparatus according to claim 2, wherein The object detection unit registering the second probability distribution calculated in the previous frame as a keyframe, and When the positions and postures of the plurality of stereo cameras change in the current frame, the key frame is deformed based on the amount of change in the positions and postures of the plurality of stereo cameras to calculate a second probability distribution.

4. The information processing apparatus according to claim 3, wherein The object detection unit compares a second probability distribution calculated by deforming the key frame and a second probability distribution calculated in the current frame with each other to filter the second probability distribution. The information processing apparatus according to claim 1 , wherein The plurality of stereo cameras include a first stereo camera and a second stereo camera, and The first stereo camera and the second stereo camera are arranged such that a direction of a base line length of the first stereo camera and a direction of a base line length of the second stereo camera are perpendicular to each other.

6. An information processing method, comprising: estimating a depth of an object included in a captured image based on captured images captured by a plurality of stereo cameras, the plurality of stereo cameras being arranged such that directions of baseline lengths of the respective stereo cameras intersect with each other; as well as The object is detected based on the depth and the reliability of the depth, the reliability of the depth being determined according to an angle of a direction of an edge line of the object relative to a direction of a baseline length of the plurality of stereo cameras, wherein the reliability of the depth is a probability distribution of positions where the edge line of the object exists in the captured image, wherein the closer the direction of the edge line is to a right angle relative to the direction of the baseline length, the higher the probability of the position being the same. Wherein detecting the object further comprises: calculating a first probability distribution based on the estimated depth and a second probability distribution indicating the reliability of the depth, calculating a map voting probability indicating a probability that the object occupies each grid in an occupancy grid map, in which a space included in the captured image is divided into a grid shape, based on a first probability distribution and a second probability distribution; and Voting is performed on each grid based on the map voting probability to create the occupancy grid map.

7. An information processing program for causing a computer to: a depth estimating unit that estimates a depth of an object included in a captured image based on captured images captured by a plurality of stereo cameras arranged such that directions of base line lengths of the respective stereo cameras intersect with each other; and an object detection unit that detects the object based on the depth estimated by the depth estimation unit and a reliability of the depth, the reliability of the depth being determined according to an angle of a direction of an edge line of the object relative to a direction of a baseline length of the plurality of stereo cameras, wherein the reliability of the depth is a probability distribution of positions where edge lines of the object exist in the captured image, the probability distribution being higher the closer the direction of the edge line is to a right angle with respect to the direction of the baseline length, The object detection unit further: calculating a first probability distribution based on the depth estimated by the depth estimation unit and a second probability distribution indicating reliability of the depth, calculating a map voting probability indicating a probability that the object occupies each grid in an occupancy grid map, in which a space included in the captured image is divided into a grid shape, based on a first probability distribution and a second probability distribution; and Voting is performed on each grid based on the map voting probability to create the occupancy grid map.

Images