Information processing device, autonomous vehicle, information processing device control method, program and storage medium

Abstract

To provide a technique for accurately determining a distance to an object.SOLUTION: An information processing device comprises: image acquisition means for acquiring first captured image of a first camera and a second captured image of a second camera; object recognition means for recognizing an object respectively from the first captured image and the second captured image; reliability calculation means for calculating reliability indicating likelihood that the object is an actual object respectively from the first captured image and the second captured image; determination means for determining whether an identical object exists in both the first captured image and the second captured image; and distance acquisition means for acquiring distance information to the object based on an object recognition result of the object recognition means for the captured image with higher reliability calculated by the reliability calculation means when the determination means determines that the identical object exists.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an autonomous vehicle, a control method for an information processing device, a program, and a storage medium. [Background technology]

[0002] Patent Document 1 discloses that a marker included in a captured image is associated with distance information from an autonomous work machine to the marker in advance and stored, and distance information of a marker detected in a captured image is acquired. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 014585 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology described in Patent Document 1, if the marker is partially or completely obscured by an obstruction, the marker cannot be adequately recognized, and as a result, distance information may not be obtained or may be erroneous.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique for obtaining the distance to an object with high accuracy. [Means for solving the problem]

[0006] To achieve the above object, an information processing device according to one aspect of the present invention comprises: an image acquisition means for acquiring a first captured image by a first camera and a second captured image by a second camera; an object recognition means for recognizing an object from each of the first captured image and the second captured image; a reliability calculation means for calculating a reliability indicating a likelihood that the object is the object for each of the first captured image and the second captured image; a determination means for determining whether or not an identical object exists between the first captured image and the second captured image; a distance acquisition means for acquiring distance information to the object based on an object recognition result of the object recognition means for the photographed image having a higher reliability calculated by the reliability calculation means when the determination means determines that the same object exists; The present invention is characterized by comprising: Effect of the Invention

[0007] According to the present invention, it is possible to obtain the distance to an object with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an overhead view of an autonomous vehicle according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an autonomous vehicle according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device (ECU) included in an autonomous vehicle according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of image conversion between captured images according to an embodiment. [Diagram 5] 1 is a flowchart showing a procedure of a process executed by an information processing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] <Overhead view of an autonomous vehicle> FIG. 1 shows an example of an overhead view of an autonomous vehicle according to this embodiment. In this embodiment, an example of an autonomous driving vehicle that recognizes the surrounding situation and autonomously controls the vehicle based on the recognition result will be described. The autonomous vehicle may be in any form as long as it is a moving body that operates autonomously, such as a three-wheeled vehicle or a four-wheeled vehicle. In this embodiment, the autonomous vehicle is a four-wheeled vehicle (electric vehicle) that can accommodate one or more passengers, such as micromobility.

[0011] 1, the autonomous vehicle 10 includes a front photographing camera 104, a rear photographing camera 105, a left oblique front photographing camera 106, a left oblique rear photographing camera 107, a right oblique front photographing camera 108, and a right oblique rear photographing camera 109. Note that, although the illustrated example includes six cameras, the number of cameras is not limited to this example and may be more or less than six.

[0012] The front photographing camera 104 is a camera installed in front of the autonomous vehicle 10 for photographing the area in front of the vehicle. The rear photographing camera 105 is a camera installed in the rear of the autonomous vehicle 10 for photographing the area behind the vehicle. The left oblique front photographing camera 106 is a camera installed on the left rear side of the autonomous vehicle 10 for photographing the area obliquely forward to the left of the vehicle.

[0013] The left oblique rear photographing camera 107 is a camera installed on the front left side of the autonomous vehicle 10 for photographing the left oblique rear of the vehicle. The right oblique front photographing camera 108 is a camera installed on the rear right side of the autonomous vehicle 10 for photographing the right oblique front of the vehicle. The right oblique rear photographing camera 109 is a camera installed on the front right side of the autonomous vehicle 10 for photographing the right oblique rear of the vehicle.

[0014] <Hardware configuration> 2 is a diagram showing an example of a hardware configuration of an autonomous vehicle according to this embodiment. The autonomous vehicle 10 includes an ECU 101, a storage device 102, a communication unit 103, and six cameras, namely, the front photographing camera 104 to the right oblique rear photographing camera 109 described in FIG.

[0015] The ECU 101 is one or more electronic control units and can function as an information processing device. The control operation of the autonomous vehicle 10 is realized by the ECU 101 reading and executing a computer program stored in the storage device 102. The storage device 102 is one or more memories that store various information. For example, the storage device 102 stores information received from other devices, computer programs that are read and executed by the ECU 101, and the like.

[0016] The communication unit 103 has a function of communicating with other devices via a network (not shown) in a wired or wireless manner. The functions of the six cameras, from the front photographing camera 104 to the right oblique rear photographing camera 109, are as described with reference to Fig. 1. The ECU 101 controls the operation of the autonomous vehicle 10 based on images captured by each of the cameras, from the front photographing camera 104 to the right oblique rear photographing camera 109.

[0017] <Functional configuration> 3 is a diagram showing an example of the functional configuration of an information processing device (ECU 101) provided in the autonomous vehicle according to this embodiment. The information processing device (ECU 101) includes a captured image acquisition unit 201, an object recognition unit 202, a reliability calculation unit 203, an image conversion unit 204, an identical object determination unit 205, an image selection unit 206, a distance acquisition unit 207, and a vehicle control unit 208.

[0018] The captured image acquisition unit 201 acquires one or more captured images captured by each of the six cameras, namely, the front capturing camera 104 to the right oblique rear capturing camera 109.

[0019] The object recognition unit 202 recognizes an object from the captured image acquired by the captured image acquisition unit 201. For example, a rectangular bounding box that surrounds the object is used. Here, the object may include fixed obstacles (e.g., buildings, curbs, walls, vending machines, traffic lights, etc.) that exist around the autonomous vehicle 10, and moving objects (e.g., people, animals, bicycles, other vehicles, bicycles, electric scooters, etc.), but an example of an object may be a person.

[0020] The reliability calculation unit 203 calculates a reliability indicating the likelihood that the object recognized by the object recognition unit 202 is the object. For example, when a person (whole body) is recognized as an object, the reliability calculation unit 203 calculates a reliability indicating the likelihood that the object is really a person. For example, consider a case where a part of a person (lower body) is hidden by an obstacle (for example, the bonnet of a vehicle).

[0021] In this case, although the object is recognized as a person (whole body), the reliability indicating the likelihood that this is really a person (whole body) may be calculated as, for example, 0.5 (wherein reliability 1.0 is the highest reliability and reliability 0 is the lowest reliability). Also, when the whole body of a person is photographed without being hidden by an obstacle, the reliability indicating the likelihood that the object is a person (whole body) may be calculated as, for example, 0.95. Note that, since various known methods can be used as a method for calculating the reliability, details will be omitted.

[0022] The image conversion unit 204 performs projective transformation of points on an image captured by one camera onto an image captured by another camera. For example, the image conversion unit 204 performs projective transformation of four points indicating the positions of four corners that represent a bounding box of an object recognized on an image captured by the front capturing camera 104 onto an image captured by the left oblique front capturing camera 106. Including this projective transformation, projective transformation is performed between the images captured by six cameras.

[0023] The same object determination unit 205 determines whether or not the same object exists between a plurality of captured images. For example, the same object determination unit 205 determines whether or not an object recognized on an image captured by the front capturing camera 104 is the same object as an object recognized on an image captured by the left oblique front capturing camera 106. The determination of whether or not the objects between the two captured images are the same can be performed, for example, as follows.

[0024] First, the image conversion unit 204 performs projective transformation of a first bounding box surrounding an object recognized in an image captured by the front photographing camera 104, for example, onto the image captured by the left oblique front photographing camera 106. Then, the same object determination unit 205 performs determination based on the overlap rate between a second bounding box surrounding an object recognized in the image captured by the left oblique front photographing camera 106 and the first bounding box obtained by projective transformation.

[0025] For example, the same object determination unit 205 determines that the same object exists between two captured images (in this example, the image captured by the front capturing camera 104 and the image captured by the left diagonal front capturing camera 106) when the overlap rate is equal to or greater than a threshold value (e.g., 50% or IoU 0.5). For example, IoU (Intersection over Union) may be used as the overlap rate. IoU is an index indicating the degree to which two regions overlap, and can be calculated by dividing the common part of the two regions by the union of the two regions (0≦IoU≦1).

[0026] The image selection unit 206 selects a captured image for acquiring distance information from among a plurality of captured images determined to include the same object by the identical object determination unit 205. For example, the captured image having a higher reliability calculated by the reliability calculation unit 203 is selected.

[0027] The distance acquisition unit 207 acquires distance information to an object based on the object recognition result of the object recognition unit 202 for the captured image selected by the image selection unit 206. The distance acquisition unit 207 acquires distance information based on table information that associates a position on the captured image with the depth distance from the camera to the corresponding position. For example, the distance acquisition unit 207 refers to the table information using information on a specific position of a bounding box of an object recognized in the captured image, and acquires distance information corresponding to that position.

[0028] The predetermined position of the bounding box of the object may be, for example, the coordinates of the vertical axis (y axis) of the position of the bottom side of the rectangle constituting the bounding box on the captured image. Alternatively, by using table information in which each coordinate of the captured image is associated with distance information, distance information corresponding to, for example, the coordinates (x, y) of the midpoint position of the bottom side of the rectangle constituting the bounding box may be acquired by referring to the table information.

[0029] The vehicle control unit 208 controls the operation of the autonomous vehicle 10 based on the distance information acquired by the distance acquisition unit 208. For example, when an object is recognized in front of the vehicle, the vehicle control unit 208 performs control to automatically execute a braking operation (brake operation) and / or a steering operation to avoid a collision. Also, when an object that may cause a collision is not recognized in front of the vehicle, the vehicle control unit 208 performs control to automatically execute an acceleration operation and / or a steering operation, and moves to a set destination.

[0030] <Processing> Next, a procedure of a process performed by the information processing device (ECU 101) according to the present embodiment will be described with reference to the explanatory diagram of Fig. 4 and the flowchart of Fig. 5. Fig. 4 is an explanatory diagram of image conversion between captured images according to one embodiment.

[0031] In S501, the captured image acquisition unit 201 acquires one or more captured images captured by each of the six cameras, the front capturing camera 104 to the right oblique rear capturing camera 109.

[0032] In S502, the object recognition unit 202 recognizes an object from the captured image acquired by the captured image acquisition unit 201. In this embodiment, an example of recognizing a person as an object will be described. The object recognition unit 202 recognizes a person using a rectangular bounding box. Various methods can be used for object recognition. For example, the characteristics of a person (whole body) are learned in advance by machine learning, and recognition processing can be performed based on whether or not the feature amount within the bounding box of the object recognized by the object recognition unit 202 matches the characteristics of the person (whole body), or to what extent.

[0033] 4 is an image captured by the front photographing camera 104, and image 402 is an image captured by the left diagonal front photographing camera 106. Although images captured by the other four cameras are also captured in S501, the following description will be given using the images captured by these two cameras.

[0034] A person 440 whose lower body is partially obscured by the bonnet 430 of the autonomous vehicle 10 and cannot be seen is recognized from the captured image 401. On the other hand, a person 450 whose entire body is visible is recognized from the captured image 402. It is assumed that no object is recognized from the captured images acquired by the other cameras.

[0035] In S503, the reliability calculation unit 203 calculates a reliability indicating the likelihood that an object recognized from an image captured by each camera by the object recognition unit 202 is the object. In this embodiment, the reliability calculation unit 203 calculates a reliability indicating the likelihood that the recognized object is really a person.

[0036] For example, consider a case where a part (lower body) of a person is hidden by an obstacle (e.g., a vehicle bonnet 430) as shown in a captured image 401. In this case, although the object is recognized as a person (whole body), the reliability indicating the likelihood that this is really a person (whole body) may be calculated as, for example, 0.5. Note that a reliability of 1.0 indicates the highest reliability, and a reliability of 0 indicates the lowest reliability. Also, as shown in a captured image 402, when a person's whole body is captured without being hidden by an obstacle, the reliability indicating the likelihood that the object is a person (whole body) may be calculated as, for example, 0.95. Note that various known methods can be used as a method of calculating the reliability.

[0037] In S504, the image conversion unit 204 performs projective transformation of points on an image captured by a certain camera onto an image captured by another camera. For example, the image conversion unit 204 performs projective transformation of a bounding box 410 of an object (person 440) recognized on an image captured by the front capturing camera 104 onto an image captured by the left oblique front capturing camera 106. The bounding box 410 is defined by four points 411, 412, 413, and 414 indicating the positions of the four corners. The bounding box 410 subjected to projective transformation is projectively transformed onto the captured image 402 to become a bounding box 410' defined by four points 411', 412', 413', and 414'.

[0038] In S505, the same object determination unit 205 determines whether or not the same object exists between the multiple captured images. If it is determined that the same object exists between the captured images, the process proceeds to S506. On the other hand, if it is determined that the same object does not exist between the captured images, the process proceeds to S507.

[0039] For example, the same object determination unit 205 determines whether an object recognized on the captured image 401 of the forward photographing camera 104 and an object recognized on the captured image 402 of the left oblique forward photographing camera 106 are the same object. As shown in FIG. 4, the same object determination unit 205 performs the determination based on the overlap rate of the feature amount between a second bounding box 420 surrounding the object recognized in the captured image 402 of the left oblique forward photographing camera 106 and a first bounding box 410' obtained by projective transformation. The second bounding box 420 is a rectangle defined by four points 421, 422, 423, and 424. For example, the same object determination unit 205 determines that the same object exists between the photographed image 401 and the photographed image 402 when the overlap rate is equal to or greater than a threshold value (e.g., IoU 0.5). In this embodiment, it is determined that the same object exists between the photographed image 401 and the photographed image 402.

[0040] In S506, the image selection unit 206 selects a captured image from which distance information is to be obtained from among the captured images determined by the identical object determination unit 205 to include the same object. For example, the captured image having a higher reliability calculated by the reliability calculation unit 203 is selected. In the example of FIG. 4, the reliability calculated for the object in the captured image 401 is 0.5, and the reliability calculated for the object in the captured image 402 is 0.95, so the captured image 402 is selected. Then, the distance acquisition unit 207 acquires distance information to the object based on the object recognition result of the object recognition unit 202 for the captured image selected by the image selection unit 206. For example, the distance information is acquired based on table information that associates a position on the captured image with a depth distance from the camera to the corresponding position. It is assumed that table information is held for each camera. The contents of the table information are not limited to this, and may be a table of depth distances from the front end or rear end of the autonomous vehicle 10 to the corresponding position.

[0041] In the example of FIG. 4, distance information in the table information of the captured image 402 (left diagonally forward photographing camera 106) corresponding to the position of the line connecting two points 422, 423 on the bottom side of the bounding box 420 of the captured image 402 is obtained.

[0042] In S507, the image selection unit 206 selects a captured image including an object. The distance acquisition unit 207 acquires distance information to the object based on the object recognition result of the object recognition unit 202 for the captured image selected by the image selection unit 206. The method of acquiring the distance information is the same as in S506, and the distance information can be acquired based on table information corresponding to the captured image (camera) selected by the image selection unit 206.

[0043] In S508, the vehicle control unit 208 controls the operation of the autonomous vehicle 10 based on the distance information acquired by the distance acquisition unit 207. For example, when an object is recognized in front of the vehicle, the vehicle control unit 208 performs control to automatically execute a braking operation (braking operation) and / or a steering operation to avoid a collision.

[0044] In S509, the information processing device (ECU 101) determines whether or not to continue the process. If the process is to be continued, the process returns to S501. On the other hand, if the process is not to be continued, the series of processes in Fig. 5 is terminated. For example, the process may be terminated when the set destination is reached or when a predetermined operation is performed by the occupant of the autonomous vehicle 10.

[0045] The order of the processes in the flowchart is not limited to the order described, and other orders may be used. In addition, other processes may be added as appropriate, and some processes may be skipped or excluded.

[0046] As described above, in this embodiment, when it is determined that the same object is present in multiple captured images, distance information to the object is obtained based on the object recognition result for the captured image that has a higher reliability indicating the likelihood that the object is the object in question.

[0047] This makes it possible to accurately obtain the distance to an object in a captured image from an image suitable for obtaining distance information. For example, when a part of an object is blocked by the bonnet of the vehicle or another object in an image captured by a camera capturing an image in front of the vehicle, it is possible to prevent the position of the object from being erroneously obtained. Therefore, it is possible to perform highly accurate control in the autonomous control of the vehicle using the obtained distance information.

[0048] [Variations] In the above embodiment, an example has been described in which the captured image with the higher reliability is selected in S506, and distance information is acquired based on the selected captured image, but this is not limiting. In the example of Fig. 4, the captured image 402 is determined to have a higher reliability, but the object (person) is located closer to the front of the front capturing camera 104 than to the left diagonal front capturing camera. Therefore, more accurate distance information can be acquired by acquiring distance information based on the captured image 401 of the front capturing camera 104.

[0049] 4, a bounding box 420 in the captured image 402 may be projected onto the captured image 401 by the image conversion unit 204. Then, distance information may be obtained from the position of the bottom side of the projectively transformed bounding box 420' (four points 421', 422', 423', 424') and table information corresponding to the captured image 401 (front capturing camera 104).

[0050] Furthermore, a priority may be determined in advance for each camera with respect to which captured image (camera) the corresponding table information can be used. For example, the priority of the front capturing camera 104 may be greater than the priority of the left oblique front capturing camera 106, or the priority of the front capturing camera 104 may be greater than the priority of the right oblique front capturing camera 108. Similarly, the priority of the rear capturing camera 105 may be greater than the priority of the left oblique rear capturing camera 107, or the priority of the rear capturing camera 105 may be greater than the right oblique rear capturing camera 109.

[0051] Also, in S506, it is assumed that the reliability calculated by the reliability calculation unit 203 for each captured image is almost the same, that is, the difference in reliability is equal to or less than a threshold. In that case, the distance acquisition unit 207 may acquire distance information to the object based on the object recognition result of the object recognition unit 202 for the captured image in which the position of the object in the captured image is closer to the center position of the captured image. This is because the closer the position is to the center of the captured image, the more accurate distance information can be acquired. For example, the distance information to the object may be acquired based on the object recognition result of the object recognition unit 202 for the captured image in which the distance between the center of gravity of the bounding box of the object and the center position of the captured image is shorter.

[0052] In addition, in S507, since the same object is not included in multiple captured images, distance information to an object in a captured image is obtained using table information corresponding to the captured image (camera), and the accuracy of the distance information may be low. Therefore, in such a case, the content of vehicle control based on the object recognition result may be changed. For example, the value of the predetermined distance when performing an avoidance operation by performing a braking operation and / or a steering operation when the object is a predetermined distance away may be changed to a larger value. This allows the avoidance operation to be performed early at a position away from the object, thereby improving safety.

[0053] Also, when the same object is not included in multiple captured images and the reliability indicating the likelihood that an object detected in a captured image is the object is low and is equal to or less than a predetermined value, a captured image of a frame earlier than the frame of the captured image may be referenced. For example, the movement of the object in the captured images may be tracked, and the position of the bounding box when the entire object (e.g., the entire body of a person) was recognized may be estimated in the captured image of the current frame, and distance information may be obtained from table information based on the estimated position.

[0054] In the above embodiment, the process according to whether the same object is captured between the image captured by the front photographing camera 104 and the image captured by the left oblique front photographing camera 106 (or the right oblique front photographing camera 108) has been described, but the present invention is not limited to this example. Similarly, the same process can be performed depending on whether the same object is captured between the image captured by the rear photographing camera 105 and the image captured by the left oblique rear photographing camera 107 (or the right oblique rear photographing camera 109). Furthermore, the same process can be performed depending on whether the same object is captured between the image captured by the left oblique front photographing camera 106 and the image captured by the left oblique rear photographing camera 107. And, the same process can be performed depending on whether the same object is captured between the image captured by the right oblique front photographing camera 108 and the image captured by the right oblique rear photographing camera 109. Also, the same process can be performed depending on whether the same object is captured between the images captured by three or more cameras.

[0055] <Summary of the embodiment> 1. The information processing device (101) according to the above embodiment is an image acquisition means (201) for acquiring a first captured image (401) by a first camera (104) and a second captured image (402) by a second camera (106); an object recognition means (202) for recognizing an object from each of the first captured image and the second captured image; a reliability calculation means (203) for calculating a reliability indicating a likelihood that the object is the object for each of the first captured image and the second captured image; a determination means (205) for determining whether or not an identical object exists between the first captured image and the second captured image; a distance acquisition means (207) for acquiring distance information to the object based on an object recognition result of the object recognition means for the photographed image having a higher reliability calculated by the reliability calculation means when the determination means determines that the same object exists; Equipped with.

[0056] This makes it possible to accurately obtain the distance to an object in a captured image from a captured image suitable for obtaining distance information.

[0057] 2. In the information processing device (101) according to the above embodiment, The method further includes a transformation means (204) for projectively transforming a first bounding box (410) surrounding an object recognized in the first captured image onto the second captured image, The determination means makes the determination based on an overlap rate (IoU) between a second bounding box (420) surrounding the object recognized in the second captured image and the first bounding box subjected to the projective transformation.

[0058] This makes it possible to accurately calculate the overlap rate between objects (bounding boxes), and therefore to accurately determine whether the objects are the same object.

[0059] 3. In the information processing device (101) according to the above embodiment, The determining means determines that the same object is present between the first captured image and the second captured image when the overlap rate is equal to or greater than a threshold value.

[0060] This makes it possible to easily determine whether the object between the captured images is the same object.

[0061] 4. In the information processing device (101) according to the above embodiment, The distance acquisition means acquires the distance information based on a predetermined position of a bounding box in one of the first bounding box in the first captured image and the second bounding box in the second captured image, whichever has a higher reliability.

[0062] This makes it possible to obtain more accurate distance information.

[0063] 5. In the information processing device (101) according to the above embodiment, The predetermined position is the position of the bottom side that constitutes the bounding box.

[0064] In this way, by using the position of the bottom edge of the bounding box of an object (for example, a person) (the position of the person's feet) as a reference, it is possible to obtain distance information from the camera to that position (the position of the ground) with high accuracy.

[0065] 6. In the information processing device (101) according to the above embodiment, the first camera is a camera (104) that is arranged in front of an autonomous vehicle (10) equipped with the information processing device and captures an image of the area in front of the autonomous vehicle; The second camera is a photographing camera (106, 108) that is arranged on the left or right side of the autonomous vehicle and photographs a left diagonal front or right diagonal front of the autonomous vehicle, The system further comprises a second transformation means (204) for performing projective transformation of a second bounding box (420) surrounding the object recognized in the second captured image onto the first captured image when the second reliability of the object recognized in the second captured image is higher than the first reliability of the object recognized in the first captured image, The distance acquisition means acquires the distance information based on a predetermined position of the second bounding box (420') that has been subjected to the projective transformation.

[0066] This makes it possible to obtain distance information using an image captured by a front-facing camera, which can obtain more accurate distance information from the object position in the captured image.

[0067] 7. In the information processing device (101) according to the above embodiment, A part of the object is not recognized in the first captured image, and the entire object is recognized in the second captured image.

[0068] In this way, by using a captured image in which the entire object is recognized, rather than a captured image in which part of the object is not recognized due to an obstruction or the like, it is possible to obtain accurate distance information. Furthermore, when distance information is obtained using a captured image in which part of the object is not recognized, there is a possibility that erroneous distance information will be obtained when, for example, distance information is obtained using table information in which a specific position of the object in the captured image (e.g., a person's feet) is associated with distance information at that position in the captured image. However, according to this feature, it is possible to prevent such erroneous distance information from being obtained.

[0069] 8. In the information processing device (101) according to the above embodiment, The predetermined position is the position of the bottom side that constitutes the second bounding box that has been subjected to the projective transformation.

[0070] In this way, by using the position of the bottom edge of the bounding box of an object (for example, a person) (the position of the person's feet) as a reference, it is possible to obtain distance information from the camera to that position (the position of the ground) with high accuracy.

[0071] 9. In the information processing device (101) according to the above embodiment, The first camera is a camera (105) that is arranged at the rear of an autonomous vehicle (10) equipped with the information processing device and captures an image of the rear of the autonomous vehicle, The second camera is a photographing camera (107, 109) that is arranged on the left or right side of the autonomous vehicle and photographs the left diagonally rear or right diagonally rear of the autonomous vehicle, The system further includes a second transformation means (204) for performing projective transformation of a second bounding box surrounding the object recognized in the second captured image onto the first captured image when the second reliability of the object recognized in the second captured image is higher than the first reliability of the object recognized in the first captured image, The distance acquisition means acquires the distance information based on a predetermined position of the second bounding box that has been subjected to the projective transformation.

[0072] This makes it possible to obtain distance information using images captured by a rear-view camera that can obtain more accurate distance information from the object positions in the captured images.

[0073] 10. In the information processing device (101) according to the above embodiment, A part of the object is not recognized in the first captured image, and the entire object is recognized in the second captured image.

[0074] In this way, by using a captured image in which the entire object is recognized, rather than a captured image in which part of the object is not recognized due to an obstruction or the like, it is possible to obtain accurate distance information. Furthermore, when distance information is obtained using a captured image in which part of the object is not recognized, there is a possibility that erroneous distance information will be obtained when, for example, distance information is obtained using table information in which a specific position of the object in the captured image (e.g., a person's feet) is associated with distance information at that position in the captured image. However, according to this feature, it is possible to prevent such erroneous distance information from being obtained.

[0075] 11. In the information processing device (101) according to the above embodiment, The predetermined position is the position of the bottom side that constitutes the second bounding box that has been subjected to the projective transformation.

[0076] In this way, by using the position of the bottom edge of the bounding box of an object (e.g., a person) (the position of the person's feet) as a reference, it is possible to obtain distance information from the camera to that position (the position of the ground) with high accuracy. In addition, by using the position of the bottom edge of the bounding box after projection transformation, it is possible to properly recognize the position of the person's feet even if the lower half of the person's body is hidden by, for example, a bonnet in the captured image after projection transformation. Therefore, it is possible to obtain accurate distance information.

[0077] 12. In the information processing device (101) according to the above embodiment, The distance acquisition means acquires the distance information based on table information that associates a position on a captured image with a depth distance from a camera to the corresponding position.

[0078] This makes it possible to easily obtain distance information to an object as long as the predetermined position of the object in the captured image is known.

[0079] 13. In the information processing device (101) according to the above embodiment, When the determining means determines that no identical object exists, the distance acquiring means acquires distance information to each object based on the object recognition result of the object recognizing means for each captured image.

[0080] This makes it possible to obtain distance information from each camera to an object included in each captured image, even if the same object does not exist.

[0081] 14. In the information processing device (101) according to the above embodiment, The information processing device according to any one of claims 1 to 9, characterized in that, when the difference in reliability calculated by the reliability calculation means is equal to or less than a threshold value, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image in which the position of the object in the captured image is closer to a center position of the captured image.

[0082] In this way, if there is little difference in reliability (i.e., if the objects are about the same degree of occlusion, or if none of the objects are occluded), accurate distance information can be obtained by using the image that shows the object closer to the front of the camera.

[0083] 15. The autonomous vehicle (10) according to the above embodiment is An autonomous vehicle equipped with the information processing device (101) of the above embodiment, The autonomous vehicle further includes a vehicle control means (208) for controlling the operation of the autonomous vehicle based on the distance information acquired by the distance acquisition means.

[0084] This enables automatic control of the vehicle based on highly accurate object recognition, thereby enabling automatic control that improves the safety of occupants and / or people outside the vehicle.

[0085] 16. The control method of the information processing device (101) according to the above embodiment is An image acquisition step (S501) of acquiring a first captured image (401) by a first camera (104) and a second captured image (402) by a second camera (106); an object recognition step (S502) of recognizing an object from each of the first captured image and the second captured image; A reliability calculation step (S503) of calculating a reliability indicating a likelihood that the object is the object for each of the first captured image and the second captured image; A determination step (S505) of determining whether or not an identical object exists between the first captured image and the second captured image; a distance acquisition step (S506) of acquiring distance information to the object based on the object recognition result of the object recognition step for the photographed image having a higher reliability calculated by the reliability calculation step when the determination step determines that the same object exists; has.

[0086] This makes it possible to accurately obtain the distance to an object in a captured image from a captured image suitable for obtaining distance information.

[0087] 17. The program according to the above embodiment is The program causes a computer to function as the information processing device of the above embodiment.

[0088] This makes it possible to realize the functions of the information processing device as a program.

[0089] 18. The storage medium according to the above embodiment is A storage medium storing a program for causing a computer to function as the information processing device of the above embodiment.

[0090] This makes it possible to realize the functions of the information processing device as a storage medium.

[0091] <Other embodiments> Furthermore, a program for realizing one or more functions described in each embodiment can be supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device can read and execute the program. The present invention can also be realized in such an embodiment.

[0092] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0093] 10: autonomous vehicle, 101: ECU, 104: front-facing camera, 106: left-diagonal front-facing camera, 201: captured image acquisition unit, 202: object recognition unit, 203: reliability calculation unit, 204: image conversion unit, 205: same object determination unit, 206: image selection unit, 207: distance acquisition unit, 208: vehicle control unit

Claims

1. An information processing device, an image acquisition means for acquiring a first captured image by a first camera and a second captured image by a second camera; an object recognition means for recognizing an object from each of the first captured image and the second captured image; a reliability calculation means for calculating a reliability indicating a likelihood that the object is the object for each of the first captured image and the second captured image; a determination means for determining whether or not an identical object exists between the first captured image and the second captured image; a distance acquisition means for acquiring distance information to the object based on an object recognition result of the object recognition means for the photographed image having a higher reliability calculated by the reliability calculation means when the determination means determines that the same object exists; An information processing device comprising:

2. a transformation unit that performs projective transformation of a first bounding box surrounding an object recognized in the first captured image onto the second captured image; 2. The information processing apparatus according to claim 1, wherein the determination means makes the determination based on an overlap rate between a second bounding box surrounding an object recognized in the second captured image and the first bounding box obtained by the projective transformation.

3. 3 . The information processing apparatus according to claim 2 , wherein the determining means determines that the same object exists between the first captured image and the second captured image when the overlap rate is equal to or greater than a threshold value.

4. The information processing device according to claim 2, characterized in that the distance acquisition means acquires the distance information based on a predetermined position of a bounding box in a captured image having a higher reliability between the first bounding box in the first captured image and the second bounding box in the second captured image.

5. 5. The information processing apparatus according to claim 4, wherein the predetermined position is a position of a bottom side that constitutes the bounding box.

6. the first camera is a camera disposed in front of an autonomous vehicle including the information processing device and captures an image of a scene ahead of the autonomous vehicle; the second camera is a photographing camera disposed on the left or right side of the autonomous vehicle and photographs an area diagonally forward to the left or right of the autonomous vehicle; a second transformation means for performing a projective transformation of a second bounding box surrounding the object recognized in the second captured image onto the first captured image when a second reliability of the object recognized in the second captured image is higher than a first reliability of the object recognized in the first captured image, The information processing apparatus according to claim 1 , wherein the distance acquisition means acquires the distance information based on a predetermined position of the second bounding box obtained by the projective transformation.

7. 7. The information processing apparatus according to claim 6, wherein a part of an object is not recognized in the first captured image, and the whole of the object is recognized in the second captured image.

8. 7. The information processing apparatus according to claim 6, wherein the predetermined position is a position of a lower side constituting the second bounding box obtained by the projective transformation.

9. the first camera is a camera disposed at a rear of an autonomous vehicle including the information processing device and photographs an image of a rear area of ​​the autonomous vehicle; The second camera is a photographing camera disposed on the left or right side of the autonomous vehicle and photographs an area diagonally rearward to the left or right of the autonomous vehicle, a second transformation means for performing a projective transformation of a second bounding box surrounding the object recognized in the second captured image onto the first captured image when a second reliability of the object recognized in the second captured image is higher than a first reliability of the object recognized in the first captured image, The information processing apparatus according to claim 1 , wherein the distance acquisition means acquires the distance information based on a predetermined position of the second bounding box obtained by the projective transformation.

10. 10. The information processing apparatus according to claim 9, wherein a part of an object is not recognized in the first captured image, and the whole of the object is recognized in the second captured image.

11. 10. The information processing apparatus according to claim 9, wherein the predetermined position is a position of a bottom side constituting the second bounding box obtained by the projective transformation.

12. 2 . The information processing apparatus according to claim 1 , wherein the distance acquisition means acquires the distance information based on table information that associates a position on a captured image with a depth distance from the camera to the corresponding position.

13. The information processing device according to claim 1, characterized in that, when the determination means determines that no identical object exists, the distance acquisition means acquires distance information to each object based on the object recognition result of the object recognition means for each captured image.

14. The information processing device according to claim 1, characterized in that, when the difference in reliability calculated by the reliability calculation means is equal to or less than a threshold value, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image in which the position of the object in the captured image is closer to a center position of the captured image.

15. An autonomous vehicle comprising the information processing device according to any one of claims 1 to 14, An autonomous vehicle comprising a vehicle control means for controlling an operation of the autonomous vehicle based on distance information acquired by the distance acquisition means.

16. A method for controlling an information processing device, comprising: an image acquiring step of acquiring a first captured image by a first camera and a second captured image by a second camera; an object recognition step of recognizing an object from each of the first captured image and the second captured image; a reliability calculation step of calculating a reliability indicating a likelihood that the object is the object for each of the first captured image and the second captured image; a determining step of determining whether an identical object exists between the first captured image and the second captured image; a distance acquisition step of acquiring distance information to the object based on an object recognition result of the object recognition step for the photographed image having a higher reliability calculated by the reliability calculation step when the determination step determines that the same object exists; 13. A method for controlling an information processing apparatus comprising the steps of:

17. A program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

18. A storage medium storing a program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Images