Information processing device, autonomous vehicle, method for controlling information processing device, storage medium, and

Abstract

The invention provides an information processing apparatus, an autonomous vehicle, a control method of the information processing apparatus, a storage medium, and a program product, for acquiring a distance to an object with high precision. An information processing device is provided with: an image acquisition means for acquiring a first captured image of a first camera and a second captured image of a second camera; an object recognition means that recognizes an object on the basis of each of the first captured image and the second captured image; a reliability calculation means for calculating, for each of the first captured image and the second captured image, a reliability indicating the probability that the object is a relevant object; a determination means for determining whether or not there is the same object between the first captured image and the second captured image; and a distance acquisition means that, when the determination means determines that there is the same object, acquires distance information from the object on the basis of the object recognition results of the object recognition means for the captured image having the high reliability calculated by the reliability calculation means.

Description

Technical Field

[0001] The present invention relates to an information processing device, an autonomous vehicle, a control method of the information processing device, and a storage medium. Background Art

[0002] Patent Document 1 discloses that a mark included in a captured image and distance information from an autonomous working machine to the mark are stored in advance in association with each other, and the distance information of the mark detected in a captured image is acquired.

[0003] Prior art literature Patent Literature Patent Document 1: International Publication No. 2021 / 014585 Summary of the invention Problems to be solved by the invention However, in the technology described in Patent Document 1, if a part or all of the mark is blocked by the presence of an obstruction, the mark cannot be fully recognized, and as a result, distance information may not be acquired, or erroneous distance information may be acquired.

[0004] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a technology for acquiring the distance to an object with high accuracy.

[0005] Means used to solve problems An information processing device according to one embodiment of the present invention for achieving the above-mentioned object is characterized by comprising: An image acquisition mechanism, which acquires a first image captured by the first camera and a second image captured by the second camera; An object recognition mechanism for respectively recognizing an object based on the first captured image and the second captured image; A reliability calculation unit that calculates the reliability indicating the probability that the object is a related object for each of the first captured image and the second captured image; a determination unit configured to determine whether the same object exists between the first captured image and the second captured image; and The distance acquisition means acquires the distance information to the object based on the object recognition result of the captured image of the party with higher reliability calculated by the reliability calculation means by the object recognition means when the determination means determines that the same object exists.

[0006] In addition, a method for controlling an information processing device according to one embodiment of the present invention to achieve the above-mentioned object is a method for controlling an information processing device, characterized by comprising: An image acquisition step, in which a first image captured by the first camera and a second image captured by the second camera are acquired; an object recognition step, in which objects are recognized based on the first captured image and the second captured image respectively; a reliability calculation step, in which reliability indicating the probability that the object is a related object is calculated for each of the first captured image and the second captured image; a determination step, in which it is determined whether the same object exists between the first captured image and the second captured image; and A distance acquisition step, in which, when it is determined by the determination step that the same object exists, the distance information to the object is acquired based on the object recognition result of the object recognition step on the captured image of the party with higher reliability calculated by the reliability calculation step.

[0007] Effects of the Invention According to the present invention, the distance to an object can be acquired with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 FIG. 1 is a top view of an autonomous vehicle according to an embodiment.

[0009] Figure 2 This is a diagram showing an example of a hardware configuration of an autonomous vehicle according to an embodiment.

[0010] Figure 3 This is a diagram showing an example of a functional configuration of an information processing device (ECU) included in an autonomous vehicle according to an embodiment.

[0011] Figure 4 This is a diagram for explaining image conversion between captured images according to one embodiment.

[0012] Figure 5 This is a flowchart showing the procedure of a process performed by an information processing device according to one embodiment.

[0013] Description of Reference Numerals 10: autonomous vehicle; 101: ECU; 104: front camera; 106: left oblique front camera; 201: captured image acquisition unit; 202: object recognition unit; 203: reliability calculation unit; 204: image transformation unit; 205: same object determination unit; 206: image selection unit; 207: distance acquisition unit; 208: vehicle control unit. DETAILED DESCRIPTION

[0014] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention involved in the technical solution. In addition, the combination of features described in the embodiments may not all be necessary for the invention. It is also possible to arbitrarily combine two or more of the multiple features described in the embodiments. In addition, the same figure mark is marked on the same or similar structure, and repeated description is omitted.

[0015] <Top view of autonomous vehicle> Figure 1 An example of a top view of an autonomous vehicle according to the present embodiment is shown. In the present embodiment, an autonomous driving vehicle that recognizes surrounding conditions and autonomously controls the vehicle based on the recognition result is used as an example for explanation. The autonomous vehicle may be any type of vehicle as long as it is a moving body that moves autonomously, such as a three-wheeled vehicle or a four-wheeled vehicle. In the present embodiment, it is a four-wheeled vehicle (electric vehicle) that can accommodate one or more passengers, such as a micro mobile vehicle.

[0016] like Figure 1 As shown, the autonomous vehicle 10 includes a front camera 104, a rear camera 105, a left oblique front camera 106, a left oblique rear camera 107, a right oblique front camera 108, and a right oblique rear camera 109. In addition, in the example shown in the figure, six cameras are provided, but the number of cameras is not limited to this example, and may be more than six or less than six.

[0017] The front camera 104 is a camera provided in front of the autonomous vehicle 10 for photographing the front of the vehicle. The rear camera 105 is a camera provided in rear of the autonomous vehicle 10 for photographing the rear of the vehicle. The left oblique front camera 106 is a camera provided in the rear left side of the autonomous vehicle 10 for photographing the left oblique front of the vehicle.

[0018] The left oblique rearward shooting camera 107 is a camera provided at the left front side of the autonomous vehicle 10 for shooting the left oblique rearward side of the vehicle. The right oblique frontward shooting camera 108 is a camera provided at the right rear side of the autonomous vehicle 10 for shooting the right oblique frontward side of the vehicle. The right oblique rearward shooting camera 109 is a camera provided at the right front side of the autonomous vehicle 10 for shooting the right oblique rearward side of the vehicle.

[0019] <Hardware Structure> Figure 2 FIG. 1 is a diagram showing an example of a hardware configuration of an autonomous vehicle according to the present embodiment. The autonomous vehicle 10 includes an ECU 101, a storage device 102, a communication unit 103, and a Figure 1 The six cameras described in the figure include the front shooting camera 104 to the right oblique rear shooting camera 109.

[0020] ECU101 is one or more electronic control units that can function as an information processing device. The control action of the autonomous vehicle 10 is realized by the ECU101 reading and executing a computer program stored in the storage device 102. The storage device 102 is one or more memories that store various information. For example, information received from other devices, computer programs read and executed by the ECU101, etc. are stored.

[0021] The communication unit 103 has a function of communicating with other devices via a network (not shown) in a wired or wireless manner. The functions of the six cameras, the front camera 104 to the right oblique rear camera 109, are as described in detail in the following description. Figure 1 The ECU 101 controls the operation of the host vehicle 10 based on the images captured by the front camera 104 to the right rear camera 109 .

[0022] <Functional Structure> Next, Figure 3 1 is a diagram showing an example of the functional configuration of an information processing device (ECU 101) included in the autonomous vehicle according to the present embodiment. The information processing device (ECU 101) includes a captured image acquisition unit 201, an object recognition unit 202, a reliability calculation unit 203, an image conversion unit 204, an identical object determination unit 205, an image selection unit 206, a distance acquisition unit 207, and a vehicle control unit 208.

[0023] The captured image acquisition unit 201 acquires one or more captured images acquired by each of the six cameras, namely, the front capturing camera 104 to the right oblique rear capturing camera 109 .

[0024] The object recognition unit 202 recognizes an object based on the captured image acquired by the captured image acquisition unit 201. For example, a rectangular bounding box surrounding the object is used. Here, the object may include fixed obstacles (e.g., buildings, curbs, walls, vending machines, traffic lights, etc.) and moving objects (e.g., people, animals, bicycles, other vehicles, motor vehicles, electric boards, etc.) existing around the autonomous vehicle 10, but as an example, it can be set to a person.

[0025] The reliability calculation unit 203 calculates the reliability indicating the probability that the object recognized by the object recognition unit 202 is the object of interest. For example, when a person (whole body) is recognized as an object, the reliability indicating the probability that the object is indeed a person is calculated. For example, consider a case where a part of the person (lower body) is blocked by an obstacle (e.g., the hood of a vehicle).

[0026] In this case, although the object is recognized as a person (full body), the reliability indicating the probability that it is indeed a person (full body) can be calculated as 0.5, for example (reliability 1.0 is the highest reliability, and reliability 0 is the lowest reliability). In addition, in the case where the whole body of a person is not blocked by an obstacle and is photographed, the reliability indicating the probability that the object is a person (full body) can be calculated as 0.95, for example. In addition, as a method for calculating the reliability, various known methods can be used, so detailed description is omitted.

[0027] The image transformation unit 204 performs projection transformation on the points on the image captured by a certain camera onto the images captured by other cameras. For example, the image transformation unit 204 performs projection transformation on the four points representing the positions of the four corners of the bounding box of the object recognized in the image captured by the front camera 104 onto the image captured by the left oblique front camera 106. Including this projection transformation, projection transformation is performed between the images captured by the six cameras.

[0028] The same object determination unit 205 determines whether the same object exists between the plurality of captured images. For example, the same object determination unit 205 determines whether the object recognized in the captured image of the front camera 104 is the same object as the object recognized in the captured image of the left oblique front camera 106. For example, the determination of whether the objects between the two captured images are the same can be performed as follows.

[0029] First, the image conversion unit 204, for example, performs a projection transformation of a first bounding box surrounding an object recognized in an image captured by the front camera 104 onto an image captured by the left oblique front camera 106. Then, the same object determination unit 205 performs a determination based on an overlap ratio between a second bounding box surrounding an object recognized in the image captured by the left oblique front camera 106 and the first bounding box after the projection transformation.

[0030] For example, when the overlap rate is greater than a threshold value (e.g., 50% or IoU 0.5), the same object determination unit 205 determines that the same object exists between two captured images (in this example, the captured image of the front camera 104 and the left oblique front camera 106). As the overlap rate, for example, IoU (Intersection over Union) can be used. IoU is an indicator indicating the degree of overlap between two regions, and can be obtained by dividing the common part of the two regions by the union of the two regions (0≤IoU≤1).

[0031] The image selection unit 206 selects a captured image for acquiring distance information from a plurality of captured images determined as including the same object by the same object determination unit 205. For example, the captured image having a higher reliability calculated by the reliability calculation unit 203 is selected.

[0032] The distance acquisition unit 207 acquires the distance information to the object based on the object recognition result of the object recognition unit 202 on the captured image selected by the image selection unit 206. The distance acquisition unit 207 acquires the distance information based on the table information in which the position on the captured image is associated with the depth distance from the camera to the position. For example, the distance information corresponding to the position of the bounding box of the object recognized in the captured image is obtained by referring to the table information.

[0033] The predetermined position of the bounding box of the object may be, for example, the coordinate of the vertical axis (y axis) on the captured image of the position of the lower side of the rectangle constituting the bounding box. Alternatively, table information in which each coordinate of the captured image is associated with the distance information may be used, and the distance information corresponding to the coordinate (x, y) of the midpoint position of the lower side of the rectangle constituting the bounding box may be obtained by referring to the table information.

[0034] The vehicle control unit 208 controls the action of the autonomous vehicle 10 based on the distance information acquired by the distance acquisition unit 208. For example, when an object is recognized in front of the vehicle, the braking operation (brake operation) and / or steering operation are automatically performed to avoid a collision. In addition, when no object with which a collision may occur is recognized in front of the vehicle, the acceleration operation and / or steering operation are automatically performed to move to the set destination.

[0035] <Processing> Next, refer to Figure 4 Illustrations and Figure 5 The flowchart of FIG. 1 illustrates the steps of processing executed by the information processing device (ECU 101 ) according to the present embodiment. Figure 4 This is a diagram for explaining image conversion between captured images according to one embodiment.

[0036] In S501 , the captured image acquisition unit 201 acquires one or more captured images acquired by each of the six cameras, namely, the front capturing camera 104 to the right oblique rear capturing camera 109 .

[0037] In S502, the object recognition unit 202 recognizes an object based on the captured image acquired by the captured image acquisition unit 201. In this embodiment, an example of recognizing a person as an object is described. The object recognition unit 202 recognizes the person using a rectangular bounding box. Various methods can be used for object recognition. For example, the features of a person (whole body) can be learned in advance through machine learning, and recognition processing can be performed based on whether the feature quantity within the bounding box of the object recognized by the object recognition unit 202 matches the features of the person (whole body) or to what extent they match.

[0038] For example, Figure 4The captured image 401 is a captured image acquired by the front capturing camera 104, and the captured image 402 is a captured image acquired by the left oblique front capturing camera 106. Although the captured images acquired by the other four cameras are acquired in S501, the captured images acquired by these two cameras are used for description here.

[0039] A person 440 whose lower body is partially hidden by the hood 430 of the host vehicle 10 is recognized from the captured image 401. On the other hand, a person 450 whose entire body can be seen is recognized from the captured image 402. It is assumed that no object is recognized from the captured images acquired by other cameras.

[0040] In S503, the reliability calculation unit 203 calculates the reliability indicating the probability that the object recognized from the captured image of each camera by the object recognition unit 202 is the object of interest. In this embodiment, the reliability indicating the probability that the recognized object is indeed a person is calculated.

[0041] For example, as shown in captured image 401, consider a case where a part of a person (lower body) is blocked by an obstacle (e.g., the hood 430 of a vehicle). In this case, although the object is recognized as a person (whole body), the reliability indicating the probability that it is indeed a person (whole body) can be calculated as 0.5, for example. In addition, a reliability of 1.0 indicates the highest reliability, and a reliability of 0 indicates the lowest reliability. In addition, as shown in captured image 402, in a case where the whole body of a person is captured without being blocked by an obstacle, the reliability indicating the probability that the object is a person (whole body) can be calculated as 0.95, for example. In addition, various known methods can be used as a method for calculating the reliability.

[0042] In S504, the image transformation unit 204 performs projection transformation of points on a captured image of a certain camera onto a captured image of another camera. For example, the image transformation unit 204 performs projection transformation of the bounding box 410 of the object (person 440) recognized in the captured image 401 of the front camera 104 onto the captured image 402 of the left oblique front camera 106. The bounding box 410 is defined by four points 411, 412, 413, and 414 indicating the positions of the four corners. The bounding box 410 after the projection transformation is projected onto the captured image 402 and becomes a bounding box 410' defined by four points 411', 412', 413', and 414'.

[0043] In S505, the same object determination unit 205 determines whether the same object exists between the plurality of captured images. If it is determined that the same object exists between the captured images, the process proceeds to S506. On the other hand, if it is determined that the same object does not exist between the captured images, the process proceeds to S507.

[0044] For example, the same object determination unit 205 determines whether the object recognized in the image 401 captured by the front camera 104 and the object recognized in the image 402 captured by the left oblique front camera 106 are the same object. Figure 4 As shown, the same object determination unit 205 makes a determination based on the overlap ratio of the feature quantity of the second bounding box 420 surrounding the object recognized in the captured image 402 of the left oblique front camera 106 and the first bounding box 410' after the projective transformation. The second bounding box 420 is a rectangle defined by four points 421, 422, 423, and 424. For example, when the overlap ratio is greater than a threshold value (for example, IoU 0.5), the same object determination unit 205 determines that the same object exists between the captured image 401 and the captured image 402. In this embodiment, it is assumed that the same object exists between the captured image 401 and the captured image 402.

[0045] In S506, the image selection unit 206 selects a captured image for obtaining distance information from a plurality of captured images determined by the same object determination unit 205 to include the same object. For example, the captured image having a higher reliability calculated by the reliability calculation unit 203 is selected. Figure 4 In the example, the reliability calculated for the object in the captured image 401 is 0.5, and the reliability calculated for the object in the captured image 402 is 0.95, so the captured image 402 is selected. Then, the distance acquisition unit 207 acquires the distance information to the object based on the object recognition result of the captured image selected by the image selection unit 206 by the object recognition unit 202. For example, the distance information is acquired based on the table information in which the position on the captured image is associated with the distance in depth from the camera to the position. It is assumed that the table information is maintained for each camera. The content of the table information is not limited to this, and it may also be a table of the distance in depth from the front end and the rear end of the autonomous vehicle 10 to the position.

[0046] exist Figure 4 In the example of , distance information of the table information of the captured image 402 (the oblique left front capturing camera 106 ) corresponding to the position of the straight line connecting the two lower points 422 and 423 of the bounding box 420 of the captured image 402 is acquired.

[0047] In S507, the image selection unit 206 selects a captured image including an object. The distance acquisition unit 207 acquires distance information from the object based on the object recognition result of the object recognition unit 202 on the captured image selected by the image selection unit 206. The method for acquiring the distance information is the same as that in S506, and the distance information can be acquired based on the table information corresponding to the captured image (camera) selected by the image selection unit 206.

[0048] In S508, the vehicle control unit 208 controls the behavior of the autonomous vehicle 10 based on the distance information acquired by the distance acquisition unit 207. For example, when an object is recognized in front of the vehicle, the vehicle control unit 208 automatically performs a braking operation (brake operation) and / or a steering operation to avoid a collision.

[0049] In S509, the information processing device (ECU 101) determines whether to continue processing. If the processing is continued, the process returns to S501. On the other hand, if the processing is not continued, the process ends. Figure 5 For example, the processing may be terminated when the set destination is reached or when the occupant of the autonomous vehicle 10 performs a predetermined operation.

[0050] In addition, the order of the processes in the flowchart is not limited to the order described, and may be another order. In addition, other processes may be added as appropriate, or some processes may be skipped or excluded.

[0051] As described above, in this embodiment, when it is determined that the same object exists between a plurality of captured images, distance information to the object is acquired based on the object recognition result for the captured image with a higher reliability indicating the probability that the object is the relevant object.

[0052] Thus, the distance to the object in the captured image can be acquired with high accuracy based on the captured image adapted to the acquisition of the distance information. For example, in the captured image of the camera in front of the vehicle, when a part of the object is blocked by the hood of the vehicle or other objects, it is possible to suppress the situation where the position of the object is erroneously acquired. Therefore, even in the autonomous control of the vehicle using the acquired distance information, high-precision control can be implemented.

[0053] [Modifications] In the above-mentioned embodiment, an example is described in which a captured image with a higher reliability is selected in S506 and distance information is acquired based on the captured image, but the present invention is not limited to this example. Figure 4 In the example, it is determined that the reliability of the captured image 402 is high, but the object (person) is located closer to the front of the front camera 104 than the left oblique front camera. Therefore, obtaining distance information based on the captured image 401 of the front camera 104 can obtain more accurate distance information.

[0054] Therefore, if Figure 4As shown, the bounding box 420 in the captured image 402 may be projected onto the captured image 401 by the image transformation unit 204. Furthermore, the distance information may be acquired based on the positions of the bottom sides of the four points 421', 422', 423', and 424' of the bounding box 420' after the projected transformation and the table information corresponding to the captured image 401 (the front capturing camera 104).

[0055] In addition, the table information corresponding to which captured image (camera) can be used may be determined in advance for each camera. For example, the priority of the front camera 104 may be set to be greater than the priority of the left oblique front camera 106, or the priority of the front camera 104 may be set to be greater than the priority of the right oblique front camera 108. Similarly, the priority of the rear camera 105 may be set to be greater than the priority of the left oblique rear camera 107, or the priority of the rear camera 105 may be set to be greater than the priority of the right oblique rear camera 109.

[0056] In addition, in S506, it is assumed that the reliability calculated by the reliability calculation unit 203 for each captured image is almost the same, that is, the difference in reliability is below the threshold. In this case, the distance acquisition unit 207 can also obtain the distance information to the object based on the object recognition result of the object recognition unit 202 for the captured image in which the position of the object in the captured image is close to the center position of the captured image. This is because the closer the position is to the center of the captured image, the more accurate the distance information can be obtained. For example, the distance information to the object can also be obtained based on the object recognition result of the object recognition unit 202 for the captured image in which the distance between the center position of the bounding box of the object and the center position of the captured image is shorter.

[0057] In addition, in S507, since the same object is not included in multiple captured images, the table information corresponding to the captured image (camera) is used to obtain the distance information from the object in a captured image, and the accuracy of the distance information may be low. Therefore, in such a case, the content of the vehicle control based on the object recognition result may also be changed. For example, the value of the predetermined distance when performing a braking operation and / or a steering operation and performing an avoidance operation when approaching the object at a predetermined distance may also be changed to a larger value. As a result, the avoidance operation can be performed as early as possible at a position far away from the object, thereby improving safety.

[0058] Alternatively, if the same object is not included in a plurality of captured images, indicating that the reliability of the probability that the object detected in a certain captured image is the object of interest is low and is below a predetermined value, a captured image of a frame preceding the frame of the captured image may be referred to. For example, the movement of the object in the captured image may be tracked, and the position of the bounding box when the entire object (e.g., the entire body of a person) is recognized and the position within the captured image of the current frame may be estimated, and the distance information may be acquired from the table information based on the estimated position.

[0059] In addition, in the above-mentioned embodiment, the processing is described based on whether the same object is reflected between the image captured by the front camera 104 and the image captured by the left oblique front camera 106 (or the right oblique front camera 108), but it is not limited to this example. Similarly, the same processing can be performed based on whether the same object is reflected between the image captured by the rear camera 105 and the image captured by the left oblique rear camera 107 (or the right oblique rear camera 109). In addition, the same processing can be performed based on whether the same object is reflected between the image captured by the left oblique front camera 106 and the image captured by the left oblique rear camera 107. In addition, the same processing can be performed based on whether the same object is reflected between the image captured by the right oblique front camera 108 and the image captured by the right oblique rear camera 109. In addition, the same processing can be performed based on whether the same object is reflected between the images captured by three or more cameras.

[0060] <Summary of implementation methods> 1. The information processing device (101) according to the above embodiment comprises: An image acquisition mechanism (201) that acquires a first captured image (401) by a first camera (104) and a second captured image (402) by a second camera (106); An object recognition mechanism (202) is configured to recognize an object based on the first captured image and the second captured image respectively; A reliability calculation unit (203) for calculating the reliability indicating the probability that the object is a related object for each of the first captured image and the second captured image; A determination unit (205) is configured to determine whether the first captured image and the second captured image contain the same object; and A distance acquisition unit (207) acquires distance information from the object based on the object recognition result of the object recognition unit on the captured image of the party with higher reliability calculated by the reliability calculation unit when the determination unit determines that the same object exists.

[0061] This makes it possible to acquire the distance to an object in a captured image with high accuracy based on the captured image suitable for acquiring distance information.

[0062] 2. Based on the information processing device (101) involved in the above embodiment, The information processing device further comprises a transformation mechanism (204) for projecting a first boundary frame (410) surrounding an object recognized in the first captured image onto the second captured image. The determination unit makes a determination based on an overlap ratio (IoU) between a second bounding box (420) surrounding the object recognized in the second captured image and the first bounding box after the projective transformation.

[0063] This makes it possible to accurately determine the overlap ratio between objects (bounding boxes), and therefore accurately determine whether objects are the same object.

[0064] 3. Based on the information processing device (101) involved in the above embodiment, When the overlap ratio is equal to or greater than a threshold value, the determination unit determines that the same object exists between the first captured image and the second captured image.

[0065] This makes it possible to easily determine whether objects between captured images are the same object.

[0066] 4. Based on the information processing device (101) involved in the above embodiment, The distance acquisition unit acquires the distance information based on a predetermined position of a bounding box in a captured image having higher reliability, out of the first bounding box in the first captured image and the second bounding box in the second captured image.

[0067] This makes it possible to acquire distance information with higher accuracy.

[0068] 5. Based on the information processing device (101) involved in the above embodiment, The predetermined position is a position constituting a lower side of the bounding box.

[0069] In this way, by using the position of the bottom side of the bounding box of the object (eg, person) (the position of the person's heel) as a reference, distance information from the camera to the position (the position of the ground) can be acquired with high accuracy.

[0070] 6. Based on the information processing device (101) involved in the above embodiment, The first camera is a camera (104) disposed in front of an autonomous vehicle (10) having the information processing device and photographing the front of the autonomous vehicle. The second camera is a camera (106, 108) disposed on the left side or the right side of the autonomous vehicle and photographing the left oblique front or the right oblique front of the autonomous vehicle. The information processing device further comprises a second transformation mechanism (204) for projecting a second bounding box (420) surrounding the object recognized in the second captured image onto the first captured image when a second reliability of the object recognized in the second captured image is higher than a first reliability of the object recognized in the first captured image. The distance acquisition mechanism acquires the distance information based on a predetermined position of the second bounding box (420') after the projective transformation.

[0071] This makes it possible to acquire distance information using an image captured by a front-mounted camera that can acquire more accurate distance information from the position of an object in the captured image.

[0072] 7. Based on the information processing device (101) involved in the above embodiment, A part of the object is not recognized in the first captured image, and the entire object is recognized in the second captured image.

[0073] In this way, instead of using a captured image in which a portion of the object is not recognized due to an obstruction or the like, a captured image in which the entire object is recognized is used, thereby enabling highly accurate distance information to be acquired. In addition, when distance information is acquired using a captured image in which a portion of the object is not recognized, for example, when distance information is acquired using table information in which a predetermined position of the object in the captured image (e.g., the heel of a person) is associated with the distance information of the position in the captured image, erroneous distance information may be acquired. However, according to this feature, the acquisition of such erroneous distance information can be suppressed.

[0074] 8. Based on the information processing device (101) involved in the above embodiment, The predetermined position is a position constituting a lower side of the second bounding box after the projective transformation.

[0075] In this way, by using the position of the bottom side of the bounding box of the object (eg, person) (the position of the person's heel) as a reference, distance information from the camera to the position (the position of the ground) can be acquired with high accuracy.

[0076] 9. Based on the information processing device (101) involved in the above embodiment, The first camera is a camera (105) disposed at the rear of an autonomous vehicle (10) equipped with the information processing device and configured to capture images of the rear of the autonomous vehicle. The second camera is a shooting camera (107, 109) disposed on the left side or the right side of the autonomous vehicle and shooting the left oblique rear side or the right oblique rear side of the autonomous vehicle. The information processing device further comprises a second transformation mechanism (204) for projecting a second bounding box surrounding the object recognized in the second captured image onto the first captured image when a second reliability of the object recognized in the second captured image is higher than a first reliability of the object recognized in the first captured image. The distance acquisition unit acquires the distance information based on a predetermined position of the second bounding box after the projective transformation.

[0077] This makes it possible to acquire distance information using an image captured by a rear-view camera that can acquire more accurate distance information from the position of an object in the captured image.

[0078] 10. Based on the information processing device (101) involved in the above embodiment, A part of the object is not recognized in the first captured image, and the entire object is recognized in the second captured image.

[0079] In this way, instead of using a captured image in which a portion of the object is not recognized due to an obstruction or the like, a captured image in which the entire object is recognized is used, thereby enabling highly accurate distance information to be acquired. In addition, when distance information is acquired using a captured image in which a portion of the object is not recognized, for example, when distance information is acquired using table information in which a predetermined position of an object in a captured image (e.g., a person's heel) is associated with distance information at the position in the captured image, erroneous distance information may be acquired. However, according to this feature, the acquisition of such erroneous distance information can be suppressed.

[0080] 11. Based on the information processing device (101) involved in the above embodiment, The predetermined position is a position constituting a lower side of the second bounding box after the projective transformation.

[0081] In this way, by using the position of the bottom edge of the bounding box of the object (e.g., person) (the position of the person's heel) as a reference, the distance information from the camera to the position (the position of the ground) can be acquired with high accuracy. In addition, by using the position of the bottom edge of the bounding box after the projective transformation, even if the lower body of the person is blocked by, for example, the hood in the captured image before the projective transformation, the position of the person's heel can be properly recognized. Therefore, it is possible to acquire high-precision distance information.

[0082] 12. Based on the information processing device (101) involved in the above embodiment, The distance acquisition unit acquires the distance information based on table information in which a position on a captured image is associated with a depth distance from a camera to the position.

[0083] Thus, as long as the expected position of an object in a captured image is known, distance information to the object can be easily acquired.

[0084] 13. Based on the information processing device (101) involved in the above embodiment, When the determination unit determines that the same object does not exist, the distance acquisition unit acquires distance information to each object based on the object recognition result performed by the object recognition unit on each captured image.

[0085] Thereby, even when the same object does not exist, it is possible to acquire distance information from each camera to the object included in each captured image.

[0086] 14. Based on the information processing device (101) involved in the above embodiment, When the reliability difference calculated by the reliability calculation unit is below a threshold, the distance acquisition unit obtains the distance information to the object based on the object recognition result performed by the object recognition unit on the captured image in which the position of the object in the captured image is close to the center position of the captured image.

[0087] In this way, as long as there is basically no difference in reliability (that is, as long as the degree of occlusion of the objects is the same, or as long as all objects are not obstructed), high-precision distance information can be obtained by using the image captured by the side that can capture the object closer to the front of the camera.

[0088] 15. The autonomous vehicle (10) according to the above embodiment is an autonomous vehicle having the information processing device (101) according to the above embodiment, The autonomous vehicle includes a vehicle control unit (208) for controlling the movement of the autonomous vehicle based on the distance information acquired by the distance acquisition unit.

[0089] This makes it possible to perform automatic control of the vehicle based on high-precision object recognition, and to perform automatic control that improves the safety of passengers and / or people outside the vehicle.

[0090] 16. The control method of the information processing device (101) involved in the above embodiment comprises: An image acquisition step (S501), in which a first captured image (401) of the first camera (104) and a second captured image (402) of the second camera (106) are acquired; An object recognition step (S502), in which objects are recognized based on the first captured image and the second captured image respectively; a reliability calculation step (S503), in which the reliability indicating the probability that the object is a related object is calculated for each of the first captured image and the second captured image; a determination step (S505), in which it is determined whether the same object exists between the first captured image and the second captured image; and A distance acquisition step (S506), in which, when it is determined by the determination step that the same object exists, distance information to the object is acquired based on the object recognition result of the object recognition step on the captured image of the party with higher reliability calculated by the reliability calculation step.

[0091] This makes it possible to acquire the distance to an object in a captured image with high accuracy based on the captured image suitable for acquiring distance information.

[0092] 17. The program according to the above embodiment is a program for causing a computer to function as the information processing device according to the above embodiment.

[0093] Thereby, the functions of the information processing device can be realized as a program.

[0094] 18. The storage medium according to the above embodiment is a storage medium storing a program for causing a computer to function as the information processing device according to the above embodiment.

[0095] Thereby, the function of the information processing device can be realized as a storage medium.

[0096] <Other embodiments> In addition, a program that realizes one or more functions described in each embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device can read and execute the program. The present invention can also be realized in this manner.

[0097] The present invention is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the present invention.

Claims

1. An information processing device, characterized in that: The information processing device comprises: An image acquisition mechanism, which acquires a first image captured by the first camera and a second image captured by the second camera; An object recognition mechanism for respectively recognizing an object based on the first captured image and the second captured image; A reliability calculation unit that calculates the reliability indicating the probability that the object is a related object for each of the first captured image and the second captured image; a determination unit configured to determine whether the same object exists between the first captured image and the second captured image; as well as The distance acquisition means acquires the distance information to the object based on the object recognition result of the captured image of the party with higher reliability calculated by the reliability calculation means by the object recognition means when the determination means determines that the same object exists.

2. The information processing device according to claim 1, characterized in that The information processing device further includes a transformation mechanism that projects a first bounding box surrounding the object recognized in the first captured image onto the second captured image. The determination unit makes a determination based on an overlap ratio between a second bounding box surrounding the object recognized in the second captured image and the first bounding box after the projective transformation.

3. The information processing device according to claim 2, characterized in that: When the overlap ratio is equal to or greater than a threshold value, the determination unit determines that the same object exists between the first captured image and the second captured image.

4. The information processing device according to claim 2, characterized in that The distance acquisition unit acquires the distance information based on a predetermined position of a bounding box in a captured image having higher reliability, out of the first bounding box in the first captured image and the second bounding box in the second captured image.

5. The information processing device according to claim 4, characterized in that: The predetermined position is a position constituting a lower side of the bounding box.

6. The information processing device according to claim 1, characterized in that The first camera is a camera that is disposed in front of an autonomous vehicle equipped with the information processing device and that captures a view in front of the autonomous vehicle. The second camera is a camera disposed on the left side or the right side of the autonomous vehicle to take a picture of the left oblique front or the right oblique front of the autonomous vehicle. The information processing device further includes a second transformation mechanism, which, when a second reliability of the object recognized based on the second captured image is higher than a first reliability of the object recognized based on the first captured image, projects a second bounding box surrounding the object recognized based on the second captured image onto the first captured image. The distance acquisition unit acquires the distance information based on a predetermined position of the second bounding box after the projective transformation.

7. The information processing device according to claim 6, characterized in that: A part of the object is not recognized in the first captured image, and the entire object is recognized in the second captured image.

8. The information processing device according to claim 6, characterized in that The predetermined position is a position constituting a lower side of the second bounding box after the projective transformation.

9. The information processing device according to claim 1, characterized in that: The first camera is a camera that is disposed behind the autonomous vehicle having the information processing device and that takes a picture of the rear of the autonomous vehicle. The second camera is a camera disposed on the left side or the right side of the autonomous vehicle to take pictures of the left oblique rear side or the right oblique rear side of the autonomous vehicle. The information processing device further includes a second transformation mechanism, which, when a second reliability of the object recognized based on the second captured image is higher than a first reliability of the object recognized based on the first captured image, projects a second bounding box surrounding the object recognized based on the second captured image onto the first captured image. The distance acquisition unit acquires the distance information based on a predetermined position of the second bounding box after the projective transformation.

10. The information processing device according to claim 9, characterized in that: A part of the object is not recognized in the first captured image, and the entire object is recognized in the second captured image.

11. The information processing device according to claim 9, characterized in that: The predetermined position is a position constituting a lower side of the second bounding box after the projective transformation.

12. The information processing device according to claim 1, characterized in that: The distance acquisition unit acquires the distance information based on table information in which a position on a captured image is associated with a depth distance from a camera to the position.

13. The information processing device according to claim 1, characterized in that: When the determination unit determines that the same object does not exist, the distance acquisition unit acquires distance information to each object based on the object recognition result performed by the object recognition unit on each captured image.

14. The information processing device according to claim 1, characterized in that: When the reliability difference calculated by the reliability calculation unit is below a threshold, the distance acquisition unit obtains the distance information to the object based on the object recognition result performed by the object recognition unit on the captured image in which the position of the object in the captured image is close to the center position of the captured image.

15. An autonomous vehicle, comprising the information processing device according to claim 1, characterized in that: The autonomous vehicle includes a vehicle control unit that controls the behavior of the autonomous vehicle based on the distance information acquired by the distance acquisition unit.

16. A method for controlling an information processing device, characterized in that: The control method of the information processing device comprises: An image acquisition step, in which a first image captured by the first camera and a second image captured by the second camera are acquired; an object recognition step, in which objects are recognized based on the first captured image and the second captured image respectively; a reliability calculation step, in which reliability indicating the probability that the object is a related object is calculated for each of the first captured image and the second captured image; a determination step, in which it is determined whether the same object exists between the first captured image and the second captured image; as well as A distance acquisition step, in which, when it is determined by the determination step that the same object exists, the distance information to the object is acquired based on the object recognition result of the object recognition step on the captured image of the party with higher reliability calculated by the reliability calculation step.

17. A storage medium, characterized in that: The storage medium stores a program for causing a computer to execute the method for controlling the information processing device according to claim 16 .

18. A program product, characterized in that The program product stores a program for causing a computer to execute the method for controlling the information processing device according to claim 16 .

Images