Information processing device, autonomous vehicle, control method for information processing device, program and storage medium
The apparatus uses multiple cameras and confidence-based object recognition to accurately determine distances in images with occlusions, improving autonomous vehicle control.
Patent Information
- Application Number
- JP2023168837
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2026-11-19
- Estimated Expiration
- 2043-09-28
AI Technical Summary
Existing techniques for acquiring distance information using markers in images fail when parts of the marker are hidden due to obstacles, leading to incomplete or incorrect distance measurements.
An information processing apparatus that utilizes multiple cameras to capture images, recognizes objects, calculates confidence levels for object recognition, determines identical objects across images, and selects the image with the highest confidence for accurate distance acquisition based on projective transformations and table information.
Enables precise distance measurement to objects, even when partial occlusion occurs, enhancing the accuracy of autonomous vehicle control operations.
Smart Images

Figure 0007954129000001 
Figure 0007954129000002 
Figure 0007954129000003
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to an information processing apparatus, an autonomous vehicle, a control method for an information processing apparatus, a program, and a storage medium. BACKGROUND ART
[0002] Patent Document 1 discloses that markers included in captured images are stored in association in advance with distance information from an autonomous work machine to the respective markers, and the distance information of a marker detected in a certain captured image is acquired. PRIOR ART DOCUMENTS PATENT DOCUMENTS
[0003] PATENT DOCUMENT 1 International Publication No. 2021 / 014585 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION
[0004] However, with the technique described in Patent Document 1, if part or all of a marker is hidden due to the presence of an obstacle, the marker cannot be sufficiently recognized, and as a result, distance information may not be acquired, or incorrect distance information may be acquired.
[0005] The present invention has been made in view of the above problems, and an object thereof is to provide a technique for accurately acquiring the distance to an object. MEANS FOR SOLVING THE PROBLEMS
[0006] An information processing apparatus according to one aspect of the present invention that achieves the above object comprises: an image acquisition unit configured to acquire a first captured image from a first camera and a second captured image from a second camera; an object recognition unit configured to recognize an object from each of the first captured image and the second captured image; A confidence calculation means that calculates a confidence level indicating the likelihood that the object is the object in question for each of the first and second captured images, A determination means for determining whether the same object exists between the first captured image and the second captured image, If the determination means determines that the same object exists, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image with the higher confidence calculated by the confidence calculation means. Equipped with 、 If the difference in reliability calculated by the reliability calculation means is less than or equal to a threshold, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image in which the position of the object in the captured image is closer to the center position of the captured image. It is characterized by the following. [Effects of the Invention]
[0007] According to the present invention, it becomes possible to accurately obtain the distance to an object. [Brief explanation of the drawing]
[0008] [Figure 1] This is an overhead view of an autonomous vehicle according to one embodiment. [Figure 2] This figure shows an example of the hardware configuration of an autonomous vehicle according to one embodiment. [Figure 3] This figure shows an example of the functional configuration of an information processing unit (ECU) installed in an autonomous vehicle according to one embodiment. [Figure 4] This is an explanatory diagram illustrating the image conversion between captured images according to one embodiment. [Figure 5] This is a flowchart showing the procedure of processing performed by an information processing device according to one embodiment. [Modes for carrying out the invention]
[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the features described in the embodiments may be combined in any way. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.
[0010] <Overhead view of an autonomous vehicle> Figure 1 shows an example of an overhead view of an autonomous vehicle according to this embodiment. In this embodiment, we will explain using as an example a vehicle that performs autonomous driving, recognizing the surrounding environment and controlling the vehicle autonomously based on the recognition results. The autonomous vehicle can take any form as long as it is a mobile body that operates autonomously, such as a tricycle or a four-wheeled vehicle. In this embodiment, it is a four-wheeled vehicle (electric vehicle) that can carry one or more occupants, such as a micromobility.
[0011] As shown in Figure 1, the autonomous vehicle 10 is equipped with a front camera 104, a rear camera 105, a left-front camera 106, a left-rear camera 107, a right-front camera 108, and a right-rear camera 109. Although the illustrated example has six cameras, the number of cameras is not limited to this example and may be more or less than six.
[0012] The front-facing camera 104 is a camera installed at the front of the autonomous vehicle 10 to photograph the area in front of the vehicle. The rear-facing camera 105 is a camera installed at the rear of the autonomous vehicle 10 to photograph the area behind the vehicle. The left-diagonal front-facing camera 106 is a camera installed at the left rear of the autonomous vehicle 10 to photograph the area diagonally to the left front of the vehicle.
[0013] The left diagonally rearward shooting camera 107 is a camera installed on the front left side of the autonomous vehicle 10 for shooting the left diagonally rearward direction of the vehicle. The right diagonally forward shooting camera 108 is a camera installed on the rear right side of the autonomous vehicle 10 for shooting the right diagonally forward direction of the vehicle. The right diagonally rearward shooting camera 109 is a camera installed on the front right side of the autonomous vehicle 10 for shooting the right diagonally rearward direction of the vehicle.
[0014] <Hardware Configuration> Figure 2 is a diagram showing an example of the hardware configuration of the autonomous vehicle according to the present embodiment. The autonomous vehicle 10 includes an ECU 101, a storage device 102, a communication unit 103, and six cameras from the front shooting camera 104 to the right diagonally rearward shooting camera 109 described with reference to Figure 1.
[0015] The ECU 101 is one or more electronic control units, and can function as an information processing apparatus. The control operation of the autonomous vehicle 10 is realized by the ECU 101 reading and executing a computer program stored in the storage device 102. The storage device 102 is one or more memories that store various types of information. For example, it stores information received from other devices, computer programs that are read and executed by the ECU 101, and the like.
[0016] The communication unit 103 has a function of communicating with other devices in a wired or wireless manner via a network (not shown). The functions of the six cameras from the front shooting camera 104 to the right diagonally rearward shooting camera 109 are as described with reference to Figure 1. The ECU 101 controls the operation of the autonomous vehicle 10 based on captured images acquired by each of the cameras from the front shooting camera 104 to the right diagonally rearward shooting camera 109.
[0017] <Functional Configuration> Subsequently, Figure 3 is a diagram showing an example of the functional configuration of the information processing apparatus (ECU 101) included in the autonomous vehicle according to the present embodiment. The information processing apparatus (ECU 101) includes a captured image acquisition unit 201, an object recognition unit 202, a reliability calculation unit 203, an image conversion unit 204, an identical object determination unit 205, an image selection unit 206, a distance acquisition unit 207, and a vehicle control unit 208.
[0018] Captured image acquisition unit 201 acquires one or more captured images each acquired by six cameras ranging from front-facing camera 104 to right-rear-oblique camera 109.
[0019] Object recognition unit 202 recognizes an object from the captured images acquired by captured image acquisition unit 201. For example, a rectangular bounding box enclosing the object is used. Here, the object may include fixed obstacles (such as buildings, curbs, walls, vending machines, traffic lights, etc.) existing around the autonomous vehicle 10, and moving objects (such as persons, animals, bicycles, other vehicles, bicycles, electric boards, etc.), and a person can be taken as an example.
[0020] Confidence calculation unit 203 calculates a confidence level indicating the likelihood that the object recognized by object recognition unit 202 is the target object. For example, when a person (whole body) is recognized as an object, a confidence level indicating the likelihood that the object is truly a person is calculated. For example, consider a case where a part (lower half) of a person is hidden behind an obstacle (e.g., the hood of a vehicle).
[0021] In this case, although the object is recognized as a person (whole body), the confidence level indicating the likelihood that this is truly a person (whole body) can be calculated as, for example, 0.5 (where a confidence level of 1.0 is the highest and a confidence level of 0 is the lowest). Further, when the whole body of the person is captured without being hidden by any obstacle, the confidence level indicating the likelihood that the object is a person (whole body) can be calculated as, for example, 0.95. Note that various known methods can be used as the confidence level calculation method, and detailed descriptions thereof are omitted.
[0022] Image conversion unit 204 performs projective transformation to map a point on a captured image of a certain camera onto a captured image of another camera. For example, image conversion unit 204 performs projective transformation on four points representing the positions of four corners indicating the bounding box of the object recognized on the captured image of front-facing camera 106 onto the captured image of left-front-oblique camera 106. Projective transformation is performed between the captured images of the six cameras, including this projective transformation.
[0023] The identical object detection unit 205 determines whether the same object exists between multiple captured images. For example, the identical object detection unit 205 determines whether an object recognized in the image captured by the front-facing camera 104 is the same object as an object recognized in the image captured by the left-front oblique camera 106. The determination of whether objects between two captured images are the same can be made, for example, as follows.
[0024] First, the image conversion unit 204 projects a first bounding box, which surrounds an object recognized in the image captured by, for example, the front-facing camera 104, onto the image captured by the left-front camera 106. Then, the identical object determination unit 205 makes a determination based on the overlap rate between the second bounding box surrounding the object recognized in the image captured by the left-front camera 106 and the projected first bounding box.
[0025] For example, the identical object detection unit 205 determines that the same object exists between two captured images (in this example, the image captured by the front camera 104 and the left-front oblique camera 106) if the overlap rate is greater than or equal to a threshold (e.g., 50% or IoU 0.5). As the overlap rate, for example, IoU (Intersection over Union) may be used. IoU is an index that represents the extent to which two regions overlap, and can be obtained by dividing the common part of the two regions by the union of the two regions (0 ≤ IoU ≤ 1).
[0026] The image selection unit 206 selects an image from which distance information can be obtained from multiple captured images that have been determined by the identical object determination unit 205 to contain the same object. For example, it selects the image with the higher confidence level calculated by the confidence level calculation unit 203.
[0027] The distance acquisition unit 207 acquires distance information to an object based on the object recognition result of the object recognition unit 202 for the captured image selected by the image selection unit 206. The distance acquisition unit 207 acquires distance information based on table information that associates a position on the captured image with the depth distance from the camera to that position. For example, it uses information on a predetermined position of the bounding box of an object recognized in the captured image to refer to the table information and acquire distance information corresponding to that position.
[0028] The predetermined position of the bounding box of an object may be, for example, the vertical axis (y-axis) coordinate of the position of the bottom edge of the rectangle constituting the bounding box on the captured image. Alternatively, using table information that associates each coordinate of the captured image with distance information, distance information corresponding to the coordinates (x, y) of the midpoint of the bottom edge of the rectangle constituting the bounding box may be obtained by referring to the table information.
[0029] The vehicle control unit 208 controls the operation of the autonomous vehicle 10 based on the distance information acquired by the distance acquisition unit 208. For example, if an object is detected in front of the vehicle, it automatically performs braking and / or steering operations to avoid a collision. If no object that could potentially collide is detected in front of the vehicle, it automatically performs acceleration and / or steering operations to move to the set destination.
[0030] <Processing> Next, the procedure for processing performed by the information processing device (ECU101) according to this embodiment will be explained with reference to the explanatory diagram in Figure 4 and the flowchart in Figure 5. Figure 4 is an explanatory diagram of image conversion between captured images according to one embodiment.
[0031] In S501, the image acquisition unit 201 acquires one or more captured images from each of the six cameras, from the front-facing camera 104 to the right-rear diagonal camera 109.
[0032] In S502, the object recognition unit 202 recognizes an object from the captured image acquired by the captured image acquisition unit 201. In this embodiment, an example of recognizing a person as an object is described. The object recognition unit 202 recognizes a person using a rectangular bounding box. Various methods can be used for object recognition. For example, the features of a person (whole body) can be learned in advance using machine learning, and the recognition process can be performed based on whether the feature quantities within the bounding box of the object recognized by the object recognition unit 202 match, or to what extent, the features of the person (whole body).
[0033] For example, image 401 in Figure 4 was captured by the forward-facing camera 104, and image 402 was captured by the left-diagonal forward-facing camera 106. Images from the other four cameras were also acquired in S501, but here we will use the images from these two cameras for the explanation.
[0034] Image 401 shows a person 440 whose lower body is partially obscured by the hood 430 of the autonomous vehicle 10. On the other hand, image 402 shows a person 450 whose entire body is visible. No objects were recognized in images acquired by other cameras.
[0035] In S503, the confidence calculation unit 203 calculates a confidence score indicating the likelihood that the object recognized from the images captured by each camera by the object recognition unit 202 is indeed that object. In this embodiment, the confidence score indicating the likelihood that the recognized object is actually a person is calculated.
[0036] For example, consider the case shown in captured image 401, where part of a person (lower body) is hidden by an obstacle (e.g., the hood of a vehicle 430). In this case, although the object is recognized as a person (whole body), the confidence level indicating the likelihood that it is indeed a person (whole body) can be calculated as, for example, 0.5. A confidence level of 1.0 represents the highest confidence level, and a confidence level of 0 represents the lowest confidence level. Furthermore, as shown in captured image 402, when the entire body of the person is captured without being hidden by an obstacle, the confidence level indicating the likelihood that the object is a person (whole body) can be calculated as, for example, 0.95. Various known methods can be used to calculate the confidence level.
[0037] In S504, the image conversion unit 204 projects a point on the image captured by one camera onto the image captured by another camera. For example, the image conversion unit 204 projects the bounding box 410 of an object (person 440) recognized on the image 401 of the forward-facing camera 104 onto the image 402 of the left-front camera 106. The bounding box 410 is defined by four points 411, 412, 413, and 414 that indicate the positions of the four corners. The projected bounding box 410 is then projected onto the image 402 to become a bounding box 410' defined by the four points 411', 412', 413', and 414'.
[0038] In S505, the identical object determination unit 205 determines whether the same object exists between multiple captured images. If it is determined that the same object exists between the captured images, the process proceeds to S506. On the other hand, if it is determined that the same object does not exist between the captured images, the process proceeds to S507.
[0039] For example, the identical object determination unit 205 determines whether an object recognized on the image 401 captured by the forward-facing camera 104 and an object recognized on the image 402 captured by the left-front camera 106 are the same object. As shown in Figure 4, the identical object determination unit 205 makes the determination based on the overlap rate of feature quantities between the second bounding box 420 surrounding the object recognized in the image 402 captured by the left-front camera 106 and the projected first bounding box 410'. The second bounding box 420 is a rectangle defined by four points 421, 422, 423, and 424. For example, the identical object determination unit 205 determines that the same object exists between the captured image 401 and the captured image 402 if the overlap rate is greater than or equal to a threshold (e.g., IoU 0.5). In this embodiment, it is assumed that it has been determined that the same object exists between the captured image 401 and the captured image 402.
[0040] In S506, the image selection unit 206 selects an image from which distance information can be obtained from multiple captured images that have been determined by the identical object determination unit 205 to contain the same object. For example, it selects the image with the higher confidence level calculated by the confidence level calculation unit 203. In the example in Figure 4, the confidence level calculated for the object in captured image 401 is 0.5, and the confidence level calculated for the object in captured image 402 is 0.95, so captured image 402 is selected. Then, the distance acquisition unit 207 acquires distance information to the object based on the object recognition result of the object recognition unit 202 for the captured image selected by the image selection unit 206. For example, it acquires distance information based on table information that associates the position on the captured image with the depth distance from the camera to that position. It is assumed that table information is maintained for each camera. The contents of the table information are not limited to this, and may be a table of depth distances from the front or rear end of the autonomous vehicle 10 to that position.
[0041] In the example shown in Figure 4, distance information from the table data of the captured image 402 (left-front oblique camera 106) is obtained, corresponding to the position of the straight line connecting two points 422 and 423 on the lower edge of the bounding box 420 of the captured image 402.
[0042] In S507, the image selection unit 206 selects an image containing an object. The distance acquisition unit 207 acquires distance information to the object based on the object recognition result of the object recognition unit 202 for the image selected by the image selection unit 206. The method for acquiring distance information is the same as in S506, and distance information can be acquired based on table information corresponding to the image selected by the image selection unit 206 (camera).
[0043] In S508, the vehicle control unit 208 controls the operation of the autonomous vehicle 10 based on the distance information acquired by the distance acquisition unit 207. For example, if an object is detected in front of the vehicle, the control unit automatically performs braking and / or steering operations to avoid a collision.
[0044] In S509, the information processing unit (ECU101) determines whether to continue processing. If processing is to be continued, it returns to S501. On the other hand, if processing is not to be continued, the series of processes shown in Figure 5 is terminated. For example, processing may be terminated when the set destination is reached or when a predetermined operation is performed by the occupant of the autonomous vehicle 10.
[0045] Note that the order of processing in the flowchart is not limited to the order described, and other orders are permitted. Furthermore, other processes may be added as appropriate, and some processes may be skipped or omitted.
[0046] As explained above, in this embodiment, if it is determined that the same object exists among multiple captured images, distance information to the object is obtained based on the object recognition result for the captured image with the higher confidence level indicating the likelihood that the object is indeed that object.
[0047] This makes it possible to accurately acquire the distance to an object in a captured image from an image suitable for acquiring distance information. For example, in an image captured by a camera that photographs the front of a vehicle, if part of an object is obscured by the vehicle's hood or other objects, it is possible to suppress the incorrect acquisition of the position of that object. Therefore, it becomes possible to perform highly accurate control in the autonomous control of the vehicle using the acquired distance information.
[0048] [Differentiation] In the above-described embodiment, an example was explained in which the captured image with higher reliability is selected in S506 and distance information is obtained based on that captured image, but the system is not limited to this example. In the example in Figure 4, captured image 402 was determined to have higher reliability, but the object (person) is located closer to the front of the forward-facing camera 104 than the left-diagonal-front-facing camera. Therefore, obtaining distance information based on the captured image 401 from the forward-facing camera 104 can yield more accurate distance information.
[0049] Therefore, as shown in Figure 4, the bounding box 420 in the captured image 402 may be projected onto the captured image 401 by the image conversion unit 204. Distance information may then be obtained from the positions of the lower edges of the projected bounding box 420' (four points 421', 422', 423', 424') and table information corresponding to the captured image 401 (forward-facing camera 104).
[0050] Furthermore, a priority may be predetermined for each camera regarding which captured image (camera) corresponds to which table information can be used. For example, the priority of the front-facing camera 104 may be set to > the priority of the left-diagonal front-facing camera 106, or the priority of the front-facing camera 104 may be set to > the priority of the right-diagonal front-facing camera 108. Similarly, the priority of the rear-facing camera 105 may be set to > the priority of the left-diagonal rear-facing camera 107, or the priority of the rear-facing camera 105 may be set to > the right-diagonal rear-facing camera 109.
[0051] Furthermore, in S506, it is also conceivable that the confidence levels calculated by the confidence level calculation unit 203 for each captured image are approximately the same, that is, the difference in confidence levels is below a threshold. In that case, the distance acquisition unit 207 may acquire distance information to the object based on the object recognition result of the object recognition unit 202 for the captured image in which the position of the object in the captured image is closer to the center position of the captured image. This is because the closer the position is to the center of the captured image, the more accurate the distance information can be obtained. For example, distance information to the object may be acquired based on the object recognition result of the object recognition unit 202 for the captured image in which the distance between the center of gravity of the object's bounding box and the center position of the captured image is shorter.
[0052] Furthermore, in S507, since the same object is not included in multiple captured images, distance information to an object in a given captured image is obtained using table information corresponding to that captured image (camera), which may result in low accuracy of distance information. Therefore, in such cases, the content of vehicle control based on the object recognition result may be changed. For example, the predetermined distance value used when performing braking and / or steering operations to execute an avoidance maneuver when the vehicle is within a predetermined distance of an object may be changed to a larger value. This allows for earlier avoidance operations at a distance from the object, thereby improving safety.
[0053] Furthermore, if the same object is not included in multiple captured images, and the confidence level indicating the likelihood that an object detected in a given image is that object is low and below a predetermined value, then captured images from frames prior to the frame of that image may be referenced. For example, the movement of an object in a captured image may be tracked, the position of the bounding box in the current frame's captured image when the entire object (e.g., a whole person) was recognized may be estimated, and distance information may be obtained from table information based on the estimated position.
[0054] Furthermore, while the above-described embodiment explained processing based on whether the same object is captured between the image captured by the front-facing camera 104 and the image captured by the left-diagonal front-facing camera 106 (or the right-diagonal front-facing camera 108), the process is not limited to this example. Similarly, the same processing can be performed depending on whether the same object is captured between the image captured by the rear-facing camera 105 and the image captured by the left-diagonal rear-facing camera 107 (or the right-diagonal rear-facing camera 109). Moreover, the same processing can be performed depending on whether the same object is captured between the image captured by the left-diagonal front-facing camera 106 and the image captured by the left-diagonal rear-facing camera 107. And the same processing can be performed depending on whether the same object is captured between the image captured by the right-diagonal front-facing camera 108 and the image captured by the right-diagonal rear-facing camera 109. Furthermore, the same processing can be performed depending on whether the same object is captured between the images captured by three or more cameras.
[0055] <Summary of Embodiments> 1. The information processing device (101) according to the above embodiment is: Image acquisition means (201) that acquires a first captured image (401) from a first camera (104) and a second captured image (402) from a second camera (106), An object recognition means (202) that recognizes an object from the first captured image and the second captured image, respectively, A confidence calculation means (203) that calculates a confidence level indicating the likelihood that the object is the object for each of the first and second captured images, A determination means (205) for determining whether the same object exists between the first captured image and the second captured image, If the determination means determines that the same object exists, the distance acquisition means (207) acquires distance information to the object based on the object recognition result of the object recognition means for the captured image with the higher confidence calculated by the confidence calculation means, It is equipped with.
[0056] This makes it possible to accurately obtain the distance to an object in a captured image from an image suitable for acquiring distance information.
[0057] 2. In the information processing device (101) according to the above embodiment, The system further includes a transformation means (204) for projecting a first bounding box (410) surrounding the object recognized in the first captured image onto the second captured image. The determination means makes a determination based on the overlap rate (IoU) between the second bounding box (420) surrounding the object recognized in the second captured image and the first bounding box that has been projected.
[0058] This allows for accurate determination of the overlap rate between objects (between bounding boxes), making it possible to accurately determine whether objects are identical.
[0059] 3. In the information processing device (101) according to the above embodiment, The determination means determines that the same object exists between the first captured image and the second captured image if the overlap rate is equal to or greater than a threshold.
[0060] This makes it easy to determine whether objects in different captured images are the same object.
[0061] 4. In the information processing device (101) according to the above embodiment, The distance acquisition means acquires the distance information based on a predetermined position of the bounding box in the image with higher reliability, which of the first bounding box in the first captured image and the second bounding box in the second captured image.
[0062] This makes it possible to obtain more accurate distance information.
[0063] 5. In the information processing device (101) according to the above embodiment, The predetermined position is the position of the lower edge that constitutes the bounding box.
[0064] In this way, by using the position of the bottom edge of the bounding box of an object (for example, a person) as a reference (the position of the person's feet), distance information from the camera to that position (the position of the ground) can be obtained with high accuracy.
[0065] 6. In the information processing device (101) according to the above embodiment, The first camera is positioned in front of the autonomous vehicle (10) equipped with the information processing device and is a camera (104) that photographs the area in front of the autonomous vehicle. The second camera is a camera (106, 108) positioned to the left or right of the autonomous vehicle, which captures images of the area diagonally to the left front or diagonally to the right front of the autonomous vehicle. If the second confidence level of an object recognized from the second captured image is higher than the first confidence level of an object recognized from the first captured image, the system further includes a second transformation means (204) that projects a second bounding box (420) surrounding the object recognized in the second captured image onto the first captured image. The distance acquisition means acquires the distance information based on a predetermined position of the projected second bounding box (420').
[0066] This makes it possible to acquire distance information using images captured by a forward-facing camera, which can obtain more accurate distance information from the position of objects in the captured image.
[0067] 7. In the information processing device (101) according to the above embodiment, In the first captured image, part of the object is not recognized, while in the second captured image, the entire object is recognized.
[0068] Thus, by using images in which the entire object is recognized, rather than images in which part of the object is not recognized due to occlusions, it becomes possible to obtain accurate distance information. Furthermore, when obtaining distance information using images in which part of the object is not recognized, there is a possibility of obtaining incorrect distance information when using table information that associates a predetermined position of the object in the image (for example, the feet of a person) with the distance information at that position in the image. However, this feature can suppress the acquisition of such incorrect distance information.
[0069] 8. In the information processing device (101) according to the above embodiment, The predetermined position is the position of the lower edge of the second bounding box that has been projected.
[0070] In this way, by using the position of the bottom edge of the bounding box of an object (for example, a person) as a reference (the position of the person's feet), distance information from the camera to that position (the position of the ground) can be obtained with high accuracy.
[0071] 9. In the information processing device (101) according to the above embodiment, The first camera is positioned behind the autonomous vehicle (10) equipped with the information processing device and is a camera (105) that photographs the area behind the autonomous vehicle. The second camera is a camera (107, 109) positioned to the left or right of the autonomous vehicle, which captures images of the left rear or right rear of the autonomous vehicle. If the second confidence level of an object recognized from the second captured image is higher than the first confidence level of an object recognized from the first captured image, the system further includes a second transformation means (204) that projects the second bounding box surrounding the object recognized in the second captured image onto the first captured image. The distance acquisition means acquires the distance information based on a predetermined position of the second bounding box that has been projected.
[0072] This makes it possible to acquire distance information using images captured by a rear-facing camera, which can obtain more accurate distance information from the position of objects in the captured images.
[0073] 10. In the information processing device (101) according to the above embodiment, In the first captured image, part of the object is not recognized, while in the second captured image, the entire object is recognized.
[0074] Thus, by using images in which the entire object is recognized, rather than images in which part of the object is not recognized due to occlusions, it becomes possible to obtain accurate distance information. Furthermore, when obtaining distance information using images in which part of the object is not recognized, there is a possibility of obtaining incorrect distance information when using table information that associates a predetermined position of the object in the image (for example, the feet of a person) with the distance information at that position in the image. However, this feature can suppress the acquisition of such incorrect distance information.
[0075] 11. In the information processing device (101) according to the above embodiment, The predetermined position is the position of the lower edge of the second bounding box that has been projected.
[0076] In this way, by using the position of the bottom edge of the bounding box of an object (for example, a person) as a reference (the position of the person's feet), distance information from the camera to that position (the position of the ground) can be obtained with high accuracy. Furthermore, by using the position of the bottom edge of the projected bounding box, even if the lower half of the person is hidden by, for example, a car hood in the projected image, the position of the person's feet can be appropriately recognized. Therefore, it becomes possible to obtain highly accurate distance information.
[0077] 12. In the information processing device (101) according to the above embodiment, The distance acquisition means acquires distance information based on table information that associates a position on the captured image with the depth distance from the camera to that position.
[0078] This makes it easy to obtain distance information to an object, provided that the predetermined position of that object in the captured image is known.
[0079] 13. In the information processing device (101) according to the above embodiment, If the determination means determines that no identical objects exist, the distance acquisition means acquires distance information to each object based on the object recognition result of the object recognition means for each captured image.
[0080] This makes it possible to obtain distance information from each camera to objects included in each captured image, even if the same object does not exist in the same image.
[0081] 14. In the information processing device (101) according to the above embodiment, The information processing apparatus according to any one of claims 1 to 9, characterized in that, if the difference in reliability calculated by the reliability calculation means is less than or equal to a threshold, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image in which the position of the object in the captured image is closer to the center position of the captured image.
[0082] Thus, if there is little difference in reliability (i.e., if the degree of object occlusion is the same, or if neither object is occluded), more accurate distance information can be obtained by using the image taken from the camera that captures the object more directly in front of the camera.
[0083] 15. The autonomous vehicle (10) according to the above embodiment is An autonomous vehicle equipped with the information processing device (101) of the above embodiment, The system includes a vehicle control means (208) that controls the operation of the autonomous vehicle based on distance information acquired by the distance acquisition means.
[0084] This enables automated vehicle control based on high-precision object recognition, improving the safety of occupants and / or people outside the vehicle.
[0085] 16. The control method for the information processing device (101) according to the above embodiment is: Image acquisition process (S501) to acquire a first image (401) from a first camera (104) and a second image (402) from a second camera (106), An object recognition step (S502) in which an object is recognized from the first captured image and the second captured image, respectively, A confidence calculation step (S503) is performed to calculate a confidence level indicating the likelihood that the object is the object in question for each of the first and second captured images, A determination step (S505) to determine whether the same object exists between the first captured image and the second captured image, If the determination step determines that the same object exists, a distance acquisition step (S506) is performed to acquire distance information to the object based on the object recognition result of the object recognition step for the captured image with the higher confidence calculated in the confidence calculation step, It has.
[0086] This makes it possible to accurately obtain the distance to an object in a captured image from an image suitable for acquiring distance information.
[0087] 17. The program according to the above embodiment is: This is a program for causing a computer to function as an information processing device according to the above embodiment.
[0088] This makes it possible to implement the functions of an information processing device as a program.
[0089] 18. The storage medium according to the above embodiment is This is a storage medium on which a program is stored that causes the computer to function as an information processing device according to the above embodiment.
[0090] This makes it possible to implement the functions of an information processing device using a storage medium.
[0091] <Other Embodiments> Furthermore, a program that implements one or more functions described in each embodiment is supplied to a system or device via a network or storage medium, and one or more processors in the computer of the system or device can read and execute this program. The present invention can also be realized in this manner.
[0092] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. [Explanation of symbols]
[0093] 10: Autonomous vehicle, 101: ECU, 104: Forward-facing camera, 106: Left-side forward-facing camera, 201: Image acquisition unit, 202: Object recognition unit, 203: Confidence calculation unit, 204: Image conversion unit, 205: Identical object determination unit, 206: Image selection unit, 207: Distance acquisition unit, 208: Vehicle control unit
Claims
1. An information processing device, Image acquisition means for acquiring a first image captured by a first camera and a second image captured by a second camera, Object recognition means for recognizing objects from the first captured image and the second captured image, respectively, A confidence calculation means for calculating a confidence level indicating the likelihood that the object is the object in question for each of the first and second captured images, A determination means for determining whether the same object exists between the first captured image and the second captured image, If the determination means determines that the same object exists, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image with the higher confidence calculated by the confidence calculation means. Equipped with, The distance acquisition means is characterized in that, when the difference in reliability calculated by the reliability calculation means is less than or equal to a threshold, it acquires distance information to the object based on the object recognition result of the object recognition means for the captured image in which the position of the object in the captured image is closer to the center position of the captured image.
2. The system further includes a transformation means for projecting a first bounding box surrounding an object recognized in the first captured image onto the second captured image, The information processing apparatus according to claim 1, characterized in that the determination means performs a determination based on the overlap rate between the second bounding box surrounding the object recognized in the second captured image and the first bounding box that has been projected.
3. The information processing apparatus according to claim 2, characterized in that the determination means determines that the same object exists between the first captured image and the second captured image when the overlap rate is equal to or greater than a threshold.
4. The information processing apparatus according to claim 2, characterized in that the distance acquisition means acquires distance information based on a predetermined position of the bounding box in the shooting image with higher reliability, which of the first bounding box in the first shooting image and the second bounding box in the second shooting image.
5. The information processing apparatus according to claim 4, characterized in that the predetermined position is the position of the lower edge constituting the bounding box.
6. An information processing device, Image acquisition means for acquiring a first image captured by a first camera and a second image captured by a second camera, Object recognition means for recognizing objects from the first captured image and the second captured image, respectively, A confidence calculation means for calculating a confidence level indicating the likelihood that the object is the object in question for each of the first and second captured images, A determination means for determining whether the same object exists between the first captured image and the second captured image, If the determination means determines that the same object exists, the distance acquisition means acquires distance information to the object based on the object recognition result of the object recognition means for the captured image with the higher confidence calculated by the confidence calculation means. Equipped with, The first camera is positioned in front of the autonomous vehicle equipped with the information processing device and is a camera that photographs the area in front of the autonomous vehicle. The second camera is positioned to the left or right of the autonomous vehicle and is a camera that captures images of the area diagonally to the left front or diagonally to the right front of the autonomous vehicle. In the first captured image, part of the object is not recognized, while in the second captured image, the entire object is recognized. If the second confidence level of an object recognized from the second captured image is higher than the first confidence level of an object recognized from the first captured image, the system further includes a transformation means for projecting the second bounding box surrounding the object recognized in the second captured image onto the first captured image. The distance acquisition means is characterized by acquiring the distance information based on the position of the lower edge constituting the second bounding box that has been projected.
7. The first camera is positioned at the rear of the autonomous vehicle equipped with the information processing device and is a camera that photographs the area behind the autonomous vehicle. The second camera is positioned to the left or right of the autonomous vehicle and is a camera that captures images of the left rear or right rear of the autonomous vehicle. If the second confidence level of an object recognized from the second captured image is higher than the first confidence level of an object recognized from the first captured image, the system further includes a second transformation means for projecting the second bounding box surrounding the object recognized in the second captured image onto the first captured image. The information processing apparatus according to claim 1, characterized in that the distance acquisition means acquires the distance information based on a predetermined position of the second bounding box that has been projected.
8. The information processing device according to claim 7, characterized in that in the first captured image, a part of the object is not recognized, and in the second captured image, the entire object is recognized.
9. The information processing apparatus according to claim 7, characterized in that the predetermined position is the position of the lower edge constituting the second bounding box that has been projected.
10. The information processing apparatus according to claim 1, characterized in that the distance acquisition means acquires distance information based on table information that associates a position on a captured image with the depth distance from the camera to that position.
11. The information processing apparatus according to claim 1, characterized in that, when the determination means determines that no identical objects exist, the distance acquisition means acquires distance information to each object based on the object recognition result of the object recognition means for each captured image.
12. An autonomous vehicle equipped with an information processing device according to any one of claims 1 to 11, An autonomous vehicle characterized by comprising a vehicle control means for controlling the operation of the autonomous vehicle based on distance information acquired by the distance acquisition means.
13. A method for controlling an information processing device, An image acquisition process that acquires a first image captured by a first camera and a second image captured by a second camera, An object recognition step in which an object is recognized from the first captured image and the second captured image, respectively, A confidence calculation step which calculates a confidence level indicating the likelihood that the object is the object for each of the first and second captured images, A determination step of determining whether the same object exists between the first captured image and the second captured image, If the determination step determines that the same object exists, a distance acquisition step is performed to acquire distance information to the object based on the object recognition result of the object recognition step for the captured image with the higher confidence calculated in the confidence calculation step, It has, A control method for an information processing device, characterized in that, in the distance acquisition step, if the difference in reliability calculated in the reliability calculation step is less than or equal to a threshold, distance information to the object is acquired based on the object recognition result in the object recognition step for the captured image in which the position of the object in the captured image is closer to the center position of the captured image.
14. A control method for an information processing device, An image acquisition process that acquires a first image captured by a first camera and a second image captured by a second camera, An object recognition step in which an object is recognized from the first captured image and the second captured image, respectively, A confidence calculation step which calculates a confidence level indicating the likelihood that the object is the object for each of the first and second captured images, A determination step of determining whether the same object exists between the first captured image and the second captured image, If the determination step determines that the same object exists, a distance acquisition step is performed to acquire distance information to the object based on the object recognition result of the object recognition step for the captured image with the higher confidence calculated in the confidence calculation step, It has, The first camera is positioned in front of the autonomous vehicle equipped with the information processing device and is a camera that photographs the area in front of the autonomous vehicle. The second camera is positioned to the left or right of the autonomous vehicle and is a camera that captures images of the area diagonally to the left front or diagonally to the right front of the autonomous vehicle. In the first captured image, part of the object is not recognized, while in the second captured image, the entire object is recognized. If the second confidence level of an object recognized from the second captured image is higher than the first confidence level of an object recognized from the first captured image, the system further includes a transformation step of projecting the second bounding box surrounding the object recognized in the second captured image onto the first captured image. A control method for an information processing device, characterized in that the distance acquisition step acquires the distance information based on the position of the lower edge constituting the second bounding box that has been projected.
15. A program for causing a computer to function as an information processing device according to any one of claims 1 to 11.
16. A storage medium storing a program for causing a computer to function as an information processing device according to any one of claims 1 to 11.
Citation Information
Patent Citations
Object tracking method and device, electronic equipment and readable storage medium
CN111932590A
Method of measuring distance, instrument therefor, program therefor, recording medium therefor, and distance measuring instrument mounting type robot apparatus
JP2003269917A
Image processing method, image processing system, and computer program
JP2013012045A
Three-dimensional object recognition device
JP2016024160A
Obstacle detection device
JP2017087758A
