Information processing apparatus, information processing method, and program

Abstract

The present invention relates to an information processing apparatus, an information processing method, and a program, by which a camera exposure can be appropriately controlled using captured images from a plurality of cameras. A first camera imaging range at a second time is predicted at a time before the second time after a first time. An image of an overlapping region is extracted from a captured image at the first time from a second camera, in which the predicted first camera imaging range at the second time overlaps with the second camera imaging range at the first time. An exposure of the first camera at the second time is controlled based on the extracted overlapping region image.

Description

Technical Field

[0001] This technology relates to information processing apparatus, information processing methods and programs, and more specifically to information processing apparatus, information processing methods and programs capable of appropriately controlling the exposure of cameras using captured images from multiple cameras. Background Technology

[0002] Patent document 1 discloses performing exposure control in an in-vehicle camera to make the brightness of the image of the recognition area appropriate.

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-134845 Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] When the position or orientation of the moving object of the camera changes rapidly and the brightness of the captured image changes rapidly, the exposure cannot keep up with the changes in brightness, and overexposure or underexposure may occur in the captured image.

[0008] This technology was developed in view of such circumstances and enables the proper control of camera exposure using images captured by multiple cameras.

[0009] Solution to the problem

[0010] The information processing apparatus according to the present technology is an information processing apparatus including a processing unit or a program for making a computer act as such an information processing apparatus, wherein the processing unit predicts the imaging range of a first camera at a second time after a first time and before a second time, extracts an image of an overlapping region from an image captured by a second camera at the first time, and controls the exposure of the first camera at the second time based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time and the imaging range of the second camera at the first time overlap.

[0011] The information processing method according to the present technology is an information processing method for an information processing apparatus including a processing unit. The information processing method includes, by means of the processing unit, predicting the imaging range of a first camera at a second time after a first time and before a second time, extracting an image of an overlapping region from a captured image of a second camera at the first time, and controlling the exposure of the first camera at the second time based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time and the imaging range of the second camera at the first time overlap.

[0012] In this technology, the imaging range of the first camera at the second time is predicted at a time after the first time and before the second time. An image of the overlapping region is extracted from the image captured by the second camera at the first time. The exposure of the first camera at the second time is controlled based on the extracted image of the overlapping region, in which the predicted imaging range of the first camera at the second time overlaps with the imaging range of the second camera at the first time. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an example configuration of a self-position estimation device applying the present technology.

[0014] Figure 2 This is a diagram showing a car as a moving body on which a self-position estimation device is mounted.

[0015] Figure 3 This is a diagram showing a drone as a moving body equipped with its own position estimation device.

[0016] Figure 4 This is a diagram describing the SLAM process.

[0017] Figure 5 This is a diagram describing normal exposure control.

[0018] Figure 6 This is a diagram illustrating a problem with normal exposure control.

[0019] Figure 7 It is shown Figure 1 A block diagram illustrating a configuration example of the first embodiment of the exposure processing unit.

[0020] Figure 8 This is a diagram describing the first calculation method for overlapping regions.

[0021] Figure 9 This is a diagram describing the first calculation method for overlapping regions.

[0022] Figure 10 This is a diagram describing the second calculation method for overlapping regions.

[0023] Figure 11 This is a graph showing the inference results of instance segmentation.

[0024] Figure 12 This is a diagram describing exposure control based on predicted brightness.

[0025] Figure 13 This is a flowchart illustrating the process of exposure control performed by the self-position estimation device.

[0026] Figure 14 It is shown Figure 1 A block diagram illustrating a configuration example of the second embodiment of the exposure processing unit.

[0027] Figure 15 This is a block diagram illustrating an example configuration of computer hardware that performs a series of processes through a program. Detailed Implementation

[0028] In the following description, embodiments of the present technology will be described with reference to the accompanying drawings.

[0029] <Implementation Method of Self-Position Estimation Device Using This Technology>

[0030] Figure 1 This is a block diagram illustrating an example configuration of a self-position estimation device applying this technology. Note that... Figure 1 The configuration related to the exposure control of the camera (imaging device) is shown.

[0031] Figure 1 The self-position estimation device 11 is mounted on a mobile body such as a drone and uses visual simultaneous localization and mapping (SLAM) technology to estimate its own position.

[0032] The self-position estimation device 11 includes camera A (first camera), camera B (second camera), self-position estimation unit 31 and exposure processing unit 32.

[0033] Cameras A and B represent two cameras with different imaging directions among a plurality of cameras held by a moving body. Each of cameras A and B provides its own position estimation unit 31 with an image obtained through capture (captured image). In this embodiment, the captured image is a moving image, but it can also be a still image. Cameras A and B are not limited to cameras used for self-position estimation, but can be cameras used for other applications.

[0034] Notice, Figure 1This diagram primarily focuses on the exposure control of camera A. Any one of the multiple cameras included in the moving body is camera A, and any other camera among the multiple cameras included in the moving body is camera B. Therefore, each of the multiple cameras included in the moving body can correspond to both camera A and camera B. On the other hand, as described later, for the exposure control of camera A, normal exposure control and exposure control based on predicted brightness are selectively performed. For the exposure control of camera B, only normal exposure control is performed. Initially, there is no difference in exposure control between camera A and camera B, and for the exposure control of camera B, similar to camera A, normal exposure control and exposure control based on predicted brightness are selectively performed. However, for the sake of simplicity, for the exposure control of camera B, it is assumed that only normal exposure control is performed, without assuming the case of exposure control based on predicted brightness being performed.

[0035] Figure 2 This is a diagram showing a car that is a moving body on which a self-position estimation device 11 is mounted.

[0036] exist Figure 2 The front, rear, and sides of the vehicle 71 are equipped with a front camera 72 for capturing images of the front of the vehicle, a rear camera 73 for capturing images of the rear of the vehicle, a left camera 74 for capturing images of the left side of the vehicle, and a right camera 75 for capturing images of the right side of the vehicle. The captured images by the front camera 72, rear camera 73, left camera 74, and right camera 75 can be used for visual SLAM processing in the self-position estimation device 11. Figure 1 Cameras A and B in the diagram represent any two of the following cameras: front camera 72, rear camera 73, left camera 74, and right camera 75.

[0037] Figure 3 This is a diagram showing a drone as a moving body on which a self-position estimation device 11 is mounted.

[0038] exist Figure 3 The drone 91 is equipped with a camera 92 for capturing images from the front and a camera 93 for capturing images from below. The images captured by these cameras 92 and 93 can be used for visual SLAM processing in the self-position estimation device 11. Figure 1 Cameras A and B in the diagram represent cameras 92 and 93.

[0039] exist Figure 1 In the above, camera A includes an image generation unit 51A and an exposure control unit 52A, and camera B includes an image generation unit 51B and an exposure control unit 52B.

[0040] Image generation units 51A and 51B perform photoelectric conversion on the image formed on the imaging surface of the imaging element by the optical system through the imaging element. Image generation units 51A and 51B perform predetermined processing such as analog-to-digital (AD) conversion on the image (image signal) converted by the imaging element through the imaging element, and then provide the processed image to their own position estimation unit 31.

[0041] Exposure control units 52A and 52B control the exposure of cameras A and B based on brightness information (described later) from exposure processing unit 32. Specifically, controlling exposure means controlling each exposure control target in image generation units 51A and 51B to achieve a target exposure value (EV value). The exposure control targets to be controlled include any one or more of the following: exposure time (shutter speed) in the imaging element, aperture (aperture value) in the optical system, and gain (corresponding to ISO sensitivity) in the processing unit for amplifying the image signal.

[0042] Exposure control unit 52A selectively performs normal exposure control and exposure control based on predicted brightness as exposure control for camera A (exposure control). Exposure control unit 52B only performs normal exposure control as exposure control for camera B. Although details of exposure control will be described later, in the normal exposure control of cameras A and B, exposure is controlled based on the brightness (luminance) of their own captured images.

[0043] In the exposure control of camera A by predicting brightness, the imaging range of camera A at the second time is predicted at a time interval between the first and second time intervals. An image of an overlapping region is extracted from the captured image of camera B at the first time interval, where the predicted imaging range of camera A at the second time interval overlaps with the imaging range of camera B at the first time interval. The exposure of camera A at the second time interval is controlled based on the brightness of the extracted image of the overlapping region. In other words, in the exposure control by predicting brightness, the future position of camera A is predicted, and the brightness of the captured image of camera A at the predicted position is predicted based on the captured image of camera B at the current or past position. The exposure of camera A is controlled based on the predicted brightness of the captured image of camera A at the predicted position, so that the captured image of camera A at the predicted position has appropriate brightness.

[0044] Based on exposure control through brightness prediction, the exposure of camera A is appropriately controlled using the captured image from camera B. That is, when the imaging range (imaging direction) of camera A changes rapidly (high-speed change) with the position or orientation of camera A, for example, the imaging range of camera A can quickly change from sunlight to shadow. In such cases, when camera B captures an image within the imaging range of camera A after this change, the exposure of camera A can be immediately adjusted to an appropriate exposure based on the captured image of camera B at that time. Therefore, overexposure and underexposure in the captured image of camera A are suppressed. In particular, overexposure and underexposure in the captured image used for self-position estimation (visual SLAM) are suppressed, thereby preventing unexpected situations where feature point detection becomes difficult and self-position estimation fails.

[0045] The self-position estimation unit 31 estimates its own position at the current time (now) based on the captured images from the image generation units 51A and 51B of cameras A and B, through visual SLAM processing (hereinafter simply referred to as SLAM processing). The self-position includes the position and orientation (tilt) of camera A, camera B, and the position and orientation (tilt) of the moving object.

[0046] The position and orientation of camera A represent its position (coordinates) and orientation (tilt) within a camera coordinate system fixed to camera A in an absolute coordinate system (world coordinate system) fixed to real space (map space generated by SLAM processing). The position and orientation of camera B represent its position (coordinates) and orientation (tilt) within a camera coordinate system fixed to camera B in an absolute coordinate system. The position and orientation of a moving body represent its position (coordinates) and orientation (tilt) within a machine coordinate system fixed to the moving body in an absolute coordinate system. Note that in the following text, position and orientation (tilt) will be referred to simply as position.

[0047] Since the positions of camera A, camera B, and the moving body have predetermined positional relationships with each other, the self-position estimation unit 31 can estimate the position of any one or two of camera A, camera B, and the moving body through SLAM processing, and derive (estimate) other positions based on the positional relationships with the estimated positions. If necessary, other positions can be derived by another processing unit, such as the exposure processing unit 32, instead of the self-position estimation unit 31.

[0048] The self-position estimation unit 31 provides the estimated self-position to the exposure processing unit 32. Note that the self-position estimated by the self-position estimation unit 31 is also provided to a processing unit (not shown) other than the exposure processing unit 32 for arbitrary processing.

[0049] The SLAM processing of the self-position estimation unit 31 will be described. SLAM processing is well known, and will be briefly described, focusing only on the case where the self-position estimation unit 31 estimates the position of camera A at the current time t (the current position) as its own position. The current position of camera B can also be estimated similarly to the current position of camera A.

[0050] Figure 4 This is a diagram describing the SLAM process.

[0051] Figure 4 The position of camera A relative to tree 111, which is an example of the subject, is shown in two dimensions at time t1 and at time t2 after time t1.

[0052] Capture images 112-1 and 112-2 represent the captured images taken by camera A at positions t1 and t2, respectively.

[0053] Upon startup, the self-position estimation unit 31 sets the initial position of camera A (the position and orientation of camera A's camera coordinate system in the absolute coordinate system) through a predetermined initial setting process. It is assumed that the position of camera A at time t1 is estimated by repeating the initial position of camera A and subsequent processing by the self-position estimation unit 31 after the initial setting process. Time t2 represents the time at the newly estimated position of camera A (the current time t).

[0054] Whenever a captured image of each frame is provided from camera A, the self-position estimation unit 31 detects feature points such as those present in the captured image.

[0055] exist Figure 4 In the example, feature points 114-1, 115-1, and 116-1 are detected from the captured image 112-1 at time t1, and feature points 114-2, 115-2, and 116-2 are detected from the captured image 112-2 at time t2. Feature points 114-1 and 114-2 are images of object point 114 belonging to tree 111, which is one of the subjects. Feature points 115-1 and 115-2 are images of object point 115 belonging to tree 111. Feature points 116-1 and 116-2 are images of object point 116 belonging to tree 111.

[0056] The self-position estimation unit 31 extracts the following feature points from the feature points detected in the captured image 112-2 at time t2: these feature points are images of the same object points relative to the feature points detected in the captured image 112-1 at time t1. Figure 4 In the example, feature points 114-2, 115-2, and 116-2 detected from the captured image 112-2 at time t2 are extracted as feature points, which are images of the same object points 114, 115, and 116 relative to feature points 114-1, 115-1, and 116-1 detected from the captured image 112-1 at time t1.

[0057] The self-position estimation unit 31 estimates the change in position (change in position and orientation) of camera A during the time interval from time t1 to time t2 based on the three-dimensional positions 114, 115, and 116 of feature points 114-1, 115-1, and 116-1 extracted from the captured image 112-1 at time t1, and the positions of feature points 114-2, 115-2, and 116-2 extracted from the captured image 112-2 at time t2. The self-position estimation unit 31 then estimates the position (position and orientation in the absolute coordinate system) of camera A at time t2 based on the estimated change in position and the position of camera A at time t1. In this way, the self-position estimation unit 31 estimates the position (current position) of camera A at time t2 (current time t) by tracking the positions of feature points on the captured images.

[0058] exist Figure 1 In this process, the exposure processing unit 32 generates brightness information for exposure control of each of cameras A and B based on captured images from cameras A and B, its own position from its own position estimation unit 31, and so on, and provides this brightness information to the exposure control units 52A and 52B of cameras A and B. The configuration and processing of the exposure processing unit 32 will be described later.

[0059] Normal exposure control performed by exposure control units 52A and 52B of cameras A and B will be described. Both exposure control units 52A and 52B perform normal exposure control in a similar manner, and therefore only the normal exposure control of camera A performed by exposure control unit 52A will be described.

[0060] In normal exposure control, the exposure processing unit 32 provides luminance information to the exposure control unit 52A of camera A, indicating the brightness (brightness) of the captured image (captured image of camera A) generated by the image generation unit 51A. The brightness (brightness value) of the captured image is, for example, the average brightness of pixels within a predetermined metering area. The metering area is, for example, the entire image area of ​​the captured image. In this embodiment, the brightness of the captured image is the average brightness of all pixels in the captured image when the entire captured image is set as the metering area. However, the range of the metering area and the brightness of the captured image in normal exposure control are not limited to this.

[0061] The exposure control unit 52A controls the exposure control target (exposure value) so that the brightness of the captured image indicated by the brightness information from the exposure processing unit 32 becomes a predetermined appropriate brightness value. Specifically, normal exposure control is performed as follows.

[0062] Brightness information indicating the brightness of the captured image of camera A at the current time t is provided from exposure processing unit 32 to exposure control unit 52A. Exposure control unit 52A determines a target exposure value (target exposure value) for the captured image of camera A to achieve an appropriate brightness value based on the brightness of the captured image of camera A at the current time t and the exposure value at the current time t (the setting value of the exposure control target). Exposure control unit 52A controls the exposure control target such that the exposure of camera A reaches the target exposure value. That is, the exposure control target includes one or more targets such as exposure time, aperture value, and gain as described above. For example, exposure control unit 52A controls the exposure control target to have a setting value pre-associated with the target exposure value. However, in normal exposure control, rapid changes in the exposure value of camera A are suppressed.

[0063] Figure 5 This is a diagram describing normal exposure control.

[0064] exist Figure 5 In the diagram, the horizontal axis represents the number of time steps used, and the vertical axis represents the magnitude of each value on the polyline C1 to C4.

[0065] Line C1 represents the time variation of the brightness of the subject (the entire subject within the imaging range) captured by camera A. Line C2 represents the time variation of the appropriate exposure value (appropriate exposure value) relative to the time variation of the subject's brightness in line C1. Line C3 represents the time variation of the exposure value set by normal exposure control (set exposure value) relative to the time variation of the subject's brightness in line C1. Line C4 represents the time variation of the brightness of the image captured by camera A relative to the time variation of the subject's brightness in line C1 and the time variation of the set exposure value in line C3.

[0066] Note that the brightness of the subject represented by the line C1 is called the subject brightness C1. The appropriate exposure value represented by the line C2 is called the appropriate exposure value C2. The set exposure value represented by the line C3 is called the set exposure value C3. The brightness of the captured image represented by the line C4 is called the brightness C4.

[0067] The brightness C1 of the subject is constant from time 0 to 2. Assume that the brightness C1 of the subject was constant even before time 0. At time 3, the brightness C1 of the subject increases rapidly, and from time 3 to 6, the brightness C1 of the subject becomes constant. At time 7, the brightness C1 of the subject decreases rapidly to the brightness level of time 0 to 2, and from time 7 to 9, the brightness C1 of the subject becomes constant. Note that the temporal variation in the brightness C1 of the subject is caused, for example, by the change in the subject captured by the image of camera A due to the temporal change in the position of camera A.

[0068] The appropriate exposure value C2 corresponds to a subject brightness C1 that is constant between times 0 and 2. Between times 3 and 6, the subject brightness C1 is brighter than between times 0 and 2, and therefore the appropriate exposure value C2 is also greater and constant than between times 0 and 2. Between times 7 and 9, the subject brightness C1 is the same as between times 0 and 2, and therefore the appropriate exposure value C2 is the same as and constant between times 0 and 2.

[0069] When the exposure value of camera A is set according to the appropriate exposure value C2, the brightness of the captured image of camera A becomes the appropriate brightness value (constant value).

[0070] The exposure value C3 is set based on the brightness of the captured image from camera A at a time step prior to each time step. C4 is changed through normal exposure control.

[0071] Between time points 0 and 2, the brightness C1 of the subject remains unchanged, and setting the exposure value C3 is the appropriate exposure value C2. Assume that the brightness C4 is the appropriate brightness value.

[0072] At time 3, the brightness C4 at time 2 is an appropriate brightness value, and the exposure value C3 is the same as the exposure value set at times 0 to 2.

[0073] At time 4, the brightness C4 at time 3 has increased in response to the increase in the brightness C1 of the subject at time 3, and therefore the exposure value C3 is changed to be greater than the exposure value C3 at time 3. However, to suppress the rapid change, the exposure value C3 is therefore less than the appropriate exposure value C2.

[0074] At time 5, the brightness C4 at time 4 is greater than the appropriate brightness value, and therefore the exposure value C3 is changed to be greater than the exposure value C3 set at time 4. However, rapid changes are suppressed, and therefore the exposure value C3 is less than the appropriate exposure value C2.

[0075] At time 6, the exposure value C3 is increased to match the appropriate exposure value C2 for the brightness C1 of the subject at time 6. Therefore, the brightness C4 becomes the appropriate brightness value.

[0076] At time 7, the brightness C4 at time 6 is the appropriate brightness value, and therefore the exposure value C3 is set to the same value as the exposure value C3 set at time 6.

[0077] As described above, at times 4 to 7, the exposure value C3 is gradually changed relative to the constant brightness C1 of the subject at times 3 to 6, in a direction approaching a constant appropriate exposure value C2. That is, when a difference occurs between the brightness C4 and the appropriate brightness value, the exposure value C3 is changed in order to gradually reduce the difference with a time step delay.

[0078] At time 8, since the brightness C4 at time 7 decreases as the brightness C1 of the subject at time 7 decreases, the exposure value C3 is set to be smaller than the exposure value C3 set at time 7. However, rapid changes are suppressed, and therefore the exposure value C3 is set to be larger than the appropriate exposure value C2.

[0079] At times 8 and 9, as at times 4 to 6, the exposure value C3 is set to gradually change relative to the brightness C1 of the subject at times 7 and 8 in a direction close to the appropriate exposure value C2.

[0080] As described above, in normal exposure control, rapid changes in the exposure setting are suppressed. The process of suppressing rapid changes in the exposure setting corresponds to a process that delays the time response of setting the exposure value to an appropriate exposure value relative to the brightness of the subject (the brightness of the captured image).

[0081] Normal exposure control can be achieved through the following methods: Exposure control unit 52A changes the target exposure value such that the difference between the brightness of the captured image, represented by the brightness information (brightness) sequentially provided from exposure processing unit 32, and an appropriate brightness value gradually decreases to zero. Exposure control unit 52A changes the setting value of the exposure control target so that the exposure value becomes the changed target exposure value. Alternatively, exposure control unit 52A uses the brightness information (brightness) sequentially provided from exposure processing unit 32 as a brightness signal to remove high-frequency components from the brightness signal. Exposure control unit 52A assumes the processed brightness signal as brightness information from exposure processing unit 32. Exposure control unit 52A uses an appropriate exposure value as the target exposure value to change the setting value of the exposure control target so that the exposure value becomes the target exposure value, wherein, for this appropriate exposure value, the brightness indicated by the assumed brightness information becomes the appropriate brightness value. However, the specific method of normal exposure control is not limited to this.

[0082] Here, camera A (and camera B) continuously provide the self-position estimation unit 31 with one captured frame at each time step Δt. In the SLAM processing of the self-position estimation unit 31, whenever a new frame of captured image is provided, the positions of feature points (feature points in an image representing the same object point) that were detected from the previous frame of captured image are detected on the captured image. The self-position is estimated based on the feature point positions of the current frame of captured image and the 3D positions of the same feature points from the previous frame of captured image.

[0083] When the position of a moving object changes slowly (or at a low speed), a portion of the captured image contains both subjects that will no longer be included and subjects that will be newly included. When the exposure changes rapidly due to the brightness of the subject, the brightness of the surrounding image of feature points present in the captured image changes rapidly between the previous and current frames. In this situation, it is difficult to detect feature points in the current frame, and self-position estimation may fail. To prevent this, rapid changes in exposure are suppressed in normal exposure control.

[0084] On the other hand, when the position of a moving object changes rapidly (at high speed), the subject appearing in the captured image varies greatly between the previous and current frames, and the brightness of the captured image may change much faster than when the position of the moving object changes slowly. For example, when the entire captured image switches from a shaded subject to a sunlit subject, or vice versa, the brightness of the captured image may change rapidly, potentially leading to overexposure or underexposure. In such cases, normal exposure control requires time to optimize the exposure, and therefore, situations arise where feature points cannot be detected and position estimation fails before the exposure becomes appropriate.

[0085] Figure 6 This is a diagram illustrating a problem with normal exposure control.

[0086] Figure 6 States 131 and 132 in the diagram represent the states of the moving body 133 at times t1 and t2, respectively. The moving body 133 includes... Figure 1 There are two cameras, A and B, with different imaging directions. The angle formed by the imaging directions (optical axes) of cameras A and B is, for example, approximately 90 degrees.

[0087] In state 131, captured images 134-1 and 135-1 are examples of captured images from cameras A and B respectively at time t1. In state 132, captured images 134-2 and 135-2 are examples of captured images from cameras A and B at time t2. Light source 136 represents a bright subject such as the sun. Note that it is assumed that a bright subject such as the sun is only light source 136.

[0088] In state 131 at time t1, assume that cameras A and B capture images 134-1 and 135-1 with appropriate exposures, respectively. Captured image 134-1 includes non-high-brightness subjects such as cars or houses, and does not include high-brightness light sources 136. Captured image 135-1 includes high-brightness light sources 136 and non-high-brightness subjects such as trees or houses. Therefore, the set exposure value for camera B is higher than that for camera A.

[0089] For example, suppose the moving body 133 is rotating at high speed. Between time t1 and time t2, the moving body 133 rotates approximately 90 degrees clockwise and changes to state 132 at time t2. In state 132 at time t2, cameras A and B capture images 134-2 and 135-2, respectively.

[0090] The captured image 134-2 includes a subject within an imaging range that is substantially the same as the imaging range of the captured image 135-1 in state 131 of camera B at time t1, and includes a high-brightness light source 136.

[0091] The captured image 135-2 includes non-high-brightness subjects such as building complexes and does not include high-brightness light sources 136.

[0092] Focusing on images 134-1 and 134-2 captured by camera A, and assuming camera A's exposure control is normal, the exposure of camera A at time t2 is set based on the brightness of the image captured at time t1. Furthermore, sudden changes in exposure are suppressed.

[0093] The high-brightness light source 136 is not included in the captured image 134-1 of camera A at time t1, but is included in the captured image 134-2 of camera A at time t2. Therefore, the exposure value of camera A at time t2 is significantly less than the appropriate exposure value, and there is a possibility that the captured image 134-2 is too bright and overexposed.

[0094] Conversely, suppose that the high-brightness light source 136 is included in the captured image 134-1 of camera A in state 131 at time t1, and that the high-brightness light source 136 is not included in the captured image 134-2 of camera B in state 132 at time t2. In this case, under normal exposure control, the exposure value of camera A at time t2 is significantly greater than the appropriate exposure value, and therefore there is a possibility that the captured image 134-2 is too dark and underexposed.

[0095] As mentioned above, when overexposure or underexposure occurs in the captured image, there is a problem that feature points are difficult to detect and the self-position estimation unit 31 has difficulty performing its own position estimation.

[0096] According to Patent Document 1 (Japanese Patent Application Publication No. 2012-134845), exposure control is performed in a vehicle-mounted camera to ensure that the brightness of the image in the recognition area is appropriate. In the exposure control, the brightness of an area farther than the current recognition area is measured based on the vehicle's speed. The exposure of the vehicle-mounted camera is controlled based on the measured brightness to ensure that the brightness of the image in the recognition area is appropriate. Therefore, even when the recognition area changes from a sunlit image to a shadowed image, the image in the recognition area still has appropriate brightness.

[0097] However, in the technology described in Patent Document 1, the subject that appears in the recognition area at the next time point must be included in the image captured by the camera at the current time point. Figure 6As shown, the technology in Patent Document 1 cannot handle situations where the subject to be captured is drastically altered by the high-speed rotation and movement of the camera.

[0098] Therefore, in this technology, in addition to normal exposure control, exposure control based on predicted brightness, as described below, is performed as the exposure control for camera A.

[0099] <First Embodiment of Exposure Processing Unit 32>

[0100] Figure 7 It is shown Figure 1 A block diagram illustrating a configuration example of the first embodiment of the exposure processing unit 32.

[0101] exist Figure 7 In the process, the exposure processing unit 32 includes a self-position prediction unit 151, an overlapping area calculation unit 152, a brightness prediction unit 153, and an overexposure and underexposure determination unit 154.

[0102] (Self-position prediction unit 151)

[0103] Based on its own position provided by the self-position estimation unit 31, the self-position prediction unit 151 predicts its own position (the position of camera A) at time t+1 after a time step Δt from the current time t. The self-position prediction unit 151 provides the predicted position at time t+1 together with the self-position at the current time t from the self-position estimation unit 31 to the overlapping area calculation unit 152.

[0104] Specifically, the self-position prediction unit 151 calculates the change in position of camera A during the time interval [t-1, t) from time t-1 to the current time t, based on the positions of camera A at time t-1 and the current time t provided by the self-position estimation unit 31. The self-position prediction unit 151 also calculates the average moving speed of camera A during the time interval [t-1, t), assuming that camera A performs constant-speed motion during the time interval [t-1, t).

[0105] The self-position prediction unit 151 assumes that camera A also performs constant speed motion in the time interval [t, t+1) from the current time t to time t+1, and uses the moving speed of camera A in the time interval [t, t+1) as the average moving speed of camera A in the time interval [t-1, t).

[0106] The self-position prediction unit 151 predicts the position of camera A at time t+1 based on the current position of camera A at the current time t and the average moving speed of camera A during the time interval [t-1, t). The predicted position of camera A at time t+1 corresponds to the position obtained by adding the position change of camera A during the time interval [t-1, t) to the current position of camera A at the current time t.

[0107] Note that in the exposure processing unit 32 of this embodiment, the estimation result of the self-position processed by the SLAM of the self-position estimation unit 31 is used when calculating the predicted position of the camera A, but it is also possible to use the detection result of the self-position by any other method.

[0108] (Calculation unit 152 for overlapping regions)

[0109] The overlapping region calculation unit 152 calculates the overlapping region based at least on the predicted position at time t+1 from its own position prediction unit 151, in which the imaging range of camera B at the current time t and the predicted imaging range of camera A at time t+1 overlap with each other.

[0110] The overlapping area refers to the entire area (called the complete overlapping area) or a part of the area (called the partial overlapping area) where the imaging range of camera B at the current time t and the imaging range of camera A at time t+1 overlap.

[0111] The overlapping region not only represents a region in real space, but also represents the entire region (called the complete overlapping region) or a part of the region (called the partial overlapping region) in the image region, where the entire overlapping region in real space appears relative to the captured images of cameras A and B at the current time t.

[0112] The overlapping region serves as information for specifying an image region from which the brightness prediction unit 153 in a subsequent stage extracts the image from the captured image of camera B at the current time t. In this embodiment, the overlapping region calculation unit 152 calculates the overlapping region in the captured image of camera B at the current time t as the overlapping region. The calculation of the overlapping region will be described later.

[0113] The overlapping region calculation unit 152 calculates the brightness of the captured image referenced in the normal exposure control of each of cameras A and B, based on the captured image from each of cameras A and B at the current time t. As mentioned above, the brightness of the captured image is, for example, the average brightness of all pixels in the captured image of each of cameras A and B at the current time t.

[0114] The overlapping area calculation unit 152 provides the calculated brightness of the captured image of camera A as brightness information to camera A (exposure control unit 52A). The overlapping area calculation unit 152 provides the calculated brightness of the captured image of camera B as brightness information to camera B (exposure control unit 52B).

[0115] Note that the brightness of the captured image of each of cameras A and B referenced in normal exposure control can be calculated by a processing unit other than the overlapping area calculation unit 152. For example, the brightness of the captured image of camera A can be calculated by the exposure control unit 52A, and the brightness of the captured image of camera B can be calculated by the exposure control unit 52B.

[0116] Here, a first to a third calculation method that can be used as a method for calculating overlapping regions in overlapping region calculation unit 152 will be described.

[0117] (First calculation method for overlapping regions)

[0118] Figure 8 and Figure 9 This is a diagram describing the first calculation method for overlapping regions.

[0119] exist Figure 8 In the diagram, the solid lines indicate the current positions of the moving body 133 and cameras A and B at time t. Cameras A and B are held by the moving body 133. The dashed lines indicate the predicted positions of the moving body 133 and cameras A and B at time t+1. The moving body 133 and cameras A and B rotate clockwise during the time interval [t, t+1) from the current time t to time t+1.

[0120] In the real space (subject space) that can be captured by cameras A and B, examples of subjects are shown: a car 181, a house 182, a tree 183, a house 184, and a group of buildings 185.

[0121] Imaging ranges 171-1 and 172-1 represent the imaging ranges of cameras A and B at the current time t, and imaging ranges 171-2 and 172-2 represent the imaging ranges of cameras A and B at time t+1.

[0122] Figure 9 Capture images 134-1 and 135-1 represent the captured images of cameras A and B at the current time t, and capture images 134-2 and 135-2 represent the captured images of cameras A and B at time t+1.

[0123] Figure 8Car 181 and house 182 appear in the captured image 134-1 of camera A at the current time t, and Figure 8 Tree 183 and house 184 appear in the captured image 135-1 of camera B at the current time t.

[0124] Figure 8 Tree 183 appears in image 134-2 captured by camera A at time t+1, and Figure 8 The group of buildings 185 appears in the captured image 135-2 of camera B at time t+1.

[0125] In the first calculation method for the overlapping region, the overlapping region calculation unit 152 calculates the imaging range 172-1 of camera B at the current time t based on its own position at the current time t from the self-position prediction unit 151 (see [link]). Figure 8 The overlapping region calculation unit 152 calculates (predicts) the imaging range 171-2 of camera A at time t+1 based on the predicted position of camera A from its own position prediction unit 151 at time t+1 (see...). Figure 8 ).

[0126] The overlapping region calculation unit 152 calculates the overlapping region 173 (see reference) of the image range 172-1 of camera B at the current time t and the image range 171-2 of camera A at time t+1. Figure 8 Tree 183 exists in the overlapping region 173.

[0127] The overlapping region calculation unit 152 calculates the overlapping region 173-1 in the captured image 135-1 of camera B at the current time t based on the calculated overlapping region 173 and the imaging range 172-1 of camera B at the current time t (see [link]). Figure 9 Therefore, the overlapping region 173-1 in the captured image 135-1 is calculated. Although tree 183 and house 184 appear in the captured image 135-1 of camera B at time t, the overlapping region 173-1 is limited to the image region in which only tree 183 is captured. The captured image 134-2 at the predicted position of camera A at time t+1 includes an image similar to the overlapping region 173-1 in the overlapping region 173-2 where overlapping region 173 appears.

[0128] Note that the overlapping region 173 can be a completely overlapping region or a partially overlapping region within a predetermined distance from camera A or camera B. In the case where the overlapping region 173 is a partially overlapping region within a predetermined distance from camera A or camera B, the overlapping region 173-1 in the captured image 135-1 of camera B at the current time t is limited to the image region in the captured image 135-1 where the partially overlapping region appears.

[0129] (Second calculation method for overlapping regions)

[0130] Figure 10 This is a diagram describing the second calculation method for the overlapping region. In the attached diagram, [the diagram shows the method for calculating the overlapping region]. Figure 8 and Figure 9 The same components are given the same reference numerals, but the reference numerals are partially omitted.

[0131] Figure 10 State 201 indicates the position of the moving body 133 and cameras A and B at the current time t. State 202 indicates the predicted position of the moving body 133 and cameras A and B at time t+1. The moving body 133 and cameras A and B rotate clockwise during the time interval [t, t+1) from the current time to time t+1.

[0132] In the real space (subject space) captured by cameras A and B, as an example of a subject, we show... Figure 8 The subjects shown include trees 183 and houses 184. Figure 8 Light source 136, not shown in the diagram.

[0133] In state 201, captured images 134-1 and 135-1 represent the captured images of cameras A and B at the current time t, respectively. In state 202, captured image 134-2 represents the captured image of camera A at time t+1.

[0134] In state 201, camera A captures a car and a house in image 134-1 at current time t (corresponding to...). Figure 8 In the image 135-1 captured by camera B at the current time t, a light source 136, a tree 183, and a house 184 appear. In the image 134-2 captured by camera A at the predicted position at time t+1 in state 202, tree 183 appears. However, in the captured image 134-1, in addition to tree 183, house 184 (not shown) may appear.

[0135] In the captured image 135-1 of camera B in state 201 at the current time t, feature points 211B and 212B represent feature points detected by SLAM processing in the self-position estimation unit 31. Feature point 211B is the image of object point 211 belonging to tree 183. Feature point 212B is the image of object point 212 belonging to house 184. Note that object points in real space that appear as feature points in the captured image are called feature object points.

[0136] In the captured image 134-2 of camera A at time t+1, feature point 211A represents the location where feature object point 211 belonging to tree 183 appears. Feature point 211A is a feature point of the same feature object point 211 as feature point 211B in captured image 135-1.

[0137] In the second calculation method for the overlapping region, the overlapping region calculation unit 152 calculates (predicts) the imaging range of camera A at time t+1 based on the predicted position of camera A from its own position prediction unit 151 at time t+1 (corresponding to...). Figure 8 The imaging range is 171-2). The overlapping region calculation unit 152 obtains from its own position estimation unit 31 the positions (coordinates in the absolute coordinate system) of feature object points 211 and 212 corresponding to feature points 211B and 212B in the captured image 135-1 of camera B at the current time t. Note that the own position estimation unit 31 estimates the current position of the moving body 133 and creates an environment map, and detects the positions of the feature object points when creating the environment map.

[0138] The overlapping region calculation unit 152, based on the three-dimensional positions of feature object points 211 and 212 obtained from its own position estimation unit 31, selects feature object point 211 (that is, feature object point 211 presented as feature point 211A relative to the captured image 134-2 of camera A at time t+1) that exists within the imaging range of camera A at time t+1, as the overlapping feature object point. The feature object point 211 selected as the overlapping feature object point exists within... Figure 8 The overlapping region is 173.

[0139] In the captured image 135-1 of camera B at the current time t, the overlap region calculation unit 152 calculates a patch region with a predetermined shape as an overlap region 221B. This patch region includes feature points 211B corresponding to feature object points 211 selected as overlap feature object points. Hereinafter, the feature points corresponding to the overlap feature object points are also referred to as overlap feature points. The patch region 221B is, for example, a rectangular region with a predetermined vertical width and a predetermined horizontal width. However, the shape of the patch region is not limited to this.

[0140] In the captured image 134-2 at the predicted position of camera A at time t+1, the overlapping region 221A, which includes the overlapping feature point 211A corresponding to the overlapping feature object point 211, is represented as an image region similar to the overlapping region 221B.

[0141] Note that within the imaging range of camera A at time t+1, there can be multiple overlapping feature object points. For example, in Figure 10 In the example, feature object point 211 belonging to tree 183 and feature object point 212 belonging to house 184 can both be overlapping feature object points. When multiple overlapping feature object points exist, the overlapping region calculation unit 152 can select any one or more overlapping feature object points as overlapping feature object points. When multiple selected overlapping feature object points exist, the captured image 135-1 of camera B at the current time t contains multiple overlapping feature points corresponding to the overlapping feature object points. In this case, the overlapping region calculation unit 152 calculates multiple patch regions as overlapping regions 221B for each of the multiple overlapping feature points.

[0142] According to the second method for calculating the overlapping region, the image of overlapping region 221B in the captured image 135-1 of camera B at the current time t can be reliably matched with the image of overlapping region 221A in the captured image 134-2 of camera A at time t+1 (the image actually captured at time t+1). That is, in the first method for calculating the overlapping region, for example, in... Figure 8 In the image captured by camera B at time t, if there is an occluding object on the side of camera B closer to the overlapping region 173 than the overlapping region 173, within the imaging range 172-1 of camera B at the current time t, the image of the overlapping region 173-1 in the image captured by camera B at the current time t is shown in the image (see...). Figure 9 The image of region 173-2, which may overlap with the image captured by camera A at time t+1 (the captured image actually captured at time t+1), is shown in the image (see [link]). Figure 9 (Mismatch)

[0143] On the other hand, in the second calculation method for the overlapping region, the overlapping region 173 (see...) will be included. Figure 8 The overlapping feature object points in the image actually appear in the overlapping feature points in the captured image 135-1 of camera B are set as overlapping region 221B. Therefore, the mismatch between the image of overlapping region 221B in the captured image 135-1 of camera B at time t and the image of overlapping region 221A in the captured image 134-2 of camera A at time t+1 due to occlusion is reduced.

[0144] According to the second calculation method of the overlapping region, it is not necessary to calculate the overlapping region 173 as in the first calculation method, and the overlapping region 221B in the captured image 135-1 can be easily determined based on whether the feature object point corresponding to the feature point detected from the captured image 135-1 of camera B at time t exists in the imaging range of camera A at time t+1 (that is, whether the feature point can be projected into the region of the captured image 134-2 of camera A at time t+1).

[0145] (Third calculation method for overlapping regions)

[0146] In the third method for calculating the overlapping region, similar to the second method, the overlapping region calculation unit 152 detects overlapping feature points among the feature points included in the captured image of camera B at the current time t. The overlapping region calculation unit 152 calculates the region of the subject image (the image of an individual, such as a tree or a house, appearing in the captured image of camera B at the current time t (corresponding to captured image 135-1 in the figure) to which the overlapping feature points belong as the overlapping region.

[0147] Specifically, the overlapping region calculation unit 152 uses an inference model with a convolutional neural network (CNN) structure to perform inference calculations through a well-known instance segmentation algorithm, and detects the attributes (type) and regions of each individual subject image (individual image) included in the captured image of camera B at the current time t.

[0148] Figure 11 This is a graph showing the inference results of instance segmentation.

[0149] Figure 11 The inference results are shown using different algorithms for comparison with instance segmentation. Inference result 241 indicates the inference result using an object detection (bounding box) algorithm. Inference result 242 indicates the inference result using a semantic segmentation algorithm. Inference result 243 indicates the inference result using an instance segmentation algorithm.

[0150] according to Figure 11Based on inference result 241, in object detection, for each individual image included in the captured image, attributes (type) and the frame (bounding box) of the enclosing region are detected. Based on inference result 242, in semantic segmentation, regions for each attribute of the individual images included in the captured image are detected. In semantic segmentation, individual images of different individuals with the same attribute are not distinguished as individual images of different individuals.

[0151] Based on inference result 243, in instance segmentation, attributes and regions are detected for each individual image included in the captured image. In instance segmentation, even if different individuals have the same attributes, they are distinguished as individual images of different individuals.

[0152] The overlapping region calculation unit 152 inputs the captured image of camera B at the current time t into the inference model, and detects the attributes and regions of each individual image included in the captured image through instance segmentation. Based on the detected attributes and regions of each individual image, the overlapping region calculation unit 152 calculates the region of the individual image to which the overlapping feature points belong, as the overlapping region in the captured image of camera B at the current time t.

[0153] Note that the overlapping region calculation unit 152 can limit the regions of individual images with attributes valid in SLAM processing to the overlapping regions in the individual images to which the overlapping feature points belong. Individual images with attributes valid in SLAM processing are, for example, images of stationary objects (e.g., trees or houses). Stationary objects can include individual images of people, etc., that are moving slowly (with a displacement equal to or less than a predetermined value in one time step). Individual images located at a position that can be considered infinitely far away, even if the individual is a stationary object, can be excluded from individual images with attributes valid in SLAM processing.

[0154] The third calculation method for overlapping regions is a combination of the second calculation method and the instance segmentation method, but it can also be a combination of the first calculation method and the instance segmentation method.

[0155] In other words, the overlapping region calculation unit 152 uses the first calculation method for overlapping regions to calculate the overlapping region in the captured image of camera B at time t as a temporary overlapping region. The overlapping region calculation unit 152 can calculate the region of the individual image with predetermined attributes detected by instance segmentation within the temporary overlapping region calculated by the first calculation method as the overlapping region.

[0156] According to the third calculation method of the overlapping region, the overlapping region in the captured image of camera B at time t is restricted to the region where individuals with predetermined attributes appear (the region of the individual image), and appropriate exposure control is performed on the individual image as the overlapping region through processing described later and exposure control by predicting brightness.

[0157] The overlapping region calculation unit 152 determines whether the overlapping region calculated by any one of the first to third calculation methods has a sufficient area (sufficient overlapping region).

[0158] For example, based on the ratio of the area of ​​the overlapping region to the captured image of camera A (overlapping region ratio), a determination is made as to whether the overlapping region is sufficient. The overlapping region calculation unit 152 calculates the overlapping region ratio and compares the calculated ratio with a predetermined threshold. The overlapping region calculation unit 152 determines that sufficient overlapping region has been obtained if the following formula is satisfied:

[0159] (Area of ​​overlapping region) / (Area of ​​the entire captured image) = (Overlapping region ratio) > (Threshold)

[0160] If the overlap region ratio is equal to or less than a predetermined threshold, the overlap region calculation unit 152 determines that it is not possible to obtain a sufficient overlap region.

[0161] Once it is determined that sufficient overlapping area has been obtained, the overlapping area calculation unit 152 provides the calculated overlapping area to... Figure 7 Brightness prediction unit 153 in the middle.

[0162] If it is determined that insufficient overlapping area has been obtained, the overlapping area calculation unit 152 does not provide the calculated overlapping area to the brightness prediction unit 153.

[0163] Note that if the overlap area calculation unit 152 does not provide the overlap area to the brightness prediction unit 153, then in the subsequent stage, the overexposure and underexposure determination unit 154 does not issue an instruction (execution instruction) to the exposure control unit 52A of camera A to use the predicted brightness for exposure control. If the overexposure and underexposure determination unit 154 does not issue an instruction for exposure control using the predicted brightness, the exposure control unit 52A assumes that an instruction for normal exposure control has been given and executes normal exposure control. Therefore, if the overlap area calculation unit 152 determines that a sufficient overlap area cannot be obtained, the exposure control unit 52A of camera A executes normal exposure control based on the brightness information provided from the overlap area calculation unit 152.

[0164] The exposure control unit 52B of camera B always performs normal exposure control based on the brightness information provided by the overlapping area calculation unit 152.

[0165] (Brightness prediction unit 153)

[0166] Figure 7 The brightness prediction unit 153 predicts the brightness of each pixel of the image in the overlapping region (the subject appearing in the overlapping region) when the image is captured by camera A at the predicted position at time t+1 using the exposure (set exposure value) of camera A at the current time t, based on the captured image from camera B at the current time t, the set exposure values ​​of cameras A and B at the current time t, and the overlapping region from the overlapping region calculation unit 152.

[0167] In other words, the brightness prediction unit 153 extracts an image of the overlapping region based on the image captured by camera B at the current time t and the overlapping region from the overlapping region calculation unit 152. The brightness of each pixel in the extracted image of the overlapping region is the brightness of the overlapping region when camera B captures the image at the current time t (setting the exposure value).

[0168] The brightness prediction unit 153 calculates the brightness of each pixel of the image of the overlapping region when the overlapping region is captured by the camera A, which is exposed at the current time t, based on the exposure values ​​set by cameras A and B at the current time t and the brightness of each pixel of the image of the overlapping region when the overlapping region is captured by camera B, which is exposed at the current time t.

[0169] Therefore, the predicted brightness of each pixel in the overlapping region image is calculated when camera A at the predicted location at time t+1 uses the exposure of camera A at the current time t to capture the image. Note that the predicted brightness of each pixel in the overlapping region image is called the predicted brightness.

[0170] For example, suppose the brightness of a predetermined pixel of interest in the overlapping area of ​​the image is 100 out of a possible value between 0 and 255. Suppose that the exposure time of camera A at current time t is twice the exposure time of camera B, and that the settings for other exposure control targets are the same. That is, suppose that the exposure setting of camera A at current time t is one less than the exposure setting of camera B at current time t.

[0171] In this scenario, if camera A at the predicted location at time t+1 uses the exposure value set by camera A at the current time t to capture an image of the overlapping region, the predicted brightness of the pixel of interest in the image of the overlapping region is 200.

[0172] The brightness prediction unit 153 provides the predicted brightness of each pixel of the image in the overlapping region to the calculated brightness prediction unit 153. Figure 7 The overexposure and underexposure determination unit 154.

[0173] (Unit 154 for determining overexposure and underexposure)

[0174] Figure 7 The overexposure and underexposure determination unit 154 calculates the predicted brightness of the overlapping region, representing the luminance of the image of the overlapping region, based on the predicted brightness of each pixel of the image of the overlapping region from the brightness prediction unit 153. Any of the following first to fourth calculation methods can be used to calculate the predicted brightness of the overlapping region. Hereinafter, the image including the predicted brightness of each pixel of the image of the overlapping region from the brightness prediction unit 153 is referred to as the predicted image of the overlapping region.

[0175] (The first method for calculating the predicted brightness of overlapping regions)

[0176] In the first method for calculating the predicted brightness of the overlapping region, the overexposure and underexposure determination unit 154 calculates the average predicted brightness of all pixels of the predicted image of the overlapping region as the predicted brightness of the overlapping region.

[0177] Note that, instead of the brightness prediction unit 153, when the overexposure and underexposure determination unit 154 calculates the predicted brightness of the overlapping area, it can perform the same processing as the brightness prediction unit 153. In this case, the overexposure and underexposure determination unit 154 acquires the captured image of camera B at the current time t, the set exposure values ​​of cameras A and B at the current time t, and the overlapping area calculated by the overlapping area calculation unit 152.

[0178] The overexposure and underexposure determination unit 154 extracts an image of the overlapping region from the captured image of camera B at the current time t, based on the image captured by camera B at the current time t and the overlapping region calculated by the overlapping region calculation unit 152. The overexposure and underexposure determination unit 154 calculates the average brightness of all pixels in the extracted overlapping region image as the brightness of the overlapping region in the captured image of camera B.

[0179] The overexposure and underexposure determination unit 154 converts the calculated brightness of the overlapping area into the brightness of the overlapping area when the exposure value set by cameras A and B at the current time t is used to capture the image of the overlapping area using the exposure (set exposure value) of camera A at the current time t, and sets the brightness as the predicted brightness of the overlapping area.

[0180] In the second to fourth calculation methods for the predicted brightness of the overlapping region, similarly, instead of the brightness prediction unit 153, when the overexposure and underexposure determination unit 154 calculates the predicted brightness of the overlapping region, the same processing as that of the brightness prediction unit 153 can be performed.

[0181] (Second method for calculating the predicted brightness of overlapping regions)

[0182] In the second method for calculating the predicted brightness of the overlapping region, the overexposure and underexposure determination unit 154 calculates the average predicted brightness of all pixels in the image of the overlapping region within the effective region of the SLAM (visual SLAM) processing of the self-position estimation unit 31, as the predicted brightness of the overlapping region. The effective region in the SLAM processing represents the region equivalent to the overlapping region calculated by the overlapping region calculation unit 152 using the second method for calculating the overlapping region. That is, the effective region in the SLAM processing represents a patch region within the overlapping region that includes feature points and has a predetermined shape (rectangular shape). When the overlapping region calculation unit 152 calculates the overlapping region using the second method for calculating the overlapping region, the entire overlapping region becomes the effective region in the SLAM processing.

[0183] Note that when the sheet region includes the outside of the overlapping region, the overlapping area of ​​the sheet region and the overlapping region is the valid area in SLAM processing.

[0184] (A third method for calculating the predicted brightness of overlapping regions)

[0185] In the third method for predicting brightness in the overlapping region, the overexposure and underexposure determination unit 154 performs inference calculations on the captured image of camera B at the current time t using semantic segmentation or instance segmentation of an inference model with a CNN structure (see [link to relevant documentation]). Figure 11 Based on the results of inference calculations, the overexposure and underexposure determination unit 154 extracts the image of the region in the predicted image of the overlapping region that corresponds to the individual image with attributes that are valid in the SLAM processing of its own position estimation unit 31.

[0186] Individual images (images of the areas where individuals appear) with attributes valid in SLAM processing are, for example, images of fixed objects such as trees or houses. In SLAM processing, when using feature points from images of moving objects at high speeds or individual images considered to be at infinity to perform self-position estimation, inappropriate estimation results may be obtained. Therefore, individual images with attributes valid in SLAM processing may not include images of moving objects or individual images considered to be at infinity. Individual images with attributes valid in SLAM processing can include individual images of people, etc., moving slowly (with displacement equal to or less than a predetermined value in one time step).

[0187] The overexposure and underexposure determination unit 154 calculates the average brightness of all pixels in the image extracted from the predicted image of the overlapping region, or an attribute-based weighted average (since people are easily moved, people are not considered, and houses are trusted more, etc.) as the predicted brightness of the overlapping region.

[0188] (The fourth method for calculating the predicted brightness of overlapping regions)

[0189] In the fourth method for predicting brightness in the overlapping region, a distance measurement sensor, such as a time-of-flight (ToF) camera, LiDAR, or stereo camera, is used to measure the distance (depth) of the subject included in the captured image (image of the overlapping region) of camera B, in units of one pixel or multiple pixels. The depth of the subject included in the image of the overlapping region can be estimated in units of one pixel or multiple pixels using an inference model of a deep neural network.

[0190] The overexposure and underexposure determination unit 154 calculates the average predicted brightness of all pixels in the image corresponding to the effective region in SLAM processing, or the depth-weighted average of the predicted images based on the overlapping regions, as the predicted brightness of the overlapping regions. The effective region in SLAM processing represents an image region with a depth equal to or less than a predetermined threshold. That is, the effective region in SLAM processing represents the image region where the subject appears at a distance equal to or less than the predetermined threshold from camera B at current time t or camera A at time t+1.

[0191] When the predicted brightness of the overlapping region is calculated using any of the first to fourth calculation methods for the predicted brightness of the overlapping region, the overexposure and underexposure determination unit 154 determines whether overexposure or underexposure has occurred in the image of the overlapping region in the captured image of the camera A at the predicted position at time t+1.

[0192] For example, if the brightness of each pixel in the captured images of cameras A and B can take values ​​from 0 to 255, then the predicted brightness of the overlapping region can also take values ​​from 0 to 255. The closer the predicted brightness of the overlapping region is to zero, the higher the probability of underexposure; and the closer the predicted brightness is to 255, the higher the probability of overexposure. The predicted brightness of the overlapping region calculated using any of the first to fourth calculation methods satisfies the following overexposure determination formula:

[0193] (Predicted brightness of overlapping areas) > (Overexposure threshold)

[0194] The overexposure and underexposure determination unit 154 determines that overexposure has occurred in the image of the overlapping area.

[0195] Under the condition that the predicted brightness of the overlapping region satisfies the following underexposure determination formula:

[0196] (Predicted brightness of overlapping areas) < (underexposure threshold),

[0197] Overexposure and underexposure determination unit 154 determines that underexposure has occurred in the image of the overlapping region. This occurs when neither the overexposure determination formula nor the underexposure determination formula is satisfied, that is, when the predicted brightness of the overlapping region satisfies the following formula:

[0198] (Underexposure threshold) ≤ (Predicted brightness of overlapping area) ≤ (Overexposure threshold)

[0199] Overexposure and underexposure determination unit 154 determines that no overexposure or underexposure has occurred in the image of the overlapping area. Note that the overexposure threshold and underexposure threshold are predetermined values.

[0200] If the predicted brightness of the overlapping area satisfies either the overexposure determination formula or the underexposure determination formula, the overexposure and underexposure determination unit 154 provides the predicted brightness of the overlapping area as brightness information to the exposure control unit 52A of camera A, and gives an instruction (execution instruction) for exposure control using the predicted brightness as a determination result. In this case, the exposure control unit 52A of camera A executes exposure control based on the brightness information from the overexposure and underexposure determination unit 154 using the predicted brightness.

[0201] If the predicted brightness of the overlapping area does not satisfy either the overexposure determination formula or the underexposure determination formula, the overexposure and underexposure determination unit 154 does not provide the predicted brightness or determination result of the overlapping area to the exposure control unit 52A of camera A. In this case, the exposure control unit 52A of camera A performs normal exposure control based on the brightness information of the captured image indicated by the overlapping area calculation unit 152.

[0202] (Exposure control based on predicted brightness)

[0203] This section describes exposure control based on the predicted brightness of camera A.

[0204] When the overexposure and underexposure determination unit 154 provides an instruction for exposure control based on predicted brightness, the exposure control unit 52A of camera A controls the exposure control target (exposure value) based on the predicted brightness of the overlapping region, which is the brightness information provided by the overexposure and underexposure determination unit 154, and the set exposure value of camera A at the current time t (the set value of the exposure control target), so that the brightness of the overlapping region becomes an appropriate brightness value. The brightness of the overlapping region is information representing the brightness of the overlapping region in the captured image of camera A, and similar to the predicted brightness of the overlapping region, it represents the average brightness of all pixels of the image in the overlapping region of the captured image of camera A.

[0205] Specifically, exposure control is performed as follows based on predicted brightness.

[0206] The predicted brightness of the overlapping region in the captured image of camera A at the predicted position at time t+1 is provided as brightness information from the overexposure and underexposure determination unit 154 to the exposure control unit 52A at the current time t. The exposure control unit 52A determines the target exposure value to be set based on the predicted brightness of the overlapping region from the overexposure and underexposure determination unit 154 and the set exposure value of camera A at the current time t (the setting value of the exposure control target). This target exposure value is determined as an exposure value based on the difference between the predicted brightness of the overlapping region and an appropriate brightness value, and is an exposure value used to make the brightness of the overlapping region reach an appropriate brightness value.

[0207] Exposure control unit 52A controls the exposure control target, causing the exposure of camera A to become the target exposure value. That is, the exposure control target includes one or more of the targets described above: exposure time, aperture value, and gain. For example, exposure control unit 52A controls the exposure control target to have a pre-set value associated with the target exposure value. Note that, unlike normal exposure control, in exposure control by predicting brightness, rapid changes in the exposure value of camera A are allowed.

[0208] Figure 12 This is a diagram describing exposure control based on predicted brightness.

[0209] exist Figure 12 In the diagram, the horizontal axis represents the number of time steps used, and the vertical axis represents the magnitude of the value of each of the polylines D1 to D4.

[0210] Line D1 represents the time variation of the brightness of the subject (the entire subject within the imaging range) captured by camera A. Line D2 represents the time variation of the appropriate exposure value relative to the time variation of the subject's brightness in line D1. Line D3 represents the time variation of the set exposure value set using exposure control based on predicted brightness relative to the time variation of the subject's brightness in line D1. Line D4 represents the time variation of the brightness of the image captured by camera A relative to the time variation of the subject's brightness in line D1 and the time variation of the set exposure value in line D3.

[0211] Note that the brightness of the subject represented by the line D1 is called the subject brightness D1. The appropriate exposure value represented by the line D2 is called the appropriate exposure value D2. The set exposure value represented by the line D3 is called the set exposure value D3. The brightness of the captured image represented by the line D4 is called the brightness D4.

[0212] Here, it is assumed that the position of camera A changes rapidly at a time step (called time t2) after the position of camera A at each time (called time t1), and there is an overlap between the imaging range of camera B at time t1 and the imaging range of camera A at the predicted position at time t2 predicted at time t1. Regarding the overlap region predicted at time t2 at each time t1, it is assumed that the overlap region calculation unit 152 determines that a sufficient overlap region has been obtained.

[0213] To simplify the description, assume that the brightness D4 of the captured image at each time t1 is matched with the brightness of the overlapping region (the actual observed brightness of the overlapping region) of the captured image of camera A at time t1, predicted at a time step prior to time t1. That is, assume that exposure control is performed based on the predicted brightness of the overlapping region of the captured image of camera A at time t1, predicted at a time step prior to time t1. At this point, assume that the brightness of the overlapping region of the captured image of camera A at time t1 becomes an appropriate brightness value, and simultaneously, the brightness of the captured image of camera A at time t1 also becomes an appropriate brightness value.

[0214] The subject's brightness D1 indicator and Figure 5 The brightness C1 of the subject in the image changes over time in a similar manner, and therefore its detailed description will be omitted.

[0215] Appropriate exposure value D2 indication and Figure 5 The time variation of the appropriate exposure value C2 is similar to the time variation in the image, and therefore its description is omitted. Note that the appropriate exposure value D2 represents the exposure value when the brightness of the image captured by camera A becomes the appropriate brightness value, and at the same time, it represents the exposure value of camera A when the brightness of the overlapping area becomes the appropriate brightness value.

[0216] The exposure value D3 is set based on the brightness D4 of the captured image of camera A at a time step prior to the time set by normal exposure control. At the time set by exposure control using predicted brightness, the exposure value D3 is set based on the predicted brightness of the overlapping area in the captured image of camera A at a time step later, which was predicted at a time step prior.

[0217] At times 0 to 2, the brightness D1 of the subject remains constant. The predicted brightness of the overlapping area in the captured image of camera A at times 0 to 2, predicted at a time step earlier, is an appropriate brightness value. In this case, the overexposure and underexposure determination unit 154 determines that neither overexposure nor underexposure occurs. Therefore, the exposure value D3 is set to an appropriate exposure value set by normal exposure control and is a constant value. The brightness D4 is an appropriate brightness value and is a constant value.

[0218] At time 3, as the brightness D1 of the subject increases, the exposure value D3 is changed in the direction that it becomes greater than the exposure value D3 set at time 2, and is immediately set to the appropriate exposure value D2. That is, as the brightness D1 of the subject increases at time 3, the predicted brightness of the overlapping area in the image captured by camera A at time 3, predicted at time 2, becomes greater than the appropriate brightness value. Therefore, it is assumed that the overexposure and underexposure determination unit 154 determines that overexposure has occurred in the image of the overlapping area. In this case, exposure control based on the predicted brightness is performed during the time period from time 2 to time 3. Therefore, the exposure value D3 set at time 3 becomes the appropriate exposure value D2 in which the brightness of the overlapping area becomes the appropriate brightness value. Brightness D4 becomes the appropriate brightness value.

[0219] At time 4, the brightness of the subject is the same as at time 3, and therefore the exposure value D3 is also the same as the exposure value D3 set at time 3. That is, the predicted brightness of the overlapping area in the captured image of camera A at time 4, predicted at time 3, becomes an appropriate brightness value. In this case, the overexposure and underexposure determination unit 154 determines that neither overexposure nor underexposure occurs. Therefore, normal exposure control is performed during the period from time 3 to time 4. In normal exposure control, the exposure value D3 is set in the direction that the difference between the brightness D4 at time 3 and the appropriate brightness value decreases.

[0220] On the other hand, the brightness D4 at time 3 is an appropriate brightness value, and the exposure value D3 is also an appropriate exposure value D2. Therefore, the exposure value D3 at time 4 is the same as the exposure value D3 at time 3. Note that, contrary to the assumptions in this specification, if the brightness of the captured image of camera A differs from the brightness of the overlapping area in the captured image of camera A, the brightness D4 at time 3 may differ from the appropriate brightness value. In this case, the exposure value set by camera A at time 4 is changed in a direction that reduces the difference between the brightness D4 at time 3 and the appropriate brightness value controlled by normal exposure.

[0221] At times 5 and 6, for reasons similar to those at time 4, the exposure value D3 is set to the same value as at times 3 and 4. Note that normal exposure control is performed at times 5 and 6. The brightness D4 is adjusted to the appropriate brightness value.

[0222] At time 7, as the brightness D1 of the subject decreases, the exposure value D3 is changed in a direction smaller than the exposure value D3 set at time 6, and is set to an appropriate exposure value D2. That is, as the brightness D1 of the subject decreases, the predicted brightness of the overlapping area in the image captured by camera A at time 7, predicted at time 6, becomes less than the appropriate brightness value. Therefore, it is assumed that the overexposure and underexposure determination unit 154 determines that underexposure has occurred in the image of the overlapping area. In this case, exposure control based on the predicted brightness is performed during the period from time 6 to time 7. Therefore, the exposure value D3 set at time 7 becomes the appropriate exposure value D2, where the brightness of the overlapping area becomes the appropriate brightness value. Brightness D4 becomes the appropriate brightness value.

[0223] At times 8 and 9, for reasons similar to those at times 4 to 6, the exposure value D3 is set to the same value as at time 7. Note that normal exposure control is performed at times 8 and 9. The brightness D4 is adjusted to an appropriate brightness value.

[0224] As described above, when the brightness of the subject changes rapidly with the rapidly changing imaging range of camera A (times 3 and 7), and there is a possibility of overexposure or underexposure, at times 2 and 6, exposure control based on the predicted brightness of the overlapping area calculated from the captured image of camera B is performed. Therefore, the exposure of camera A immediately becomes the appropriate exposure value, and overexposure or underexposure of the captured image of camera A is suppressed.

[0225] The self-position estimation device 11 of this embodiment can also perform exposure control as follows.

[0226] The overexposure and underexposure determination unit 154 uses a distance measurement sensor or inference model, such as that of a ToF camera, to obtain the distance (depth) of the subject appearing in the overlapping region in the captured image of camera B at the current time t, as in the case of employing the fourth calculation method for the predicted brightness of the overlapping region. Based on the obtained depth of the subject, the overexposure and underexposure determination unit 154 detects the range (maximum depth) of the subject's depth where feature points (feature object points) can be detected in the SLAM processing of its own position estimation unit 31 as the effective depth of the subject. The overexposure and underexposure determination unit 154 provides the detected effective depth of the subject, together with the predicted brightness of the overlapping region, to the exposure control unit 52A of camera A.

[0227] As in the above embodiment, the exposure control unit 52A determines the target exposure value based on the predicted brightness of the overlapping area when performing exposure control by predicting brightness. In the above embodiment, the exposure control unit 52A sets the exposure control target to a predetermined setting value corresponding to, for example, the target exposure value; however, in this embodiment, the exposure control unit 52A sets the exposure control target to a predetermined value corresponding to the target exposure value and the depth of the effective subject.

[0228] Here, the shorter the effective depth of the subject, the shallower (shorter) the depth of field for camera A to focus on the subject within that depth. A shallower depth of field allows for a smaller aperture value (the aperture can be opened). With a constant target exposure value, the exposure time can be shortened as the aperture value decreases, thus suppressing motion blur (blurring of the subject due to movement of camera A).

[0229] Therefore, the exposure control unit 52A uses, for example, a data graph in which the settings for exposure time, aperture value, and gain, which are the targets for exposure control, are associated with all combinations of each exposure value that can be taken as the target exposure value and each depth that can be taken as the effective subject depth. By setting the gain to a predetermined fixed value and fixing the target exposure value to an arbitrary exposure value, a data graph is created such that the shorter the effective subject depth, the smaller the aperture value, and therefore the shorter the exposure time.

[0230] The exposure control unit 52A acquires the setting value of the exposure control target corresponding to the target exposure value and the depth of the effective subject from the data graph, and controls the exposure control target to make it become the acquired setting value.

[0231] <Processing procedure of self-position estimation device 11>

[0232] Figure 13 This is a flowchart illustrating the exposure control process performed by the self-position estimation device 11.

[0233] In step S11, the image generation units 51A and 51B of cameras A and B generate captured images and provide them to their own position estimation unit 31. The process proceeds from step S11 to step S12.

[0234] In step S12, the self-position estimation unit 31 estimates its own position based on the captured images from cameras A and B in step S11. The process proceeds from step S12 to step S13.

[0235] In step S13, the self-position prediction unit 151 predicts the position of camera A at time t+1 based on its own position at the current time t estimated in step S12. The process proceeds from step S13 to step S14.

[0236] In step S14, the overlapping region calculation unit 152 calculates the overlapping region where the imaging range of camera A at time t+1 overlaps with the imaging range of camera B at the current time t, based on the position of camera B estimated in step S12 and the position of camera A at time t+1 predicted in step S13. Specifically, the overlapping region calculation unit 152 calculates the image region that overlaps with the captured image of camera A at time t+1 within the entire region of the captured image of camera B at the current time t as the overlapping region. The process proceeds from step S14 to step S15.

[0237] In step S15, the overlapping region calculation unit 152 determines whether the area of ​​the overlapping region calculated in step S14 is greater than a predetermined threshold.

[0238] If the area of ​​the overlapping region is determined to be no greater than a predetermined threshold in step S15, the process proceeds to step S16.

[0239] In step S16, the exposure control unit 52A of camera A performs normal exposure control. The process returns to step S11 and repeats the process that began in step S11.

[0240] If it is determined in step S15 that the area of ​​the overlapping region is greater than a predetermined threshold, the process proceeds to step S17.

[0241] In step S17, the brightness prediction unit 153 predicts the brightness of the overlapping region in the captured image of camera A at time t+1 based on the brightness of the overlapping region in the captured image of camera B at the current time t (predicted brightness). The process proceeds from step S17 to step S18.

[0242] In step S18, the overexposure and underexposure determination unit 154 determines whether overexposure and underexposure have occurred based on the brightness of the overlapping area predicted in step S17 (predicted brightness).

[0243] If it is determined in step S18 that neither overexposure nor underexposure occurs, the process proceeds to step S16, and the exposure control unit 52A of camera A performs normal exposure control.

[0244] If overexposure or underexposure is determined to have occurred in step S18, the process proceeds to step S19.

[0245] In step S19, the exposure control unit 52A of camera A performs exposure control based on the predicted brightness of the overlapping area predicted in step S17 (exposure control by predicted brightness). The process returns to step S11 and repeats the process that started from step S11.

[0246] By using the self-position estimation device 11 of the first embodiment, which is equipped with the exposure processing unit 32 described above, when the brightness of the subject changes rapidly with the imaging range of camera A and there is a possibility of overexposure or underexposure, the exposure control of camera A is appropriately performed based on the predicted brightness of the overlapping area calculated from the captured image of camera B. Therefore, overexposure or underexposure in the captured image of camera A is suppressed. In self-position estimation, overexposure or underexposure in the captured image is suppressed, enabling appropriate self-position estimation to be performed.

[0247] <Second Embodiment of Exposure Processing Unit 32>

[0248] Figure 14 It is shown Figure 1A block diagram illustrating a configuration example of the second embodiment of the exposure processing unit 32. Note that in the accompanying drawings, [the text is incomplete and lacks context]. Figure 7 The corresponding parts of the exposure processing unit 32 are indicated by the same reference numerals, and their descriptions will be omitted.

[0249] Figure 14 The exposure processing unit 32 includes a self-position prediction unit 261, an overlap area calculation unit 152, a brightness prediction unit 153, an overexposure and underexposure determination unit 154, an inertial measurement unit (IMU) 262, and an integration unit 263. Therefore, Figure 14 Exposure processing unit 32 and Figure 7 The similarities between these two systems lie in the inclusion of an overlapping area calculation unit 152, a brightness prediction unit 153, and an overexposure and underexposure determination unit 154. However, Figure 14 Exposure processing unit 32 and Figure 7 The difference in the case is that a self-position prediction unit 261 is set instead of Figure 7 The system includes a self-position prediction unit 151, and newly added an IMU 262 and an integration unit 263.

[0250] The self-position prediction unit 261 predicts its own position (the position of camera A) at time t+1 based on its own position at the current time t provided by the self-position estimation unit 31 and the change in its own position during the time interval [t, t+1] from the current time t to time t+1 provided by the integration unit 263. The self-position prediction unit 261 provides the predicted position to the overlapping area calculation unit 152.

[0251] IMU 262 includes an accelerometer and an angular velocity sensor, and provides an acceleration signal indicating acceleration on three orthogonal axes and an angular velocity signal indicating angular velocity about three orthogonal axes generated in the moving body 133 to the integration unit 263.

[0252] Integrator 263 performs integration processing on the acceleration and angular velocity from IMU 262 and calculates the change in its own position (change in position and attitude) during the time interval [t, t+1) from the current time t to time t+1. In other words, integrator 263 predicts the change in its own position during the time interval [t, t+1] based on the acceleration and angular velocity signals provided by IMU 262 at times after the current time t and before time t+1.

[0253] The integrator 263 provides the predicted position change to its own position prediction unit 261.

[0254] According to the second embodiment of the exposure processing unit 32, the self-position prediction unit 261 can predict the position of camera A at time t+1 relative to the current time t using the measurement data of IMU 262. By predicting the position of camera A at time t+1 using the measurement data of IMU 262 after the current time t, the prediction can be performed with high accuracy. The position of camera A at time t+1 can also be predicted for any positional changes of camera A that begin after the current time t.

[0255] The self-position estimation device 11 of the second embodiment, including the exposure processing unit 32, appropriately performs exposure control of camera A when the brightness of the subject changes rapidly with the imaging range of camera A and there is a possibility of overexposure or underexposure. This is done based on the predicted brightness of the overlapping area calculated from the captured image of camera B. Therefore, overexposure or underexposure in the captured image of camera A is suppressed. In self-position estimation, overexposure or underexposure of the captured image is suppressed, thereby appropriately performing self-position estimation.

[0256] Although the above-described embodiment of the self-position estimation device 11 relates to exposure control of a camera used for self-position estimation, this technology can be applied to exposure control of multiple cameras for applications other than the self-position estimation device.

[0257] In the above embodiment, the exposure processing unit 32 predicts the position (imaging range) of camera A at time t+1, one time step after the current time t, at the current time t. However, it can also predict the position (imaging range) of camera A at any time (second time) after the current time t, at a time before the second time.

[0258] Although the exposure processing unit 32 in the above embodiment extracts the image of the overlapping area from the captured image of camera B at the current time t, it is also possible to store the captured image of camera B at a time before the current time t in the storage unit, and to extract the image of the overlapping area from the captured image of camera B at any time before the current time t (first time).

[0259] In other words, the exposure processing unit 32 of the above embodiment can also predict the position (imaging range) of camera A at the second time after the first time at the time before the second time, extract the image of the overlapping area from the captured image of camera B at the first time, and control the exposure of camera A at the second time based on the extracted image of the overlapping area. In the overlapping area, the predicted imaging range of camera A at the second time overlaps with the imaging range of camera B at the first time.

[0260] <program>

[0261] Some or all of the processes in the self-position estimation device 11 described above can be executed by hardware or software. In the case where the processes are executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer embedded in dedicated hardware, such as a general-purpose personal computer that can perform various functions by installing various programs, and so on.

[0262] Figure 15 This is a block diagram illustrating an example configuration of computer hardware that performs the above series of processes through a program.

[0263] In a computer, the central processing unit (CPU) 601, read-only memory (ROM) 602, and random access memory (RAM) 603 are interconnected via a bus 604.

[0264] The input / output interface 605 is further connected to the bus 604. The input unit 606, output unit 607, storage unit 608, communication unit 609, and driver 610 are connected to the input / output interface 605.

[0265] Input unit 606 includes a keyboard, mouse, microphone, etc. Output unit 607 includes a display, speakers, etc. Storage unit 608 includes a hard disk, non-volatile memory, etc. Communication unit 609 includes a network interface, etc. Driver 610 drives removable media 611 such as disk, optical disk, magneto-optical disk, or semiconductor memory.

[0266] In a computer configured as described above, for example, CPU 601 loads a program stored in storage unit 608 into RAM 603 via input / output interface 605 and bus 604, and executes the program to perform the series of processes described above.

[0267] For example, a program executed by a computer (CPU 601) can be provided by recording it in a removable medium 611, such as a packaging medium. Alternatively, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0268] In a computer, a program can be installed in the storage unit 608 via the input / output interface 605 by installing the removable medium 611 onto the drive 610. Alternatively, the program can be received via a wired or wireless transmission medium through the communication unit 609 and installed in the storage unit 608. Additionally, the program can be pre-installed in the ROM 602 or the storage unit 608.

[0269] Note that a program executed by a computer may be a program for processing in a time sequence in the order described in this specification, or it may be a program for parallel processing or for processing such as necessary timing when a call is made.

[0270] This technology can have the following configurations.

[0271] (1) An information processing device, comprising:

[0272] The processing unit predicts the imaging range of the first camera at the second time point, which is a time point between the first time point and the second time point, extracts an image of an overlapping region from the captured image of the second camera at the first time point, and controls the exposure of the first camera at the second time point based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time point and the imaging range of the second camera at the first time point overlap.

[0273] (2) The information processing apparatus according to claim 1, wherein,

[0274] The first time is the time before the current time, and the second time is the time after the current time.

[0275] (3) The information processing apparatus according to (1), wherein,

[0276] The first camera and the second camera have different imaging directions.

[0277] (4) The information processing apparatus according to (1) or (3) further includes

[0278] The self-position estimation unit estimates its own position based on the captured images from the first camera and the captured images from the second camera, and the position and attitude of at least one or more of the moving bodies of the first camera and the second camera.

[0279] (5) The information processing apparatus according to (4), wherein,

[0280] The processing unit predicts the imaging range of the first camera by predicting the position and orientation of the first camera at the second time based on its own position estimated by the self-position estimation unit at a time prior to the second time.

[0281] (6) The information processing apparatus according to any one of (1) to (5), wherein,

[0282] The first camera and the second camera are cameras used for self-position estimation based on captured images.

[0283] (7) The information processing apparatus according to any one of (1) to (6), wherein,

[0284] The processing unit

[0285] The imaging range of the first camera at the second time is estimated based on the changes in the position and orientation of the first camera during different time periods before the second time.

[0286] (8) The information processing apparatus according to any one of (1) to (6), wherein,

[0287] The imaging range of the first camera at the second time is estimated based on the acceleration and angular velocity from the inertial measurement unit during different time periods prior to the second time.

[0288] (9) The information processing apparatus according to any one of (1) to (8), wherein,

[0289] The processing unit restricts the image of the overlapping region extracted from the captured image of the second camera at the first time to an image region in the captured image of the second camera that appears within a predetermined distance from the first camera or the second camera.

[0290] (10) The information processing apparatus according to any one of (1) to (9), wherein,

[0291] The processing unit restricts the image of the overlapping region extracted from the captured image of the second camera at the first time point to an image of a portion of the captured image of the second camera that includes feature points.

[0292] (11) The information processing apparatus according to any one of (1) to (10), wherein,

[0293] The processing unit restricts the image of the overlapping region extracted from the captured image of the second camera at the first time to an image of the region in the captured image of the second camera where an individual with a predetermined attribute appears.

[0294] (12) The information processing apparatus according to (11), wherein,

[0295] The predetermined attributes are limited to fixed objects.

[0296] (13) The information processing apparatus according to any one of (1) to (12), wherein,

[0297] The processing unit controls the exposure of the first camera based on the image of the overlapping region after the brightness of each pixel of the image of the overlapping region extracted from the image captured by the second camera at the first time is converted to the brightness corresponding to the exposure of the first camera at the current time.

[0298] (14) The information processing apparatus according to any one of (1) to (13), wherein,

[0299] The processing unit controls the exposure of the first camera based on the average brightness of all pixels in the image of the overlapping region.

[0300] (15) The information processing apparatus according to any one of (1) to (13), wherein,

[0301] The processing unit controls the exposure of the first camera based on the brightness of a portion of the image including feature points in the overlapping region.

[0302] (16) The information processing apparatus according to (15), wherein,

[0303] The processing unit controls the exposure of the first camera based on the brightness of the region of the image that includes feature points and has a predetermined shape in the overlapping region.

[0304] (17) The information processing apparatus according to any one of (1) to (16), wherein,

[0305] The processing unit controls the exposure of the first camera based on the brightness of the image of the region in which an individual with a predetermined attribute appears in the image of the overlapping region.

[0306] (18) The information processing apparatus according to any one of (1) to (17), wherein,

[0307] The processing unit controls the exposure of the first camera based on the brightness of the image of the overlapping region, in which the distance from the first camera or the second camera is equal to or less than a predetermined threshold.

[0308] (19) An information processing method for an information processing apparatus including a processing unit, the information processing method comprising, through the processing unit:

[0309] The system predicts the imaging range of the first camera at a second time after the current time, extracts an image of an overlapping region from a captured image of the second camera at a first time before the current time, and controls the exposure of the first camera based on the extracted image of the overlapping region, in which the predicted imaging range of the first camera at the second time and the imaging range of the second camera at the first time overlap.

[0310] (20) A program for enabling a computer to be used as a processing unit:

[0311] The processing unit predicts the imaging range of the first camera at a second time after the current time, extracts an image of an overlapping region from a captured image of the second camera at a first time before the current time, and controls the exposure of the first camera based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time and the imaging range of the second camera at the first time overlap in the overlapping region.

[0312] Reference tag list

[0313] 11 Self-position estimation device, 31 Self-position estimation unit, 32 Exposure processing unit, 51A, 51B Image generation units, 52A, 52B Exposure control unit, 151 Self-position prediction unit, 152 Overlapping area calculation unit, 153 Brightness prediction unit, 154 Overexposure and underexposure determination unit, A, B cameras

Claims

1. An information processing apparatus, comprising: The processing unit predicts the imaging range of the first camera at the second time point, which is a time point between the first and second time points, after the first time point; extracts an image of an overlapping region from the image captured by the second camera at the first time point; and controls the exposure of the first camera at the second time point based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time point overlaps with the imaging range of the second camera at the first time point. The processing unit limits the image of the overlapping region extracted from the captured image of the second camera at the first time to an image region in the captured image of the second camera that appears within a predetermined distance from the first camera or the second camera.

2. The information processing apparatus according to claim 1, wherein, The first time is the time before the current time, and the second time is the time after the current time.

3. The information processing apparatus according to claim 1, wherein, The first camera and the second camera have different imaging directions.

4. The information processing apparatus according to claim 1, further comprising: The self-position estimation unit estimates its own position based on the captured images from the first camera and the captured images from the second camera, and the position and attitude of at least one or more of the first camera and the second camera.

5. The information processing apparatus according to claim 4, wherein, The processing unit predicts the imaging range of the first camera by predicting the position and orientation of the first camera at the second time based on its own position estimated by the self-position estimation unit at a time prior to the second time.

6. The information processing apparatus according to claim 1, wherein, The first camera and the second camera are cameras used for self-position estimation based on captured images.

7. The information processing apparatus according to claim 1, wherein, The processing unit The imaging range of the first camera at the second time is estimated based on the changes in the position and orientation of the first camera during different time periods before the second time.

8. The information processing apparatus according to claim 1, wherein, The imaging range of the first camera at the second time is estimated based on the acceleration and angular velocity from the inertial measurement unit during different time periods prior to the second time.

9. The information processing apparatus according to claim 1, wherein, The processing unit controls the exposure of the first camera based on the image of the overlapping region after the brightness of each pixel of the image of the overlapping region extracted from the image captured by the second camera at the first time is converted to the brightness corresponding to the exposure of the first camera at the current time.

10. The information processing apparatus according to claim 1, wherein, The processing unit controls the exposure of the first camera based on the average brightness of all pixels in the image of the overlapping region.

11. The information processing apparatus according to claim 1, wherein, The processing unit controls the exposure of the first camera based on the brightness of a portion of the image including feature points in the overlapping region.

12. The information processing apparatus according to claim 11, wherein, The processing unit controls the exposure of the first camera based on the brightness of the region of the overlapping region that includes feature points and has a predetermined shape.

13. The information processing apparatus according to claim 1, wherein, The processing unit controls the exposure of the first camera based on the brightness of the image of the region in which an individual with a predetermined attribute appears in the image of the overlapping region.

14. The information processing apparatus according to claim 1, wherein, The processing unit controls the exposure of the first camera based on the brightness of the image of the overlapping region, in which the distance from the first camera or the second camera is equal to or less than a predetermined threshold.

15. An information processing apparatus, comprising: The processing unit predicts the imaging range of the first camera at the second time point, which is a time point between the first and second time points, after the first time point; extracts an image of an overlapping region from the image captured by the second camera at the first time point; and controls the exposure of the first camera at the second time point based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time point overlaps with the imaging range of the second camera at the first time point. The processing unit limits the image of the overlapping region extracted from the captured image of the second camera at the first time point to an image of a portion of the captured image of the second camera that includes feature points.

16. An information processing apparatus, comprising: The processing unit predicts the imaging range of the first camera at the second time point, which is a time point between the first and second time points, after the first time point; extracts an image of an overlapping region from the image captured by the second camera at the first time point; and controls the exposure of the first camera at the second time point based on the extracted image of the overlapping region, wherein the predicted imaging range of the first camera at the second time point overlaps with the imaging range of the second camera at the first time point. The processing unit limits the image of the overlapping region extracted from the captured image of the second camera at the first time point to images of regions in the captured image of the second camera that contain individuals with predetermined attributes. The predetermined attribute is limited to a fixed object.

17. An information processing method for an information processing apparatus including a processing unit, the information processing method comprising, through the processing unit: The imaging range of the first camera at the second time is predicted at a time point between the first and second time points, after the first time point. An image of the overlapping region is extracted from the image captured by the second camera at the first time point. The exposure of the first camera at the second time point is controlled based on the extracted image of the overlapping region, where the predicted imaging range of the first camera at the second time point overlaps with the imaging range of the second camera at the first time point. in, The image of the overlapping region extracted from the captured image of the second camera at the first time point is limited to the image of the region in the captured image of the second camera that appears within a predetermined distance from the first camera or the second camera.

18. A program product comprising a program for enabling a computer to be used as a processing unit. The processing unit predicts the imaging range of the first camera at the second time point, which is a time point between the first and second time points, and extracts an image of the overlapping region from the image captured by the second camera at the first time point. Based on the extracted image of the overlapping region, the unit controls the exposure of the first camera at the second time point. In the overlapping region, the predicted imaging range of the first camera at the second time point overlaps with the imaging range of the second camera at the first time point. in, The processing unit restricts the image of the overlapping region extracted from the captured image of the second camera at the first time to an image region in the captured image of the second camera that appears within a predetermined distance from the first camera or the second camera.

19. A removable medium storing a program that enables a computer to be used as a processing unit. The processing unit predicts the imaging range of the first camera at the second time point, which is a time point between the first and second time points, and extracts an image of the overlapping region from the image captured by the second camera at the first time point. Based on the extracted image of the overlapping region, the unit controls the exposure of the first camera at the second time point. In the overlapping region, the predicted imaging range of the first camera at the second time point overlaps with the imaging range of the second camera at the first time point. in, The processing unit restricts the image of the overlapping region extracted from the captured image of the second camera at the first time to an image region in the captured image of the second camera that appears within a predetermined distance from the first camera or the second camera.

Images