Surrounding image estimation device
The surrounding image estimation device estimates future images based on current images and calculates matching degrees to enhance accuracy and account for transmission delays, facilitating real-time remote control of moving objects.
Patent Information
- Application Number
- JP2021151888
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-09-17
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-09-17
AI Technical Summary
Existing image information processing devices cannot estimate a near-future surrounding image of a moving object based on captured images.
A surrounding image estimation device that estimates a surrounding image to be captured after a predetermined time using a moving object's imaging device, calculates the matching degree between estimated and actual images, and outputs results based on this matching.
Enables the estimation of a near-future surrounding image of a moving object with enhanced accuracy and consideration for transmission delays, allowing for real-time remote control without image transmission delays.
Smart Images

Figure 0007577042000001 
Figure 0007577042000002 
Figure 0007577042000003
Abstract
Description
[Technical field]
[0001] One aspect of the present disclosure relates to a surrounding image estimation device that estimates a surrounding image. [Background technology]
[0002] Patent Document 1 listed below discloses an image information processing device that uses surrounding images captured around a moving object. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2021-117833 A Summary of the Invention [Problem to be solved by the invention]
[0004] Since the image information processing device uses an actually captured surrounding image, it is not possible to perform processing based on an estimated near-future surrounding image, for example. Therefore, it is desired to estimate a near-future surrounding image of a moving object. [Means for solving the problem]
[0005] An ambient image estimation device according to one aspect of the present disclosure includes an estimation unit that estimates a ambient image to be captured by an imaging means after a predetermined time based on a ambient image captured by the imaging means provided on a moving object, a calculation unit that calculates a degree of match between the ambient image estimated by the estimation unit and the ambient image actually captured by the imaging means after the predetermined time, and an output unit that provides an output based on the degree of match calculated by the calculation unit.
[0006] In this aspect, a surrounding image that will be captured after a predetermined time by an imaging device provided on the moving object is estimated, that is, a surrounding image in the near future of the moving object can be estimated. Effect of the Invention
[0007] According to one aspect of the present disclosure, it is possible to estimate a near-future surrounding image of a moving object. [Brief description of the drawings]
[0008] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a surrounding image estimation system including a surrounding image estimation device according to an embodiment. [Diagram 2] This is a conceptual diagram of scenario 1. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a surrounding image estimation device according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a surrounding image. [Diagram 5] FIG. 1 is a diagram showing an example of past, present, and future surrounding images. [Figure 6] FIG. 13 is a diagram showing a comparison between a future surroundings image and an actual surroundings image. [Figure 7] 4 is a flowchart illustrating an example of processing executed by the surrounding image estimation device according to the embodiment. [Figure 8] This is a conceptual diagram of scenario 2. [Figure 9] FIG. 13 is a diagram showing an example of a surrounding image near a T-junction. [Figure 10] FIG. 13 is a diagram showing an example of a surrounding image when turning right at a T-junction. [Figure 11] This is a conceptual diagram of turning right at a T-junction. [Figure 12] FIG. 13 is a diagram showing an example of an estimated surrounding image when turning right at a T-junction; [Figure 13] This is a conceptual diagram of scenario 3. [Figure 14] FIG. 13 is a diagram showing an example of an estimated surrounding image (including objects) when turning right at a T-junction. [Figure 15] 10 is a flowchart illustrating another example of processing executed by the surrounding image estimation device according to the embodiment. [Figure 16] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used in the surrounding image estimation device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated descriptions will be omitted. In addition, the embodiments of the present disclosure in the following description are specific examples of the present invention, and the present invention is not limited to these embodiments unless otherwise specified to limit the present invention.
[0010] Fig. 1 is a diagram showing an example of a system configuration of a surrounding image estimation system 4 including a surrounding image estimation device 1 according to an embodiment. As shown in Fig. 1, the surrounding image estimation system 4 includes a surrounding image estimation device 1 (surrounding image estimation device), a moving body 2 (moving object), and a camera 3 (imaging means) provided on the moving body 2. The surrounding image estimation device 1 and the moving body 2 are communicatively connected to each other via a network or the like, and can transmit and receive any information to and from each other at any timing.
[0011] The surrounding image estimation device 1 is a computer device that estimates a near-future surrounding image of a moving body 2. The surrounding image estimation device 1 may be located at a location (remote) away from the moving body 2, or may be provided in the moving body 2. Also, a part of the surrounding image estimation device 1 may be located at a location (remote) away from the moving body 2, and the remaining part of the surrounding image estimation device 1 may be provided in the moving body 2. In this case, the part and the remaining part of the surrounding image estimation device 1 are communicatively connected to each other via a wireless network or the like, and the part and the remaining part function as the surrounding image estimation device 1 together. Details of the surrounding image estimation device 1 will be described later.
[0012] The moving object 2 is a moving (movable) object. The object may be an inanimate object such as an artificial object, or a living object such as a person. The moving object 2 is, for example, a wheelchair, a bicycle, a vehicle such as a car or a truck, an aircraft such as an airplane, a helicopter, a rocket or a drone, a ship (including a submarine), a person, or an animal. The moving object 2 may move using a moving capability that it has, or may move using another moving capability without having a moving capability by itself. The moving object 2 may be equipped with a GPS (Global Positioning System), an acceleration sensor, an angular velocity sensor, a magnetic sensor, or the like that can acquire position information. The moving object 2 may generate moving information including its own current position, acceleration, speed, and moving (traveling) direction as well as a current timestamp using various sensors periodically (for example, every second), and transmit the information to the surrounding image estimation device 1 via a network or the like.
[0013] The camera 3 is an imaging means. The camera 3 captures the surroundings (surroundings, vicinity, close, surrounding, neighbourhood, side, around) and outputs a surrounding image, which is an image of the surroundings. The camera 3 may output a surrounding video (surrounding video), which is a temporal sequence of the surrounding images. In this embodiment, it is assumed that a surrounding image is used, but this is not limiting, and a surrounding video may also be used. That is, in this embodiment, the term "image" may be replaced with "video" or "video" as appropriate.
[0014] The camera 3 may be composed of a camera 3a (first camera) capable of measuring the distance (depth information) to an object in an image being captured, and a camera 3b (second camera) capable of capturing an image of the surroundings. The camera 3a is, for example, a stereo camera or a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) camera. The camera 3b is, for example, a 360-degree camera or a camera capable of capturing an image in multiple directions. The camera 3b may be a general camera rather than a special camera. The camera 3 may be composed of multiple (two or more) cameras. In this embodiment, multiple surrounding images captured by the camera 3 composed of multiple cameras are collectively referred to as surrounding images captured by the camera 3 as appropriate. When outputting the surrounding image, the camera 3 may output surrounding information, which is any information related to the surrounding image. The surrounding information may include a timestamp of the timing at which the surrounding image is captured, and information related to an object in the surrounding image and a distance to the object.
[0015] The surrounding image and surrounding information output by the camera 3 may be transmitted to the surrounding image estimation device 1 via the moving object 2 and a network, and may be displayed to a controller controlling the movement of the moving object 2 via an output device 1006 (described later) of the surrounding image estimation device 1. The controller may also transmit control information relating to control of the movement of the moving object 2 (forward, right turn, stop, etc.) from the surrounding image estimation device 1 to the moving object 2, and the moving object 2 may move in accordance with the control information. In other words, the controller may remotely control (remotely steer, remotely operate) the movement of the moving object 2 based on the surrounding image and surrounding information transmitted from the moving object 2.
[0016] The camera 3 may be equipped with the various sensors described above that the moving body 2 is equipped with, and the camera 3 (rather than the moving body 2) may generate the above-mentioned movement information periodically (e.g., every second) and transmit it to the surrounding image estimation device 1 via a network or the like.
[0017] In Fig. 1, a moving object 2 is a car traveling (moving) on a road. A surrounding image S is an image of the moving direction captured by a camera 3 provided on the moving object 2 and facing in the traveling direction (forward direction) of the moving object 2. The surrounding image S mainly captures the road on which the moving object 2 is traveling, the sidewalk beside the road, people walking on the sidewalk, and scenery such as mountains that spread out in the distance in the traveling direction.
[0018] [Scenario 1] An embodiment in Scenario 1 will be described below. Fig. 2 is a conceptual diagram of Scenario 1. In Scenario 1, as shown in Fig. 1, a scene is assumed in which a moving object 2 such as a car travels on a road with good visibility in the travel direction, and a (moving) object B1 such as a person or an obstacle is present in the travel direction. In Scenario 1, a surrounding image in the near future is estimated from a surrounding image (and surrounding information) that can be captured by (only) a camera 3 provided on the moving object 2.
[0019] Fig. 3 is a diagram showing an example of a functional configuration of the surrounding image estimation device 1 according to the embodiment. As shown in Fig. 3, the surrounding image estimation device 1 includes a storage unit 10 (storage unit), an acquisition unit 11 (acquisition unit), an estimation unit 12 (estimation unit), a calculation unit 13 (calculation unit), and an output unit 14 (output unit). Note that the functional configuration shown in Fig. 3 and specific contents of the functional configuration described below are not limited to Scenario 1, but are also common to Scenario 2 and Scenario 3 described below.
[0020] Each functional block of the surrounding image estimation device 1 is assumed to function within the surrounding image estimation device 1, but is not limited to this. For example, some of the functional blocks of the surrounding image estimation device 1 may function in a computer device different from the surrounding image estimation device 1 and connected to the surrounding image estimation device 1 through a network, while appropriately transmitting and receiving information with the surrounding image estimation device 1. Also, some of the functional blocks of the surrounding image estimation device 1 may be omitted, multiple functional blocks may be integrated into one functional block, or one functional block may be decomposed into multiple functional blocks.
[0021] Hereinafter, each function of the surrounding image estimation device 1 shown in FIG. 3 will be described.
[0022] The storage unit 10 stores any information used in calculations and the like in the surrounding image estimation device 1, and the results of calculations in the surrounding image estimation device 1. The information stored by the storage unit 10 may be referred to by each function of the surrounding image estimation device 1 as appropriate.
[0023] The acquisition unit 11 acquires any information used for calculations in the surrounding image estimation device 1. More specifically, the acquisition unit 11 acquires information from an external device or the like via a network or the like. The acquisition unit 11 causes the storage unit 10 to store the acquired information, or outputs the information to other functional blocks of the surrounding image estimation device 1.
[0024] The acquisition unit 11 acquires a surrounding image, surrounding information, and movement information from the moving object 2 via a network or the like. The surrounding image, surrounding information, and movement information acquired by the acquisition unit 11 are stored by the storage unit 10 as history information in chronological order based on timestamps included in the surrounding information and movement information.
[0025] The estimation unit 12 estimates a surrounding image to be captured by the camera 3 after a predetermined time based on a surrounding image captured by the camera 3 provided in the moving object 2 (or history information stored by the storage unit 10). For example, the estimation unit 12 estimates a surrounding image to be captured by the camera 3 at a time "13:00:02" that is a predetermined time "2 seconds" after "13:00:00" which is a reference time (for example, the current time) based on a plurality of surrounding images captured by the camera 3 up to the present time. Here, since the moving object 2 is movable, the moving object 2 may be in a position different from the position at the time "13:00:02" at the time "13:00:00". The estimation unit 12 may estimate the surrounding image periodically (for example, every second) or continuously. A specific estimation process by the estimation unit 12 will be described later.
[0026] The estimation unit 12 may estimate surrounding images corresponding to a plurality of different predetermined times. For example, the estimation unit 12 estimates a surrounding image captured by the camera 3 at a time "13:00:02" that is a predetermined time "2 seconds" after the reference time "13:00:00", a surrounding image captured by the camera 3 at a time "13:00:04" that is a predetermined time "4 seconds" after the reference time "13:00:00", and a surrounding image captured by the camera 3 at a time "13:00:06" that is a predetermined time "6 seconds" after the reference time "13:00:00".
[0027] The estimation unit 12 may estimate the distance based on the distance to the object in the image captured by the camera 3a and the surrounding image captured by the camera 3b. The specific estimation process will be described later.
[0028] The estimation unit 12 may cause the estimated surrounding image to be stored in the storage unit 10, or may output the estimated surrounding image to another functional block of the surrounding image estimation device 1.
[0029] FIG. 4 is a diagram showing an example of a surrounding image. The situation in FIG. 4 is the same as that in FIG. 1, and differences will be mainly described. The surrounding image Sa is a surrounding image captured three-dimensionally (so that the distance to the object can be measured) by the camera 3a. The surrounding image Sb is a surrounding image captured omnidirectionally by the camera 3b. In the example shown in FIG. 4, the surrounding image Sb is a surrounding image capturing an area wider than the surrounding image Sa, and the area of the surrounding image Sb includes (contains) the area of the surrounding image Sa. The area shown in the surrounding image Sa may be a region of interest that the controller pays attention to. In this case, the area of the surrounding image Sb, other than the area shown in the surrounding image Sa, is outside the region of interest that the controller does not pay attention to. In the surrounding images Sa and Sb, a person B2 (object) walking on the sidewalk on the left side from the back to the front (opposite to the traveling direction of the moving object 2) is captured. In the surrounding image Sb, a person B3 (object) is shown walking on the sidewalk on the right side from the foreground to the background (in the traveling direction of the moving object 2).
[0030] FIG. 5 is a diagram showing an example of surrounding images of the past, present, and future.
[0031] The past surrounding images shown in the left diagram in FIG. 5 are surrounding images Sa and Sb captured by cameras 3a and 3b in the past (for example, 2 seconds before the present time), respectively. Among the past surrounding images, the surrounding image Sa, which is the area of interest, is called the past frame. The distance "20 m" to person B2 shown in the past frame indicates the distance to person B2 measured by camera 3a. Note that in this embodiment, information regarding the distance to the object measured by camera 3a is displayed in the surrounding image Sa in FIG. 5 for convenience, but is actually included in the surrounding information linked to the surrounding image Sa.
[0032] The current surrounding image shown in the middle diagram in FIG. 5 is the surrounding image Sa and the surrounding image Sb currently (at the present time) captured by the camera 3a and the camera 3b, respectively. Among the current surrounding images, the surrounding image Sa, which is the area of interest, is called the current frame. The distance to the person B2 shown in the current frame is "15 m," and the person B2 is closer to the moving object 2 compared to the past frame. On the other hand, the person B3 shown in the current surrounding image is farther away from the moving object 2 compared to the past surrounding image. The current frame has moved upward within the area of the surrounding image Sb compared to the past frame.
[0033] The future surrounding image shown in the right diagram of FIG. 5 is the surrounding image Sa and the surrounding image Sb estimated by the estimation unit 12 to be captured by the camera 3a and the camera 3b, respectively, t seconds (a predetermined time) from the present. Of the future surrounding images, the surrounding image Sa, which is the area of interest, is called the future frame. The distance to person B2 shown in the future frame is "10 m," and person B2 is closer to the moving object 2 compared to the current frame. Meanwhile, person B3 shown in the future surrounding image (and the future frame) is farther away from the moving object 2 compared to the current surrounding image. The person B3 in the future frame has moved upward within the area of the surrounding image Sb compared to the current frame.
[0034] A specific estimation process of the future frame by the estimation unit 12 will be described with reference to the future surrounding image in FIG. 5. First, the surrounding image Sa excluding the person B2 and the person B3 is estimated (by the estimation unit 12) based on the surrounding image Sa of the past surrounding image and its timestamp, the surrounding image Sa and the surrounding image Sb of the current surrounding image and their timestamps, and t seconds. For example, the upward movement amount in t seconds is calculated (assuming that the surrounding image Sa of the past surrounding image moves linearly upward) from the difference in the timestamp between the surrounding image Sa of the past surrounding image and the surrounding image Sa of the current surrounding image, and the difference (amount of upward movement) between the position of the surrounding image Sa of the past surrounding image and the position of the surrounding image Sa of the current surrounding image, and the surrounding image moved by the calculated amount of upward movement is estimated as the future frame (excluding the person B2 and the person B3). The image of the region R1 (upper side of the future frame) shown in FIG. 5, which needs to be newly estimated when moved upward, may be synthesized based on the surrounding image Sb. For example, the image of the region R1 may be generated using the image of the corresponding part in the surrounding image Sb of the current surrounding image. In the above calculation, information on the position, acceleration, speed, and moving direction of the moving object 2 included in the movement information of the history information stored by the storage unit 10 may be used to estimate the accurate movement of the moving object 2, thereby improving the estimation accuracy of the ultimately estimated surrounding image. The surrounding image Sb of the future surrounding image may also be estimated in the same manner as the future frame.
[0035] Next, person B2 in the future frame is estimated (by the estimation unit 12) based on the surrounding information linked to the past frame and the current frame. For example, based on the distance to person B2 included in the surrounding information linked to the past frame and the current frame, the movement vector of person B2 in the x, y, and z axis directions is calculated from the time difference between the past frame and the current frame, and person B2 in the future frame is estimated based on the calculated movement vector and t seconds in the same way as the estimation of the surrounding image Sa described above.
[0036] Finally, person B3 in the future frame is estimated (by the estimation unit 12) based on the surrounding image Sb acquired by the camera 3b in the past surrounding image and the current surrounding image. For example, based on the surrounding image Sb acquired by the camera 3b in the past surrounding image and the current surrounding image, the movement vectors in the x and y axis directions of person B3 are calculated from the time difference between the past surrounding image and the current surrounding image, and person B3 in the future frame is estimated similarly to the estimation of the surrounding image Sa described above based on the calculated movement vector and t seconds. That is, person B3 is estimated to be reflected in the attention area from the movement vector calculated based on the surrounding image Sb, and is superimposed on the future frame.
[0037] As described above, the estimation unit 12 estimates a future frame (surrounding image) that will be captured by the camera 3 provided on the moving object 2 in t seconds, based on the surrounding image captured by the camera 3.
[0038] The estimation unit 12 may output the estimation accuracy for each object when estimating the surrounding image to another functional block by linking it with the surrounding image. For example, the estimation unit 12 may output a low estimation accuracy for objects that move at high speed and objects that do not move in a straight line, and may output a high estimation accuracy for stationary objects and objects that move in a straight line at a low speed.
[0039] The calculation unit 13 calculates the degree of match between the surrounding image estimated by the estimation unit 12 and the surrounding image actually captured by the camera 3 after a predetermined time. For example, when the estimation unit 12 estimates the surrounding image captured by the camera 3 at the time "13:00:02", which is the predetermined time "2 seconds" after the reference time "13:00:00", the calculation unit 13 calculates the degree of match between the estimated surrounding image and the surrounding image actually captured by the camera 3 at the time "13:00:02", which is the predetermined time "2 seconds" after the reference time "13:00:00".
[0040] The surrounding image estimated by the estimation unit 12, which is one of the targets for which the calculation unit 13 calculates the degree of match, may be one output by the estimation unit 12, or may be one stored by the storage unit 10. The surrounding image actually captured by the camera 3 after a predetermined time, which is the other target for which the calculation unit 13 calculates the degree of match, is one received by the surrounding image estimation device 1 from the moving object 2 via a network or the like and stored by the storage unit 10.
[0041] The degree of match is the degree to which two images match. The degree of match may be the similarity between images, which is an existing technology. The degree of match may be expressed, for example, as a real value between "0" and "1", with a value closer to "0" indicating a poorer match and a value closer to "1" indicating a better match.
[0042] The calculation unit 13 may calculate the degree of match based on the position of the object in the surrounding image. For example, assume that an object at position A (a, b) is at position B (c, d) after a predetermined time. Also assume that the estimation unit 12 estimates that the object is at position X (x, y) after the predetermined time. In this case, for example, the calculation unit 13 may calculate the degree of match by calculating the similarity between the surrounding image of position B and the surrounding image of position X.
[0043] The calculation unit 13 may calculate a degree of match for each of the multiple surrounding images estimated by the estimation unit 12. For example, it is assumed that the estimation unit 12 estimates a surrounding image at a time "13:00:02" that is a predetermined time "2 seconds" after the reference time "13:00:00", a surrounding image at a time "13:00:04" that is a predetermined time "4 seconds" after the reference time "13:00:00", and a surrounding image at a time "13:00:06" that is a predetermined time "6 seconds" after the reference time "13:00:00". In this case, the calculation unit 13 calculates the degree of match between the estimated surrounding image at the time "13:00:02" and the surrounding image actually captured at that time, calculates the degree of match between the estimated surrounding image at the time "13:00:04" and the surrounding image actually captured at that time, and calculates the degree of match between the estimated surrounding image at the time "13:00:06" and the surrounding image actually captured at that time.
[0044] The calculation unit 13 may cause the storage unit 10 to store the calculated degree of match, or may output the degree of match to another functional block of the surrounding image estimation device 1.
[0045] FIG. 6 is a diagram showing a comparison between a future surrounding image and an actual surrounding image. The future surrounding image shown in the upper diagram in FIG. 6 is the same as the future surrounding image in FIG. 5. That is, the future frame in the future surrounding image is a surrounding image estimated by the estimation unit 12. The actual surrounding image shown in the lower diagram in FIG. 6 is a surrounding image Sa and a surrounding image Sb actually captured by the camera 3a and the camera 3b, respectively, t seconds (a predetermined time) after the capture time of the current frame in FIG. 5. Among the actual surrounding images, the surrounding image Sa, which is a region of interest, is called an actual frame. The calculation unit 13 calculates (compares) the degree of agreement between the future frame shown in FIG. 6 and the actual frame. Note that, when the future frame is compared with the actual frame, parameter tuning of the estimation performed by the estimation unit 12 may be performed based on the comparison result to improve the accuracy of the estimation. By performing parameter tuning, for example, it is possible to reflect the characteristics of the speed and traveling direction of the moving object 2 due to the habits of the controller.
[0046] The output unit 14 performs output based on the degree of match calculated by the calculation unit 13. For example, the output unit 14 may display the degree of match calculated by the calculation unit 13 to the controller via the output device 1006 (described later) or the like. In addition, for example, the output unit 14 may display the degree of match calculated by the calculation unit 13 and the surrounding image estimated by the estimation unit 12 on which the degree of match is calculated to the controller via the output device 1006 (described later) or the like. The output by the output unit 14 is not limited to display, and may be, for example, output to another functional block of the surrounding image estimation device 1, or transmission to another device via a network or the like.
[0047] The output unit 14 may calculate a predetermined time at which the degree of match satisfies a predetermined criterion based on the degree of match calculated by the calculation unit 13, and output a surrounding image after the predetermined time estimated by the estimation unit 12. For example, when the degree of match is expressed by a real value between "0" and "1" as described above, the output unit 14 calculates a predetermined time (e.g., "2 seconds") at which the degree of match calculated by the calculation unit 13 is equal to or greater than the predetermined criterion of "0.8", and outputs a surrounding image after the predetermined time (from the reference time) estimated by the estimation unit 12.
[0048] The output unit 14 may select a degree of match that satisfies a predetermined criterion from among the multiple degrees of match calculated by the calculation unit 13, and calculate the predetermined time that is the basis of the selected degree of match as the predetermined time at which the degree of match (mentioned above) satisfies the predetermined criterion. For example, if the degree of match for the surrounding image at time "13:00:02" a predetermined time "2 seconds" after the reference time "13:00:00" is "0.9", the degree of match for the surrounding image at time "13:00:04" a predetermined time "4 seconds" after the reference time "13:00:00" is "0.7", the degree of match for the surrounding image at time "13:00:06" a predetermined time "6 seconds" after the reference time "13:00:00" is "0.5", and the predetermined criterion is "0.8" or more, the output unit 14 selects the degree of match of "0.9" that satisfies the predetermined criterion of "0.8" or more, and calculates the predetermined time "2 seconds" on which the selected degree of match is based as the predetermined time for which the degree of match satisfies the predetermined criterion. Then, the output unit 14 outputs the surrounding image at the predetermined time "2 seconds" after the reference time estimated by the estimation unit 12.
[0049] The output unit 14 may perform output based on a transmission delay time until the surrounding image captured by the camera 3 is displayed to the controller who controls the movement of the moving object 2. The transmission delay time is, for example, a delay time involved in image transmission from the moving object 2 to a controller who is remote (at a remote location). The transmission delay time may be specified (predicted) by a difference between a timestamp included in the surrounding information associated with the surrounding image and a timestamp (at the timing of displaying to the controller) obtained by a time function previously provided in the surrounding image estimation device 1. It is assumed that the surrounding image estimation device 1 and the camera 3 are synchronized in time. For example, the output unit 14 may calculate a predetermined time for which the matching degree satisfies a predetermined criterion and the difference from the transmission delay time is equal to or less than a predetermined criterion (for example, "0.5 seconds") based on the degree of matching calculated by the calculation unit 13, and output the surrounding image after the predetermined time estimated by the estimation unit 12.
[0050] The output unit 14 may calculate a predetermined time at which the degree of match satisfies a predetermined criterion based on the degree of match calculated by the calculation unit 13, and when the transmission delay time is within the predetermined time, output a surrounding image after the transmission delay time estimated by the estimation unit 12. For example, as in the above example, when the predetermined time "2 seconds" at which the degree of match satisfies the predetermined criterion of "0.8" or more is calculated and the transmission delay time is "1.5 seconds", the transmission delay time "1.5 seconds" is within the predetermined time "2 seconds", so the output unit 14 outputs a surrounding image after the transmission delay time "1.5 seconds" estimated by the estimation unit 12.
[0051] The output unit 14 may perform output further based on the estimation accuracy of the estimation by the estimation unit 12. For example, the output unit 14 may output (display) the value of the estimation accuracy for each object in the surrounding image output by the estimation unit 12 on the object, or may color-code the object according to the estimation accuracy, thereby making it possible to output the estimation accuracy to the controller.
[0052] Next, an example of processing executed by the surrounding image estimation device 1 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of processing executed by the surrounding image estimation device 1 according to the embodiment.
[0053] First, the estimation unit 12 estimates the surrounding image S to be captured by the camera 3 after a predetermined time t seconds based on the surrounding image S captured by the camera 3 provided on the moving object 2 (step S1, estimation step). Next, the calculation unit 13 calculates the degree of match between the surrounding image S estimated in S1 and the surrounding image S actually captured by the camera 3 after the predetermined time t seconds (step S2, calculation step). Next, the output unit 14 performs output based on the degree of match calculated in S2 (step S3, output step).
[0054] The above is a description of the embodiment in scenario 1.
[0055] [Scenario 2] An embodiment in Scenario 2 will be described below. FIG. 8 is a conceptual diagram of Scenario 2. In Scenario 2, a scene is assumed in which a moving body 2 such as a wheelchair advances along a corridor in a building and turns right at a T-junction (a curve). FIG. 9 is a diagram showing an example of a surrounding image near the T-junction. As in Scenario 1, the moving body 2 can be remotely controlled by a controller. Before the moving body 2 turns right at the T-junction, the camera 3 equipped on the moving body 2 cannot capture an image of the right turn at the T-junction, so a surrounding image in the near future cannot be estimated only from the surrounding image captured by the camera 3. FIG. 10 is a diagram showing an example of a surrounding image when turning right at a T-junction. The surrounding image shown in FIG. 10 is actually captured by the camera 3 equipped on the moving body 2 after the moving body 2 turns right at the T-junction, and as described above, it cannot be captured or estimated before the moving body turns right at the T-junction.
[0056] The estimation unit 12 may estimate based on map information of the vicinity of the predicted position of the moving object 2 after a predetermined time. FIG. 11 is a conceptual diagram of turning right at a T-junction. The estimation unit 12 estimates a surrounding image of an imaging range that can be imaged from a point after a predetermined time t seconds based on map information (map data) of the vicinity of the point. The map data can be acquired from a cloud via a network or the like, and the map data may also include image data (aerial photos, textures, landscape photos, etc.) linked to the map data. The estimation unit 12 may predict a point after a predetermined time t seconds from the speed and current position of the moving object 2. FIG. 12 is a diagram showing an example of an estimated surrounding image when turning right at a T-junction. The area R2 of the surrounding image shown in FIG. 12 is an area estimated by the estimation unit 12 based on map information. The above is an explanation of the embodiment in scenario 2.
[0057] [Scenario 3] An embodiment in scenario 3 will be described below. Fig. 13 is a conceptual diagram of scenario 3. In scenario 3, in addition to the scene in scenario 2, a scene is assumed in which a surveillance camera C, a person B4, and a moving object 2a different from the moving object 2 exist in the corridor at the back of the T-junction. In scenario 3, person B4 (object, obstacle, dynamic object) who turns right at the T-junction is also estimated.
[0058] The estimation unit 12 may make the estimation based on a surrounding image captured by another camera 3 (monitoring camera C or a camera provided on the moving body 2a) existing near the predicted position of the moving body 2 after a predetermined time. The estimation unit 12 may make the estimation based on map information of the vicinity of the predicted position of the moving body 2 after a predetermined time and an object included in the surrounding image captured by another camera 3 existing near the predicted position.
[0059] That is, when a moving object 2a or a monitoring camera C is present at the right turn of a T-junction, the person B4 is estimated by utilizing an image captured by the moving object 2a or the monitoring camera C. The moving object 2a has the same functional configuration as the moving object 2, and transmits the captured surrounding image together with the position information of the moving object 2a to the surrounding image estimation device 1. The moving object 2a or the monitoring camera C estimates the position of the person B4 after a predetermined time t seconds by adopting a method equivalent to scenario 1. The estimation unit 12 of the moving object 2 estimates the surrounding image after the predetermined time t seconds by a method equivalent to scenario 2, and then estimates the person B4 based on the position coordinates and the like (the person B4 can be reproduced). FIG. 14 is a diagram showing an example of an estimated surrounding image (including an object) when turning right at a T-junction. The area R2 of the surrounding image shown in FIG. 14 includes the person B4 estimated by the estimation unit 12. The above is an explanation of the embodiment in scenario 3.
[0060] Next, an example of the processing of scenarios 1 to 3 executed by the surrounding image estimation device 1 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing another example of the processing executed by the surrounding image estimation device 1 according to the embodiment.
[0061] First, the surrounding image estimation device 1 predicts a transmission delay time d (step S10). Next, the surrounding image estimation device 1 predicts the position (self-position) of the moving object 2 d seconds later (step S11). Next, the surrounding image estimation device 1 determines whether or not the surrounding image d seconds later can be estimated only from the surrounding image captured by the moving object 2 (the surrounding image captured from the self-position d seconds later does not have a corner, and estimation is possible only from the surrounding image captured by the moving object 2) (step S12). If it is determined that estimation is possible in S12 (S12: YES), the surrounding image estimation device 1 estimates the surrounding image in the pattern of scenario 1 (step S13). Next, the surrounding image estimation device 1 performs parameter tuning by comparing with the surrounding image actually captured in d seconds later (step S14), and ends the process. If it is determined that estimation is not possible in S12 (S12: NO), the surrounding image estimation device 1 estimates the surrounding image in the pattern of scenario 2 (step S15). Following S15, it is determined whether or not a camera present near the predicted position of the moving object 2 after d seconds has detected an object (step S16). If it is determined in S16 that the object has not been detected (step S16: NO), the process proceeds to S14. If it is determined in S16 that the object has been detected (step S16: YES), the surrounding image estimation device 1 estimates the object in the pattern of scenario 3 (step S17), and the process proceeds to S14.
[0062] Next, the effects of the surrounding image estimation device 1 according to the embodiment will be described.
[0063] According to the surrounding image estimation device 1, the estimation unit 12 estimates a surrounding image to be captured by the camera 3 after a predetermined time based on a surrounding image captured by the camera 3 provided on the moving body 2, the calculation unit 13 calculates a degree of match between the surrounding image estimated by the estimation unit 12 and the surrounding image actually captured by the camera 3 after the predetermined time, and the output unit 14 performs output based on the degree of match calculated by the calculation unit 13. With this configuration, a surrounding image to be captured by the camera 3 provided on the moving body 2 after a predetermined time is estimated. That is, a surrounding image in the near future of the moving body 2 can be estimated. In addition, since an output is performed based on the degree of match between the estimated surrounding image and the surrounding image actually captured, for example, the estimation accuracy of the estimated surrounding image can be easily and effectively utilized.
[0064] Furthermore, according to the surrounding image estimation device 1, the output unit 14 may calculate a predetermined time at which the degree of match satisfies a predetermined criterion based on the degree of match calculated by the calculation unit 13, and output the surrounding image after the predetermined time estimated by the estimation unit 12. With this configuration, it is possible to output an estimated surrounding image with a certain degree of estimation accuracy.
[0065] Moreover, according to the surrounding image estimation device 1, the estimation unit 12 may estimate surrounding images corresponding to each of a plurality of different predetermined times, the calculation unit 13 may calculate a degree of match for each of the plurality of surrounding images estimated by the estimation unit 12, and the output unit 14 may select a degree of match that satisfies a predetermined criterion from the plurality of degrees of match calculated by the calculation unit 13, and calculate the predetermined time on which the selected degree of match is based as the predetermined time at which the degree of match satisfies the predetermined criterion. With this configuration, it is possible to more easily and reliably output an estimated surrounding image with a certain estimation accuracy.
[0066] According to the surrounding image estimation device 1, the camera 3 may be composed of a camera 3a capable of measuring the distance to an object in an image being captured, and a camera 3b capable of capturing an image of the surroundings, and the estimation unit 12 may estimate based on the distance to the object in the image captured by the camera 3a and the surrounding image captured by the camera 3b. This configuration allows for estimation that takes into account the distance to the object, and also allows for estimation based on images captured by different cameras, resulting in more accurate estimation. Also, by acquiring information in the z-axis (depth) direction with the camera 3a, it is possible to estimate a large vector movement in the z-axis direction.
[0067] Furthermore, according to the surrounding image estimation device 1, the output unit 14 may perform output further based on a transmission delay time until the surrounding image captured by the camera 3 is displayed to the controller who controls the movement of the moving object 2. With this configuration, it is possible to perform output taking the transmission delay time into consideration. For example, by outputting a surrounding image in the near future corresponding to the transmission delay time to the controller, it is possible to output a surrounding image of the moving object 2 that appears as if it were in real time to the controller.
[0068] Furthermore, according to the surrounding image estimation device 1, the output unit 14 may calculate a predetermined time at which the degree of match satisfies a predetermined criterion based on the degree of match calculated by the calculation unit 13, and if the transmission delay time is within the predetermined time, output a surrounding image after the transmission delay time estimated by the estimation unit 12. With this configuration, for example, it is possible to output a surrounding image of the moving object 2 in real time with a certain estimation accuracy.
[0069] Furthermore, according to the surrounding image estimation device 1, the output unit 14 may perform output further based on the estimation accuracy of the estimation by the estimation unit 12. With this configuration, the estimation accuracy can be easily and effectively utilized.
[0070] Moreover, according to the surrounding image estimation device 1, the estimation unit 12 may perform estimation based on map information about the vicinity of the predicted position of the moving object 2 after a predetermined time. With this configuration, even parts that cannot be estimated by the moving object 2 itself can be supplemented by the map information.
[0071] Furthermore, according to the surrounding image estimation device 1, the estimation unit 12 may make an estimation based on a surrounding image captured by another camera 3 that is present in the vicinity of the predicted position of the moving object 2 after a predetermined time. With this configuration, even parts that cannot be estimated by the moving object 2 itself can be supplemented by the surrounding image captured by the other camera 3.
[0072] Furthermore, according to the surrounding image estimation device 1, the estimation unit 12 may perform estimation based on map information of the vicinity of the predicted position of the moving object 2 after a predetermined time and an object included in a surrounding image captured by another camera 3 that is present in the vicinity of the predicted position. With this configuration, even parts that cannot be estimated by the moving object 2 itself can be supplemented by the map information and the object included in the surrounding image captured by the other camera 3.
[0073] Here, we will explain the conventional problems. In order to operate a moving object from a remote location, the remote operator needs to grasp the local situation in real time, and currently the mainstream method is to grasp the local situation through video. In order to deliver the local video to the remote operator, a delay of tens to hundreds of milliseconds occurs due to processing such as video transmission, encoding, and decoding, which leads to a delayed response to an emergency stop or a sense of discomfort on the part of the operator.
[0074] The surrounding image estimation device 1 of this embodiment considers the delay that occurs when transmitting an image (video) to a controller (remote operator) and virtually generates an on-site image of about tens to hundreds of milliseconds in the future according to the transmission delay time, thereby realizing a remote control system in which the image transmission delay is not felt. The surrounding image estimation device 1 can generate an image of the controller's attention area in the future by the transmission delay time related to the image transmission from the traveling direction and speed of the moving body 2 and the vector of the surrounding object. When creating a future image, the surrounding image estimation device 1 generates a future image for the image transmission delay at that time. However, if the matching rate is below a predetermined matching rate, a future image with a slight delay is generated so that the matching rate is reached. For example, if there is a delay of 5 seconds and generating a future image of 5 seconds would lower the matching rate, a future image of 3 seconds is generated so that the matching rate is higher. The surrounding image estimation device 1 makes it possible to prevent a delay in remote control response and realize a remote control that does not feel strange. The method by the surrounding image estimation device 1 is a near-future image generation method for real-time remote control.
[0075] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.
[0076] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.
[0077] For example, the surrounding image estimation device 1 according to an embodiment of the present disclosure may function as a computer that performs processing of the surrounding image estimation method of the present disclosure. Fig. 16 is a diagram showing an example of a hardware configuration of the surrounding image estimation device 1 according to an embodiment of the present disclosure. The above-mentioned surrounding image estimation device 1 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.
[0078] In the following description, the term "apparatus" may be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the surrounding image estimation device 1 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.
[0079] Each function in the surrounding image estimation device 1 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communication via a communication device 1004, and control at least one of reading and writing of data in the memory 1002 and the storage 1003.
[0080] The processor 1001 controls the entire computer by running an operating system, for example. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned acquisition unit 11, estimation unit 12, calculation unit 13, output unit 14, etc. may be realized by the processor 1001.
[0081] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to the read out programs. As the program, a program that causes a computer to execute at least a part of the operations described in the above-mentioned embodiment is used. For example, the acquisition unit 11, the estimation unit 12, the calculation unit 13, and the output unit 14 may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be realized in a similar manner. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from a network via a telecommunication line.
[0082] The memory 1002 is a computer-readable recording medium, and may be configured by at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement a wireless communication method according to an embodiment of the present disclosure.
[0083] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.
[0084] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., in order to realize at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD). For example, the above-mentioned acquisition unit 11, estimation unit 12, calculation unit 13, output unit 14, etc. may be realized by the communication device 1004.
[0085] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).
[0086] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.
[0087] Furthermore, the surrounding image estimation device 1 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.
[0088] Notification of information is not limited to the aspects / embodiments described in this disclosure and may be performed using other methods.
[0089] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other suitable systems, and next-generation systems extended based on these. In addition, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A and 5G, etc.) may be applied.
[0090] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0091] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.
[0092] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).
[0093] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).
[0094] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.
[0095] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
[0096] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0097] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0098] In addition, terms explained in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings.
[0099] As used in this disclosure, the terms "system" and "network" are used interchangeably.
[0100] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values from a predetermined value, or may be expressed using other corresponding information.
[0101] The names used for the above parameters are not limiting in any way, and furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure.
[0102] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.
[0103] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.
[0104] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0105] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.
[0106] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.
[0107] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.
[0108] In this disclosure, where articles have been added due to translation, such as, for example, a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.
[0109] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]
[0110] 1...surrounding image estimation device, 2, 2a...moving body, 3, 3a, 3b...camera, 4...surrounding image estimation system, 10...storage section, 11...acquisition section, 12...estimation section, 13...calculation section, 14...output section, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus, B1...object, B2, B3, B4...person, C...surveillance camera, R1, R2...area, S, Sa, Sb...surrounding image.
Claims
1. an estimation unit that estimates a surrounding image to be captured by an imaging means provided on a moving object after a predetermined time based on the surrounding image captured by the imaging means; a calculation unit that calculates a degree of match between the surrounding image estimated by the estimation unit and a surrounding image actually captured by the imaging means after the predetermined time; an output unit that performs output based on the degree of match calculated by the calculation unit; A surrounding image estimation device comprising:
2. The output unit calculates the predetermined time at which the degree of match satisfies a predetermined criterion based on the degree of match calculated by the calculation unit, and outputs the surrounding image after the predetermined time estimated by the estimation unit. The surrounding image estimation device according to claim 1 .
3. The estimation unit estimates surrounding images corresponding to a plurality of different predetermined times, The calculation unit calculates the degree of match for each of the plurality of surrounding images estimated by the estimation unit, The output unit selects a degree of match that satisfies a predetermined criterion from among the plurality of degrees of match calculated by the calculation unit, and calculates the predetermined time on which the selected degree of match is based as the predetermined time at which the degree of match satisfies the predetermined criterion. The surrounding image estimation device according to claim 2 .
4. the imaging means is composed of a first camera capable of measuring a distance to an object in an image being captured, and a second camera capable of capturing an image of the surroundings; The estimation unit estimates the distance to an object in an image captured by the first camera based on an surrounding image captured by the second camera. The surrounding image estimation device according to any one of claims 1 to 3.
5. The output unit performs output based on a transmission delay time until the surrounding image captured by the imaging means is displayed to a controller who controls the movement of the object. The surrounding image estimation device according to any one of claims 1 to 4.
6. The output unit calculates the predetermined time at which the degree of match satisfies a predetermined criterion based on the degree of match calculated by the calculation unit, and when the transmission delay time is within the predetermined time, outputs a surrounding image after the transmission delay time estimated by the estimation unit. The surrounding image estimation device according to claim 5 .
7. The output unit performs output further based on the estimation accuracy of the estimation by the estimation unit. The surrounding image estimation device according to any one of claims 1 to 6.
8. The estimation unit estimates the position of the object based on map information of a vicinity of the predicted position of the object after the predetermined time. The surrounding image estimation device according to any one of claims 1 to 7.
9. the estimation unit estimates the position of the object based on a surrounding image captured by another imaging means present in the vicinity of the predicted position of the object after the predetermined time. The surrounding image estimation device according to any one of claims 1 to 8.
10. the estimation unit estimates the object based on map information of a vicinity of the predicted position of the object after the predetermined time and an object included in a surrounding image captured by another imaging means that is present in the vicinity of the predicted position. The surrounding image estimation device according to any one of claims 1 to 9.
Citation Information
Patent Citations
Lane follow-up device
JP2005038225A
Route guiding device
JP2008145364A
Vehicle control system
JP2020055400A
Image information processor, image information processing method, and image information processing program
JP2021117833A
Camera-based vehicle environment modeling
JP2021513173A
