Information processing apparatus, information processing method, and program

Abstract

To appropriately estimate a movement amount and a movement direction of a moving object.SOLUTION: There is provided an information processing apparatus including: an acquisition unit that acquires each image captured at each point in time by an image capturing device mounted on a moving object; a determination unit that determines the type of movement of the moving object based on the respective images; and an estimation unit that, based on a first image and a second image captured at a time interval corresponding to the type of movement from the first image, estimates a movement amount and a movement direction of the moving object from a first point in time when the first image was captured to a second point in time when the second image was captured.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to, for example, an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, a technique for calculating an optical flow, which is a vector indicating the movement of feature points in temporally consecutive time-series images, is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, estimating the position where the imaging device has moved using the optical flow has not been studied. Other problems and novel features will become apparent from the description of this specification and the accompanying drawings.

Means for Solving the Problems

[0005] In one embodiment, there is provided an information processing apparatus including: an acquisition unit that acquires each image captured at each time point by an imaging device mounted on a moving body; a determination unit that determines the type of movement of the moving body based on each of the images; and an estimation unit that estimates the amount and direction of movement of the moving body from a first time point at which the first image was captured to a second time point at which the second image was captured, based on the first image and a second image captured at a time interval corresponding to the type of movement from the first image.

Effects of the Invention

[0006] According to the above-described embodiment, the amount and direction of movement of the moving body can be appropriately estimated. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a moving object according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a hardware configuration of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of processing by the information processing device according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of an optical flow according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are provided for illustrative purposes only, to help those skilled in the art understand and practice the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0009] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0010] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that each drawing is merely an example for describing one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than with only one particular embodiment. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings, for example, to create an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0011] <System configuration> The configuration of a moving body 1 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a moving body 1 according to an embodiment. In the example of Fig. 1, the moving body 1 has an information processing device 10 and an image capturing device 20. In the example of Fig. 1, the information processing device 10 and the image capturing device 20 are connected so as to be able to communicate with each other via a network N. Note that the number of information processing devices 10 and image capturing devices 20 is not limited to the example of Fig. 1.

[0012] Examples of the network N include, for example, the Internet, a mobile communication system, a wireless LAN (Local Area Network), a LAN, and a bus, etc. Examples of the mobile communication system include, for example, a fifth generation mobile communication system (5G), a sixth generation mobile communication system (6G, Beyond 5G), a fourth generation mobile communication system (4G), a third generation mobile communication system (3G), etc.

[0013] The photographing device 20 is a photographing device (camera, monocular camera) mounted on the moving body 1 and captures images at each point in time. The photographing device 20 may be installed, for example, so that the traveling direction of the moving body 1 is the photographing range.

[0014] The information processing device 10 is, for example, a microcomputer, an ECU (Electronic Control Unit), a server, a cloud server, etc. The information processing device 10 estimates the movement amount and movement direction of the moving object 1 based on each image captured by the imaging device 20, for example.

[0015] The moving body 1 may be, for example, a vehicle that runs on a road or the like using wheels, a railway vehicle that runs on a track, a robot that moves on land using wheels or legs, an unmanned aerial vehicle (drone), an aircraft, a ship, or the like.

[0016] The moving body 1 may assist the driving operation of a driver (operator) by an advanced driver-assistance system (ADAS) based on, for example, the amount of movement and the direction of movement estimated by the information processing device 10. In addition, the moving body 1 may grasp the current position of the moving body 1 based on, for example, the amount of movement and the direction of movement estimated by the information processing device 10, and move (travel) automatically (autonomously).

[0017] <Configuration of information processing device 10> The configuration of the information processing device 10 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 10 according to the embodiment. The information processing device 10 has an acquisition unit 11, a determination unit 12, and an estimation unit 13. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and a memory of the information processing device 10.

[0018] The acquisition unit 11 acquires each image captured at each time point by the imaging device 20 mounted on the moving object 1. The determination unit 12 determines the type of movement of the moving object 1 based on each image acquired by the acquisition unit 11.

[0019] The estimation unit 13 estimates the amount and direction of movement of the moving body 1 from a first point in time when the first image is taken to a second point in time when the second image is taken, based on the first image and a second image taken at a time interval from the first image that corresponds to the type of movement of the moving body 1.

[0020] <Hardware configuration> Fig. 3 is a block diagram showing an example of a hardware configuration of an information processing device 10 according to an embodiment. In the example of Fig. 3, the information processing device 10 includes a processor 101, a memory 102, and a communication interface 103, which may be connected by a bus. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0021] When the program 104 is executed by the processor 101 and the memory 102 in cooperation with each other, the information processing device 10 performs at least a part of the processing of the embodiment of the present disclosure. The memory 102 may be of any type. The memory 102 may be a non-transitory computer-readable storage medium, as a non-limiting example. The memory 102 may also be implemented using any suitable data storage technology, such as a semiconductor-based memory device, a magnetic memory device, an optical memory device, a fixed memory, and a removable memory. Although only one memory 102 is shown in the information processing device 10, several physically different memory modules may exist in the information processing device 10. The processor 101 may be of any type. The processor 101 may include one or more of a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as a non-limiting example.

[0022] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0023] <Processing> Next, an example of the process of the information processing device 10 according to the embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of the process of the information processing device 10 according to the embodiment. Fig. 5 is a diagram showing an example of the optical flow according to the embodiment.

[0024] In step S101, the acquisition unit 11 acquires each image captured at each time point by the imaging device 20 mounted on the moving object 1. Here, the acquisition unit 11 may acquire each image captured at a specific frame rate (for example, 60 fps (frames per second)), for example.

[0025] Subsequently, the determination unit 12 determines the type of movement of the moving body 1 based on each image acquired by the acquisition unit 11 (step S102). Here, the type of movement of the moving body 1 may indicate at least one of the moving speed of the moving body 1, the curvature of the moving direction of the moving body 1, and the change in the gradient of the moving direction of the moving body 1. The curvature of the moving direction may be, for example, information indicating the degree of turning to the right or left when the moving body 1 is traveling on a curve such as a road. The curvature may be the reciprocal of the radius of curvature. The radius of curvature may be the radius of the circle when the curve is locally regarded as an arc of a circle. The larger the radius of curvature, the gentler the turning of the curve. Therefore, the greater the curvature, the sharper the turning of the curve.

[0026] The change in the gradient of the moving direction may be, for example, information indicating the degree of change in the upward or downward inclination when the moving body 1 is traveling on an uphill or downhill slope.

[0027] The determination unit 12 may calculate an optical flow based on two images taken at a specific time interval (for example, 1 / 60 second), and determine the type of movement based on the optical flow. In this case, the determination unit 12 may determine the type of movement based on, for example, the magnitude and direction of each vector (flow vector) included in the calculated optical flow. The optical flow is a vector indicating the movement of an object, which is calculated based on an image taken at a certain point in time and an image taken at a point in time after that point. The determination unit 12 may calculate the optical flow using, for example, the gradient method, which is a method of estimating the flow vector from the conditions of the constraint equation of the spatio-temporal differential of the image. In addition, the determination unit 12 may calculate the optical flow using, for example, the block matching method, which is a method of dividing the image of the previous frame into regions of a specific size, searching for them in the image of the next frame, and detecting the region with the highest similarity to the region of interest in the previous frame.

[0028] For example, the determination unit 12 may determine that the greater the deviation of the position where the horizontal component of each flow vector changes from the center of the image (the traveling direction of the moving body 1) to the left or right, the greater the curvature of the moving direction of the moving body 1 (the sharper the curve).

[0029] 5, in image 501, flows 521, etc. to the left of region 517 are oriented leftward, and flows 522, etc. to the right of region 517 are oriented rightward. Region 517, where the horizontal components of the flow vectors change, is located at a position shifted to the right of the center of image 501 to a certain extent. In this case, determination unit 12 may determine, for example, that the type of movement of moving object 1 is a "gentle curve."

[0030] In this case, the determination unit 12 may divide the image 501 into, for example, eight regions 511 to 518 of equal width in the horizontal direction. Then, the determination unit 12 may search for positions (regions) where the horizontal components of the flow vectors in each of the regions 511 to 518 switch between positive and negative. In this case, the determination unit 12 may, for example, calculate the average value of the horizontal components of each of the flow vectors in each of the regions 511 to 518. Then, the determination unit 12 may, for example, determine the region where the positive and negative values ​​of the average values ​​of the horizontal components of each of the flow vectors in each of the regions 511 to 518 change.

[0031] The determination unit 12 may also determine a "stop" when, for example, the average value of the magnitude (length) of each flow vector is smaller than a threshold value. The determination unit 12 may also determine a "sharp turn" when, for example, all of the horizontal components of each flow vector are in the same direction. The determination unit 12 may also determine a "straight forward" when, for example, the position at which the horizontal component of each flow vector changes is near the center of the image. The determination unit 12 may also determine that the change in the gradient of the moving direction of the moving object 1 is greater as, for example, the position at which the vertical component of each flow vector changes is shifted upward or downward from the center of the image.

[0032] (An example of determining the type of movement of moving object 1 based on machine learning) The determination unit 12 may determine the type of movement of the moving object 1 based on machine learning. In this case, the determination unit 12 may determine (infer) the type of movement of the moving object 1 according to each vector included in the optical flow using, for example, a convolutional neural network (CNN), a deep neural network (DNN), a transformer, or a support vector machine (SVM).

[0033] (Example of determining the type of movement of moving object 1 based on a model) The determination unit 12 may determine the type of movement of the moving object 1 based on a preregistered optical flow model corresponding to each type of movement. In this case, the determination unit 12 may determine the type of movement of the moving object 1 based on, for example, the similarity between each vector included in each model and each vector included in the calculated optical flow. In this case, the determination unit 12 may determine, for example, the type of movement of the model having the highest similarity to the calculated optical flow as the type of movement of the moving object 1.

[0034] (Example of determining the type of movement of moving object 1 based on a sensor) The determination unit 12 may determine the type of movement of the moving object 1 based on, for example, an acceleration sensor mounted on the moving object 1 or operation information of a steering wheel or the like acquired from an ECU of the moving object 1.

[0035] Next, the estimation unit 13 specifies the time interval between the capture times of the two images for estimating the amount of movement and the direction of movement of the moving object 1 based on the type of movement of the moving object 1 (step S103).

[0036] Here, the estimation unit 13 may determine the time interval to be longer, for example, as the curvature of the moving direction of the moving object 1 is smaller (the curve is gentler). In this case, the estimation unit 13 may determine the time interval to be a first time interval (for example, 1 / 60 seconds) in the case of the above-mentioned "sharp curve," and may determine the time interval to be a second time interval (for example, 1 / 10 seconds) longer than the first time interval in the case of the above-mentioned "gentle curve."

[0037] When calculating the optical flow, flow vectors unrelated to the movement of the moving object 1 may be detected due to the movement of an object, erroneous matching, etc. When the magnitude of each calculated flow vector is relatively large, it is relatively easy to remove the flow vector unrelated to the movement of the moving object 1 as noise. This is because, for example, the flow vector unrelated to the movement of the moving object 1 has a relatively clear difference in magnitude and direction compared to other nearby flow vectors in the image.

[0038] When the moving object 1 is moving, the magnitude of each calculated flow vector can be made relatively large by determining the time interval to be long. Therefore, it becomes relatively easy to remove flow vectors that are unrelated to the movement of the moving object 1 as noise. Therefore, the amount and direction of movement of the moving object can be appropriately estimated.

[0039] Also, the estimation unit 13 may determine the time interval to be longer as the moving speed of the moving object 1 is slower. This makes it relatively easy to remove, as noise, flow vectors unrelated to the movement of the moving object 1, as described above. Therefore, the amount and direction of movement of the moving object can be appropriately estimated.

[0040] Also, the estimation unit 13 may determine the time interval to be longer, for example, as the change in gradient in the moving direction of the moving object 1 is smaller. This makes it relatively easy to remove, as noise, flow vectors unrelated to the movement of the moving object 1, as described above. Therefore, the amount and direction of movement of the moving object can be appropriately estimated.

[0041] Next, the estimation unit 13 estimates the amount and direction of movement of the moving object 1 from the first time point when the first image is captured to the second time point when the second image is captured based on the first image and the second image captured at the time interval from the first image (step S104). This makes it possible to estimate, for example, the position of the moving object 1 after it has moved. The estimation unit 13 may output the estimation result to an ECU for autonomous driving or the like. This makes it possible to implement, for example, Visual SLAM (Simultaneous Localization and Mapping), which creates an environmental map and estimates the moving object 1's own position on the environmental map by sensing the surrounding environment while the moving object 1 is traveling.

[0042] Here, the acquisition unit 11 may acquire each image captured by the image capturing device 20 at a specific frame rate. Then, the estimation unit 13 may extract a first image and a second image from each image acquired by the acquisition unit 11. This makes it possible to estimate the position of the moving object 1 after it has moved while keeping the frame rate of the image capturing by the image capturing device 20 constant.

[0043] <Other> When realizing autonomous driving or driving assistance of a moving object, adding sensors other than a camera increases the cost of the system. In addition, the complexity of the processing increases, so the amount of software development increases and the processing delay of the data pipeline also increases. On the other hand, according to the present disclosure, the movement amount and movement direction of a moving object can be appropriately estimated without using sensors other than a camera.

[0044] <Modification> The information processing device 10 may be a device contained in one housing, but the information processing device 10 of the present disclosure is not limited to this. Each unit of the information processing device 10 may be realized by cloud computing configured by one or more computers, for example. In addition, the information processing device 10 and the photographing device 20 may be housed in the same housing and configured as an integrated information processing device. In addition, at least a part of the processing of each functional unit of the information processing device 10 may be executed by the photographing device 20. Such information processing devices 10 are also included in examples of the "information processing device" of the present disclosure.

[0045] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be appropriately combined with other embodiments. [Explanation of symbols]

[0046] 1. Mobile 10. Information processing device 11 Acquisition Department 12 Judgment section 13 Estimation part 20 Imaging Equipment

Claims

1. an acquisition unit that acquires images captured at each time point by an image capturing device mounted on the moving object; A determination unit that determines a type of movement of the moving object based on each of the images; an estimation unit that estimates a movement amount and a movement direction of the moving object from a first time point when the first image is captured to a second time point when the second image is captured, based on a first image and a second image captured at a time interval corresponding to the type of movement from the first image; An information processing device having the above configuration.

2. The type of movement indicates at least one of a change in a moving speed of the moving body, a curvature in a moving direction of the moving body, and a change in a gradient in the moving direction of the moving body. The information processing device according to claim 1 .

3. The estimation unit determines the time interval to be longer as the moving speed of the moving object is slower. The information processing device according to claim 1 .

4. The estimation unit determines the time interval to be longer as the curvature of the moving direction of the moving object becomes smaller. The information processing device according to claim 1 .

5. the estimation unit determines the time interval to be longer as the change in gradient of the moving direction of the moving object is smaller; The information processing device according to claim 1 .

6. the determination unit calculates an optical flow based on each of the images, and determines the type of the movement based on the optical flow; The information processing device according to claim 1 .

7. The acquisition unit acquires each of the images captured by the imaging device at a specific frame rate, The estimation unit extracts the first image and the second image from each of the images. The information processing device according to claim 1 .

8. An information processing device, Acquire images taken at each time point by an imaging device mounted on the moving object, determining a type of movement of the moving object based on each of the images; estimating a movement amount and a movement direction of the moving object from a first time point when the first image is captured to a second time point when the second image is captured, based on a first image and a second image captured at a time interval corresponding to the type of movement from the first image; Information processing methods.

9. Acquire images taken at each time point by an imaging device mounted on the moving object, determining a type of movement of the moving object based on each of the images; estimating a movement amount and a movement direction of the moving object from a first time point when the first image is captured to a second time point when the second image is captured, based on a first image and a second image captured at a time interval corresponding to the type of movement from the first image; A program that causes a computer to carry out processing.

Images