Information processing device, information processing method, and program

Abstract

The invention relates to an information processing apparatus, an information processing method, and a program. In order to appropriately estimate a motion amount and a motion direction of a moving object, an information processing device is provided with: an acquisition unit that acquires each image captured at each point in time by a photographing device mounted on the moving object; a determination unit that determines a motion type of the moving object based on the corresponding image; and an estimation unit that estimates a motion amount and a motion direction of the moving object from a first time point when the first image is captured to a second time point when the second image is captured, based on the first image and a second image captured at a time interval corresponding to a motion type from the first image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The disclosure of Japanese Patent Application No. 2023-193147 filed on November 13, 2023, including the specification, drawings and abstract, is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to, for example, an information processing device, an information processing method, and a program. Background Art

[0004] Devices The disclosed technology is listed below.

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-004329

[0006] Conventionally, a technique for calculating an optical flow, which is a vector indicating the motion of a feature point in a temporally continuous time-series image, is known (for example, see Patent Document 1). Summary of the invention

[0007] However, Patent Document 1 does not consider using optical flow to estimate the position to which the photographic device has moved. Other objects and novel features will become apparent from the description of this specification and the accompanying drawings.

[0008] In one embodiment, an information processing device is provided, including: an acquisition unit, which acquires each image captured by a photographic device installed on a moving object at each time point; a determination unit, which determines the movement type of the moving object based on the corresponding image; and an estimation unit, which estimates the movement amount and movement direction of the moving object from a first time point when the first image is captured to a second time point when the second image is captured, based on a first image and a second image captured at a time interval corresponding to the movement type.

[0009] According to one embodiment, the amount and direction of movement of a moving object can be appropriately estimated. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram showing an example of a configuration of a moving object according to an embodiment.

[0011] Figure 2 is a block diagram showing an example of the configuration of an information processing apparatus according to an embodiment.

[0012] Figure 3 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to one embodiment.

[0013] Figure 4: is a flowchart showing an example of processing of an information processing apparatus according to one embodiment.

[0014] Figure 5 is a diagram showing an example of optical flow according to one embodiment. DETAILED DESCRIPTION

[0015] The present disclosure is described with reference to several exemplary embodiments. These embodiments are described for illustrative purposes only and are not intended to limit the scope of the present disclosure. It should be understood that these embodiments are helpful for those skilled in the art to understand and implement the present disclosure. The disclosure described in this specification may be implemented in a different manner than described below.

[0016] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0017] In the following, embodiments of the present disclosure will be described with reference to the accompanying drawings. Each of the accompanying drawings is merely illustrative and is used to describe one or more embodiments. Each of the accompanying drawings is not limited to a single specific embodiment, but may be related to one or more other embodiments. As will be appreciated by those skilled in the art, various features or steps described with reference to any one of the accompanying drawings may be combined with features or steps shown in one or more other accompanying drawings to create embodiments that are not explicitly shown or described. Not all features or steps shown in any of the accompanying drawings describing exemplary embodiments are necessary, and some features or steps may be omitted. The order of steps described in any of the accompanying drawings may be changed appropriately.

[0018] (System Configuration)

[0019] refer to Figure 1 , a configuration of a moving object 1 according to an embodiment will be described. Figure 1 FIG. 1 is a diagram showing an example of a configuration of a moving object 1 according to an embodiment. Figure 1 In the example of FIG. 1 , the moving object 1 includes an information processing device 10 and a photographing device 20. Figure 1 In the example of FIG. 1 , the information processing apparatus 10 and the photographing device 20 are connected so as to be able to communicate via the network N. The number of the information processing apparatus 10 and the photographing device 20 is not limited to Figure 1 Examples in .

[0020] Examples of the network N include, for example, the Internet, a mobile communication system, a wireless LAN (Local Area Network), a LAN, and a bus, etc. Examples of the mobile communication system include, for example, a fifth generation mobile communication system (5G), a sixth generation mobile communication system (6G, beyond 5G), a fourth generation mobile communication system (4G), a third generation mobile communication system (3G), etc.

[0021] The photographing device 20 is installed on the moving object 1 and is an imaging device (camera, monocular camera) that captures an image at each time point. The photographing device 20 may be installed, for example, so that the traveling direction of the moving object 1 becomes the imaging range.

[0022] The information processing device 10 is a device such as, for example, a microcomputer, an ECU (Electronic Control Unit), a server, a cloud server, etc. The information processing device 10 estimates, for example, the amount and direction of movement of the moving object 1 based on each image captured by the photographing device 20 .

[0023] For example, the moving object 1 may be a vehicle running on wheels on a road, a railway vehicle running on a track, a robot moving on land by wheels or legs, an unmanned aerial vehicle (UAV), an airplane, a ship, and the like.

[0024] The moving object 1 can support the driving operation of the driver (operator) through an advanced driver assistance system (ADAS) based on the amount of movement and the direction of movement estimated by the information processing device 10. In addition, the moving object 1 can grasp its current position based on the amount of movement and the direction of movement estimated by the information processing device 10, and move (drive) automatically (autonomously).

[0025] (Configuration of Information Processing Device)

[0026] refer to Figure 2 , the configuration of the information processing apparatus 10 according to one embodiment will be described. Figure 2 1 is a diagram showing an example of a configuration of an information processing device 10 according to an embodiment. The information processing device 10 has an acquisition unit 11, a determination unit 12, and an estimation unit 13. These units can be implemented by cooperation of one or more programs installed in the information processing device 10 and hardware of the information processing device 10 (such as a processor and a memory, etc.).

[0027] The acquisition unit 11 acquires each image captured at each time point by the photographing device 20 mounted on the moving object 1. The determination unit 12 determines the type of movement of the moving object 1 based on each image acquired by the acquisition unit 11.

[0028] The estimation unit 13 estimates the movement amount and movement direction of the moving object 1 from a first time point when the first image is captured to a second time point when the second image is captured based on the first image and the second image captured at a time interval corresponding to the movement type of the moving object 1 .

[0029] (Hardware Configuration)

[0030] Figure 3 1 is a block diagram showing an example of the hardware configuration of the information processing device 10 according to one embodiment. Figure 3In the example of FIG. 1 , the information processing device 10 includes a processor 101, a memory 102, and a communication interface 103. These may be connected via a bus. The memory 102 stores at least a portion of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0031] When the program 104 is executed by the cooperation of the processor 101 and the memory 102, at least a portion of the processing of the embodiments disclosed herein is performed by the information processing device 10. The memory 102 may be of any type. By way of non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. In addition, the memory 102 may be implemented using any appropriate data storage technology, such as semiconductor-based memory devices, magnetic memory devices, optical memory devices, fixed memory, and removable memory. Although only one memory 102 is shown in the information processing device 10, there may be several physically different memory modules in the information processing device 10. The processor 101 may be of any type. The processor 101 may include one or more processors based on a microprocessor, a digital signal processor (DSP), and by way of non-limiting example, a multi-core processor architecture.

[0032] When the program is loaded into a computer, the program includes a set of instructions (or software codes) for causing the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, a computer-readable medium or a tangible storage medium may include a random access memory (RAM), a read-only memory (ROM), flash memory, a solid-state drive (SSD) or other memory technology, a CD-ROM, a digital versatile disc (DVD), CD or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device. The program may be transmitted on a transient computer readable medium or communication medium. By way of example and not limitation, a transient computer readable medium or communication medium may include electrical, optical, acoustic or other forms of propagated signals.

[0033] (Processing Flow)

[0034] refer to Figure 4 and Figure 5 , an example of processing in the information processing apparatus 10 according to one embodiment will be described. Figure 4 1 is a flowchart showing an example of processing in the information processing device 10 according to one embodiment. Figure 5 is a diagram showing an example of optical flow according to one embodiment.

[0035] In step S101, the acquisition unit 11 acquires each image captured at each time point by the photographing device 20 mounted on the moving object 1. Here, the acquisition unit 11 may acquire each image captured at a specific frame rate, for example, 60 fps (frames per second).

[0036] Subsequently, the determination unit 12 determines the motion type of the moving object 1 based on each image acquired by the acquisition unit 11 (in step S102). Here, the motion type of the moving object 1 may indicate at least one of the following: the motion speed of the moving object 1, the curvature of the motion direction of the moving object 1, and the gradient change of the motion direction of the moving object 1. For example, the curvature of the motion direction may indicate information about the degree of turning right or left when the moving object 1 travels on a curve of a road or the like. The curvature may be the inverse of the radius of curvature. When the curve is locally regarded as an arc, the radius of curvature may be the radius of the circle. The larger the radius of curvature, the flatter the curve. So the tighter the curve, the greater the curvature.

[0037] When the moving object 1 travels on an uphill or downhill slope, the gradient change of the moving direction may be information indicating the degree of change of the upward or downward inclination.

[0038] The determination unit 12 may calculate the optical flow based on two images captured at a specific time interval (e.g., 1 / 60 seconds), and may determine the type of motion based on the optical flow. In this case, the determination unit 12 may determine the type of motion based on the size and direction of each vector (flow vector) included in the calculated optical flow. The optical flow is a vector indicating the motion of an object, and is calculated based on an image captured at a certain point in time and another image captured at a later point in time. The determination unit 12 may calculate the optical flow using a gradient method, which is a technique for estimating a flow vector based on a condition of a spatiotemporal differential equation based on an image. In addition, the determination unit 12 may calculate the optical flow using a block matching method, which is a technique for dividing an image of a previous frame into regions of a specific size, searching within the image of the next frame, and detecting the region having the highest similarity to the region of interest in the previous frame.

[0039] The more the position of the change in the horizontal component of each flow vector shifts leftward or rightward from the center of the image (the traveling direction of the moving object 1 ), the greater the curvature (the tighter the curve) of the moving direction of the moving object 1 may be determined by the determination unit 12 .

[0040] exist Figure 5 In the example of , in the image 501, each flow 521, etc. on the left side of the region 517 points to the left, and each flow 522, etc. on the right side of the region 517 points to the right. Also, the region 517 where the horizontal component of each flow vector changes is located on the right side of the center of the image 501. In this case, the determination unit 12 may determine the motion type of the moving object 1 as a “gentle curve”.

[0041] In this case, the determination unit 12 may divide the image 501 into eight horizontal regions 511 to 518 of equal width. The determination unit 12 may search for a position (region) where the horizontal component of each flow vector switches between positive and negative in each region 511 to 518. In this case, the determination unit 12 may calculate the average value of the horizontal component of each flow vector in each region 511 to 518. Then, the determination unit 12 may determine the region where the average value of the horizontal component of each flow vector in each region 511 to 518 switches between positive and negative.

[0042] In addition, if the average value of the size (length) of each flow vector is less than a threshold value, the determination unit 12 may determine it as "stop". Moreover, if the horizontal components of all flow vectors are in the same direction, the determination unit 12 may determine it as a "sharp curve". In addition, if the position where the horizontal component of each flow vector changes is near the center of the image, the determination unit 12 may determine it as "straight ahead". In addition, the determination unit 12 may determine that the greater the change in the gradient of the moving direction of the moving object 1, the more the position where the vertical component of each flow vector changes is offset upward or downward from the center of the image.

[0043] (Example 1 of determining the motion type of a moving object)

[0044] As Example 1, the determination unit 12 may determine the motion type of the moving object 1 based on machine learning. In this case, the determination unit 12 may use, for example, CNN (Convolutional Neural Network), DNN (Deep Neural Network), Transformer, or SVM (Support Vector Machine) to determine (infer) the motion type of the moving object 1 from each vector included in the optical flow. (Example 2 of determining the motion type of a moving object)

[0045] As Example 2, the determination unit 12 may determine the motion type of the moving object 1 based on the pre-registered optical flow model for each motion type. In this case, the determination unit 12 may determine the motion type of the moving object 1 based on the similarity between each vector included in each model and each vector included in the calculated optical flow. In this case, the determination unit 12 may determine the motion type of the moving object 1 as the motion type of the model having the highest similarity to the calculated optical flow.

[0046] (Example 3 of determining the motion type of a moving object)

[0047] As Example 3, the determination unit 12 may determine the motion type of the moving object 1 based on, for example, an acceleration sensor mounted on the moving object 1 or operation information such as a handle obtained from an ECU of the moving object 1 .

[0048] Subsequently, the estimation unit 13 specifies the time interval between the photographing times of two images for estimating the amount and direction of movement of the moving object 1 based on the movement type of the moving object 1 (step S103 ).

[0049] Here, the estimation unit 13 may, for example, determine that the time interval is longer as the curvature of the moving direction of the moving object 1 is smaller (the curve is gentler). In this case, the estimation unit 13 may set the time interval to a first time interval (e.g., 1 / 60 seconds) in the case of the aforementioned “sharp curve”, and set the time interval to a second time interval (e.g., 1 / 10 seconds) longer than the first time interval in the case of the aforementioned “gentle curve”.

[0050] In the case of calculating the optical flow, a flow vector that is not related to the motion of the moving object 1 due to the motion or misalignment of the object, etc., may be detected. When each calculated flow vector is relatively large, it becomes relatively easy to remove the flow vector that is not related to the motion of the moving object 1 as noise. This is because, for example, the flow vector that is not related to the motion of the moving object 1 is relatively significantly different in size and direction from other nearby flow vectors in the image.

[0051] When the moving object 1 is moving, by determining a longer time interval, the magnitude of each calculated flow vector can be made relatively large. Therefore, it becomes relatively easy to remove the flow vectors that are not related to the movement of the moving object 1 as noise. Therefore, the movement amount and movement direction of the moving object can be appropriately estimated.

[0052] In addition, the estimation unit 13 may decide a longer time interval, for example, the slower the moving speed of the moving object 1. By doing so, similarly, it becomes relatively easy to remove as noise the flow vector that is irrelevant to the movement of the moving object 1. Therefore, the movement amount and movement direction of the moving object can be appropriately estimated.

[0053] In addition, the estimation unit 13 may determine a longer time interval, for example, the smaller the gradient change of the moving direction of the moving object 1. By doing so, similarly, it becomes relatively easy to remove the flow vector that is irrelevant to the movement of the moving object 1 as noise. Therefore, the movement amount and the moving direction of the moving object can be appropriately estimated.

[0054] Subsequently, the estimation unit 13 estimates the amount of movement and the direction of movement of the moving object 1 from the first time point at which the first image is taken to the second time point at which the second image is taken based on the first image and the second image taken at the time interval (step S104). By doing so, for example, the position of the moving object 1 after the movement can be estimated. The estimation unit 13 can output the estimation result to an ECU for autonomous driving, etc. By doing so, for example, visual SLAM (Simultaneous Localization and Mapping) can be implemented, which creates an environmental map and performs self-position estimation on the environmental map by sensing the surrounding environment while the moving object 1 is traveling.

[0055] Here, the acquisition unit 11 may acquire each image photographed by the photographing device 20 at a specific frame rate. And the estimation unit 13 may extract the first image and the second image from the image acquired by the acquisition unit 11. By doing so, while keeping the frame rate photographed by the photographing device 20 constant, the position of the moving object 1 after the movement, etc. can be estimated.

[0056] In the case of realizing autonomous driving or driving assistance of a moving object, adding sensors other than a camera increases the cost of the system. Moreover, the complexity of processing increases, resulting in an increase in the amount of software development, and also increases the processing delay in the data pipeline. On the other hand, according to the present disclosure, the amount and direction of motion of a moving object can be appropriately estimated without using sensors other than a camera.

[0057] (Modified embodiment)

[0058] The information processing device 10 may be a device contained in a single housing, but the information processing device 10 of the present disclosure is not limited thereto. For example, each part of the information processing device 10 may be implemented by cloud computing composed of one or more computers. In addition, the information processing device 10 and the photographic device 20 may be housed in the same housing and configured as an integrated information processing device. In addition, at least a portion of the processing of each functional part of the information processing device 10 may be performed by the photographic device 20. Such an information processing device 10 is also included as an example of the “information processing device” of the present disclosure.

[0059] As described above, the present disclosure has been described with reference to the embodiments, but the present disclosure is not limited to the described embodiments. As will be appreciated by those skilled in the art, various changes may be made to the configuration and details of the present disclosure within the scope of the present disclosure. And each embodiment may be appropriately combined with other embodiments.

Claims

1. An information processing device, comprising: an acquisition unit, configured to acquire each image captured at each time point by a shooting device mounted on the moving object; a determining unit, configured to determine a motion type of the moving object based on each of the images; as well as An estimation unit is used to estimate the movement amount and movement direction of the moving object from a first time point when the first image is captured to a second time point when the second image is captured based on a first image and a second image captured at a time interval corresponding to the movement type from the first image.

2. The information processing device according to claim 1, The motion type includes at least one of the following: a motion speed of the motion object, a curvature of the motion direction of the motion object, and a gradient change of the motion direction of the motion object.

3. The information processing device according to claim 1, The estimation unit determines that the time interval is longer as the moving speed of the moving object is slower.

4. The information processing device according to claim 1, The estimating unit determines that the time interval is longer as the curvature of the moving direction of the moving object is smaller.

5. The information processing device according to claim 1, The estimation unit determines that the time interval is longer as the gradient change of the motion direction of the moving object is smaller.

6. The information processing device according to claim 1, The determining unit calculates an optical flow based on each of the images, and determines the motion type based on the optical flow.

7. The information processing device according to claim 1, The acquisition unit acquires each of the images captured by the capturing device at a specific frame rate, and the estimation unit extracts the first image and the second image from the images.

8. An information processing method performed by an information processing device, comprising: acquiring each image captured at each time point by a capturing device mounted on the moving object; determining the motion type of the moving object based on each of the images; as well as Based on a first image and a second image captured at a time interval corresponding to the type of motion from the first image, the amount and direction of motion of the moving object from a first point in time when the first image is captured to a second point in time when the second image is captured are estimated.

9. A program for causing a computer to perform the following processing: acquiring each image captured at each time point by a capturing device mounted on a moving object, determining the type of movement of the moving object based on each of the images, and estimating the amount of movement and direction of movement of the moving object from a first time point when the first image is captured to a second time point when the second image is captured based on a first image and a second image captured at a time interval corresponding to the type of movement from the first image.

Images