Information processing device, information processing method, and program for estimating the amount of movement of a moving object

Abstract

To estimate behavior of a moving body in a simplified and redundant method.SOLUTION: Provided is an information processing device for estimating a moving amount of a mounted moving body, including: photographing means for obtaining photographed images in which an external world of the moving body is captured; and estimation means for estimating the moving amount of the moving body based on a first photographed image captured at a first clock time and a second photographed image captured at a second clock time prior to the first clock time. Here, the estimation means estimates the moving amount of the moving body, based on a difference between each image information of a prescribed area in an altered image which is the second photographed image altered based on the predicted moving amounts, each of which differs one another, and image information obtained from curving out the prescribed area of the first photographed image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program for estimating a movement amount of a moving object. [Background technology]

[0002] In recent years, drive recorders that are installed inside a vehicle (also called a moving body) to capture images of the front and rear of the vehicle and record the state of the vehicle at the time of and immediately before an accident have become widespread. A drive recorder that not only records images of the accident at the time of the accident at a recording medium, but also detects a dangerous state that did not result in an accident, such as erratic driving, and records images using the detection of the state as a trigger (Patent Document 1). In Patent Document 1, the vehicle's driving position is detected using the position of the white line on the road recognized from the image information in front of the vehicle, and erratic driving is detected based on the detected vehicle position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-150573 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, some drive recorders are installed separately in a vehicle after purchase and operate independently without communicating with the vehicle's system. In order to provide a driving assistance function in such a drive recorder that notifies the occupants of a specific vehicle behavior, a technology that estimates the vehicle behavior based on captured image information is useful.

[0005] In order to realize driving assistance functions using drive recorders, there is a demand for technology that detects the behavior (e.g., amount of movement) of a moving object in a simple and redundant manner that does not assume the presence of specific objects (e.g., white lines) on the road.

[0006] The present invention has been made in view of the above problems, and has an object to realize a technology capable of estimating the behavior of a moving object using a simple and redundant method. [Means for solving the problem]

[0007] According to the present invention, An information processing device for estimating a movement amount of a moving object to which the information processing device is attached, An imaging means for acquiring an image of an external environment of the moving body; an estimation means for estimating a movement amount of the moving object based on a first captured image captured at a first time and a second captured image captured at a second time prior to the first time, An information processing device is provided, characterized in that the estimation means estimates the amount of movement of the moving body based on the difference between each piece of image information of a specified area of ​​a changed image obtained by changing the second captured image based on mutually different predicted movement amounts and image information obtained by cutting out a specified area of ​​the first captured image. Effect of the Invention

[0008] According to the present invention, it becomes possible to estimate the behavior of a moving object using a simple and redundant method. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a schematic configuration example of a moving body (vehicle) according to a first embodiment of the present invention; [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of a drive recorder as an example of an information processing device according to a first embodiment; [Diagram 3] FIG. 1 is a diagram for explaining a process for determining the amount of movement (stop state) of a moving body (vehicle) according to the first embodiment. [Figure 4] FIG. 1 is a diagram for explaining a process for estimating a moving amount (speed level) of a moving body (vehicle) according to the first embodiment; [Diagram 5] FIG. 1 is a diagram for explaining a process for estimating a movement amount (amount of movement in a horizontal direction) of a moving body (vehicle) according to the first embodiment. [Figure 6] 1 is a flowchart showing a series of operations in a movement amount estimation process according to the first embodiment. [Figure 7] 11 is a flowchart showing a series of operations in a movement amount estimation process according to the second embodiment. [Figure 8] 11 is a flowchart showing a series of operations in a movement amount estimation process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Embodiment 1) This embodiment relates to an information processing device for detecting the amount of movement of a moving body to which it is attached. In the following embodiment, the information processing device will be described as an example in which it is a drive recorder attached to the moving body. However, the information processing device may be any other device as long as it is attached to the moving body and can capture an image of the outside world of the moving body. In addition, in the following embodiment, the information processing device will be described as an example in which the moving body is, for example, a vehicle. The vehicle is typically a four-wheeled vehicle, but this embodiment may be applied to other types of vehicles.

[0012] An example of the mounting position of a drive recorder will be described with reference to Fig. 1. A drive recorder 101 is mounted on a vehicle 100. The drive recorder 101 is mounted near a rear-view mirror or on the rear-view mirror of the vehicle 100 at a position on the windshield of the vehicle 100 that does not obstruct the driver's field of vision. The mounting position of the drive recorder 101 is not limited to this, and may be any position that can capture an image of the outside world of the vehicle 100.

[0013] The drive recorder 101 can operate as a standalone device, but may also receive power from the vehicle 100.

[0014] <Example of functional configuration of a drive recorder> An example of the functional configuration of the drive recorder 101 will be described with reference to Fig. 2. The drive recorder 101 includes, for example, a control unit 201, a storage device 202, an imaging device 203, an input device 204, an output device 205, and a communication device 206.

[0015] The control unit 201 includes, for example, a CPU (Central Processing Unit) and controls the overall operation of the drive recorder 101. The control unit 201 may be realized, for example, by the CPU expanding and executing a program stored in the storage device 202. In addition to the CPU, the control unit 201 may further include a GPU (Graphics Processing Unit) and dedicated circuits for image processing, machine learning, and the like.

[0016] The storage device 202 is a device that stores data and programs related to the operation of the drive recorder 101. For example, the storage device 202 may store a program that defines the operation of the drive recorder 101, temporary data during execution of the program, images captured by the imaging device 203, environmental sounds collected by a microphone, and the like. The storage device 202 may be realized by a memory such as a random access memory (RAM) or a read only memory (ROM). Furthermore, the storage device 202 may include an auxiliary storage device such as a solid state drive (SSD).

[0017] The imaging device 203 includes a camera for capturing an image of the outside world (e.g., scenery) in front of the vehicle 100. The imaging device 203 may be, for example, a wide-angle camera. This camera is located in front of the drive recorder 101 (on the front side of the vehicle 100). The imaging device 203 may further include a camera capable of capturing images of the rear and left and right sides of the vehicle 100 in addition to the camera capturing images of the front side of the vehicle 100. In this case, the camera capturing images of the rear may be disposed on the rear side of the drive recorder (on the rear side of the vehicle), and the camera capturing images of the sides may be disposed on the side of the drive recorder. Some of the cameras capturing images in multiple directions may be separate from the drive recorder. Alternatively, the imaging device 203 may include, for example, a fisheye camera.

[0018] The input device 204 is a device for inputting instructions from a passenger of the vehicle 100. The input device 204 includes, for example, a button and a touch panel. The input device 204 may also include a microphone for inputting the voice of the passenger and sounds from the outside world.

[0019] The output device 205 is a device for outputting information to a passenger of the vehicle 100. The output device 205 may be a display device (e.g., a liquid crystal display or an indicator) that provides visual information, or may be an audio device (e.g., a speaker) that provides auditory information, or may include both of these.

[0020] The communication device 206 is a device for the drive recorder 101 to communicate with other devices (e.g., a user's smartphone, not shown). The communication device 206 may, for example, transmit images and sounds stored in the storage device 202 to the user's smartphone. The communication device 206 may communicate with other devices via a wired connection, or may communicate with other devices wirelessly. The wireless communication may be communication conforming to a protocol such as short-range communication (e.g., Bluetooth (registered trademark)). The communication device 206 may be configured to be able to communicate with the control ECU of the vehicle 100 via a wired connection or wirelessly. The communication device 206 may be optional.

[0021] <Outline of vehicle movement distance estimation process in drive recorder> Next, a process for estimating the amount of movement of the vehicle to which the drive recorder 101 is attached will be described with reference to Fig. 3 to Fig. 5. Note that the process described with reference to Fig. 3 to Fig. 5 will be described with reference to an example in which the process is performed using an image captured by the imaging device 203 capturing an image of the area ahead of the vehicle 100. The following example is realized, for example, by the control unit 201 executing a program stored in the storage device 202.

[0022] First, the process of determining the amount of movement (stopped state) of the vehicle will be described with reference to Fig. 3. The drive recorder 101 acquires captured images, for example, at a predetermined time interval, by the imaging device 203. The control unit 201 acquires, for example, a captured image 301 captured at a certain time (for example, the current time; this time is k) and a captured image 302 captured at a time (for example, k-1) prior to the time k, calculates the difference in pixel values ​​between the captured images 301 and 302, and generates a difference image.

[0023] The control unit 201 binarizes the obtained difference image to generate a binarized image 303. Binarization is a process of converting an image into two gradations of black and white, and binarizing the difference image can make the difference image clearer. An appropriate value may be set in advance through experiments or the like as a threshold value used for binarization. Note that, in this embodiment, an example is described in which the difference image is binarized, but other image processing may be performed instead of binarizing the difference image. Furthermore, the control unit 201 may convert each image into a luminance image or perform predetermined image processing such as emphasizing edges before calculating the difference.

[0024] The control unit 201 calculates the sum of pixel values ​​using pixel values ​​of the binarized image, and judges whether the calculated sum is equal to or less than a predetermined threshold. When the control unit 201 judges that the calculated sum is equal to or less than the predetermined threshold, it judges that the vehicle is in a stopped state. At this time, the control unit 201 sets, for example, a predetermined vehicle state parameter (for example, Vstate) to "0" indicating a vehicle stopped state. On the other hand, when the control unit 201 judges that the calculated sum is greater than the predetermined threshold, it judges that the vehicle is in a traveling state or the movement amount of surrounding traffic participants is large. At this time, the control unit 201 sets, for example, the vehicle state parameter (Vstate) to "1" indicating that the vehicle state is a traveling vehicle or the movement amount of traffic participants is large.

[0025] That is, the control unit 201 calculates the difference between the image captured at time k and the image captured previously, and if there are many white pixel values ​​in the difference image, it determines that the similarity between the two images is high and that the vehicle is stopped. However, depending on how the pixel color values ​​are taken as positive or negative, the color indicating high similarity does not have to be white. In this embodiment, an example will be described in which it is determined that the similarity between the two images is high when there are many white pixel values ​​in the difference image, but black pixel values ​​may appear in the difference image when the similarity is high.

[0026] Next, a process of estimating the amount of movement (speed level) of a vehicle will be described with reference to FIG. 4. The speed level refers to the moving speed of the vehicle. However, it may also represent a degree of speed (e.g., high speed, medium speed, low speed, etc.) previously associated with the moving speed. The drive recorder 101 acquires captured images, for example, at a predetermined time interval, by the imaging device 203. For example, the control unit 201 acquires a captured image 401 captured at a certain time (e.g., k) and a captured image captured at a time (e.g., k-1) prior to the time k. The control unit 201 generates a scaled image by enlarging or reducing the captured image captured at the time k-1 at a plurality of magnifications different from each other. For example, the captured image captured at the time k-1 is scaled at a magnification factor {E n , E m , ,1, ,R m , R n}(where E n >E m >···>1>···>R m >R n >0) (for example, scaled images 411-415). Then, the control unit 201 cuts out a predetermined area 421 from each of the generated scaled images 411-415 and the captured image 401. At this time, each of the cut-out predetermined areas has the same size and has the same center as the center of the captured images captured at times k and k-1. n , E m , ,1, ,R m , R n Each of the scale factors} is associated with a speed level at which the vehicle is moving forward or backward, such that an image scaled with a certain scale factor will resemble an image captured at the next timing (i.e., time k) when the vehicle is traveling at the associated speed.

[0027] The control unit 201 calculates the difference between the pixel values ​​of a predetermined region cut out from the captured image 301 at time k and the pixel values ​​of the predetermined regions of each of the scaled images 411-415 obtained by scaling the captured image 302, and generates a plurality of difference images.

[0028] The control unit 201 generates a binary image by binarizing each of the obtained difference images, and calculates the sum of pixel values ​​of the binary image. The control unit 201 selects the magnification ratio that minimizes the sum of pixel values ​​from the sums of pixel values ​​corresponding to each magnification ratio. For example, n , E m , 1, R m , R n The image captured at time k-1 is scaled, and among them, E m When the sum of pixel values ​​using the scaled image scaled by E becomes the smallest, the control unit 201 m As described above, each scaling factor is associated with the vehicle speed level, so the scaling factor E m The speed level associated with is estimated as the speed level (movement amount) of the vehicle 100.

[0029] The binarization may be the same as the binarization process in the process described with reference to Fig. 3. In this embodiment, the case where the difference image is binarized will be described as an example, but other image processing may be performed instead of binarizing the difference image. Furthermore, the control unit 201 may convert each image into a luminance image or perform predetermined image processing such as emphasizing edges before calculating the difference.

[0030] Further, the process of estimating the movement amount (movement amount in the horizontal direction) of the vehicle will be described with reference to FIG. 5. As described above, the drive recorder 101 obtains captured images, for example, at a predetermined time interval, by the imaging device 203. The control unit 201 obtains, for example, a captured image 501 captured at a certain time (for example, k) and a captured image captured at a time (for example, k-1) prior to the time k. The control unit 201 cuts out a predetermined area, which is shifted in the horizontal direction from the center of the image by a plurality of shift amounts which are different from each other, from the captured image captured at the time k-1. For example, the control unit 201 cuts out a predetermined area, which is shifted in the horizontal direction by a plurality of shift amounts which are different from each other, from the captured image captured at the time k-1. n , -S m , ,0, ,S m , S n}(where S n >S m The control unit 201 cuts out a predetermined area (for example, predetermined areas 521 to 525) shifted by the shift amount {-S n , -S m , ,0, ,S m , S n Each of the shift amounts is associated with the amount of movement of the vehicle to the left or right, so that the area cut out at a certain shift amount resembles a predetermined area cut out from the center of an image captured when the vehicle moves by the associated amount of movement.

[0031] The control unit 201 calculates the difference between the pixel values ​​of a predetermined area 520 cut out from a captured image 501 at time k and the pixel values ​​of each of predetermined areas 521 to 525 cut out from a captured image 511, and generates a plurality of difference images.

[0032] The control unit 201 generates a binary image by binarizing each of the obtained difference images, and calculates the sum of pixel values ​​of the binary image. The control unit 201 selects the shift amount that minimizes the sum of pixel values ​​from the sums of pixel values ​​corresponding to each shift amount. For example, in the image captured at time k-1, -S n , -S m , 0, S m , S n Within the specified area shifted by the shift amount of S m When the sum of pixel values ​​using the predetermined area shifted by the shift amount S becomes the smallest, the control unit 201 m As described above, each shift amount is associated with the vehicle movement amount, so the shift amount S m is estimated as the horizontal movement of the vehicle 100.

[0033] The binarization may be the same as the binarization process in the process described with reference to Fig. 3. In this embodiment, the case where the difference image is binarized will be described as an example, but other image processing may be performed instead of binarizing the difference image. Furthermore, the control unit 201 may convert each image into a luminance image or perform predetermined image processing such as emphasizing edges before calculating the difference.

[0034] Next, a series of operations of the movement amount estimation process in the drive recorder 101 will be described with reference to Fig. 6. This process is realized, for example, by the control unit 201 of the drive recorder 101 executing a program stored in the storage device 202. This process may be started at any timing while the vehicle 100 is traveling, and the processes of S601 to S608 described later may be executed repeatedly.

[0035] In S601, the control unit 201 determines the vehicle state. Specifically, the control unit 201 executes the process of determining the amount of movement (stopped state) of the vehicle, which has been described with reference to FIG. 3. As described above, the control unit 201 obtains, for example, a captured image captured at a certain time k (which may be the present) and a captured image captured at a time before the time k (for example, k-1), and obtains a difference between the captured images. The control unit 201 calculates the sum of pixel values ​​using a binarized image of the obtained difference, and determines whether the vehicle is in a stopped state or in another state (the vehicle is traveling or the amount of movement of surrounding traffic participants is large) according to the calculated sum. When the control unit 201 determines that the vehicle is in a stationary state, for example, it sets a vehicle state parameter to "0" indicating a vehicle stopped state, and when it is not, it sets the vehicle state parameter to "1".

[0036] In S602, the control unit 201 determines whether the vehicle state is a stopped state. For example, the control unit 201 refers to the value of the vehicle state parameter set in S601 and determines whether the vehicle state parameter is set to 0. If it is determined that the vehicle state is a stopped state, this series of operations is terminated. On the other hand, if the control unit 201 determines that the vehicle state is not a stopped state, the process proceeds to S603.

[0037] In S603, the control unit 201 estimates the speed level of the vehicle. Specifically, the control unit 201 executes the process of estimating the movement amount (speed level) of the vehicle, which has been described with reference to Fig. 4. As described above, the control unit 201 acquires, for example, a captured image captured at a certain time k and a captured image captured at a time (for example, k-1) prior to the time k. The two captured images may be captured images acquired in the process of S601.

[0038] The control unit 201 generates a plurality of scaled images by scaling the captured image captured at time k-1 with gradually different scale factors, and cuts out a predetermined area (also called a predetermined area of ​​the scaled image) having the same center from each scaled image. The control unit 201 also cuts out a predetermined area of ​​the same size from the captured image at time k. The control unit 201 generates a difference image between the predetermined area cut out from the captured image at time k and each of the predetermined areas cut out from the scaled image, and calculates the sum of pixel values ​​of the binarized difference image. The control unit 201 specifies a scale factor that minimizes the sum of pixel values ​​of each binarized difference image. As described above, the specified scale factor is associated with the speed level when the vehicle moves forward or backward, so the control unit 201 can estimate the speed level of the vehicle from the specified scale factor. The control unit 201 acquires the speed level corresponding to the scale factor, for example, by referring to a speed level table prestored in the storage device 202.

[0039] In S604, the control unit 201 determines whether the obtained speed level is equal to or greater than a predetermined speed threshold. The predetermined speed threshold is, for example, a threshold for determining whether or not to perform a process for estimating the horizontal movement amount of the vehicle. When the control unit 201 determines that the speed level of the vehicle is equal to or greater than the predetermined speed threshold, the control unit 201 advances the process to S607 without performing a process for estimating the horizontal movement amount of the vehicle. On the other hand, when the control unit 201 determines that the speed level of the vehicle is smaller than the predetermined speed threshold, the control unit 201 advances the process to S606, and executes a process for estimating the horizontal movement amount of the vehicle.

[0040] 6, the speed level estimation process and the horizontal movement amount estimation process are separate processes, and an example is described in which the horizontal movement amount estimation process is performed when the speed level is equal to or less than a predetermined speed threshold. However, the speed level estimation and the horizontal movement amount estimation may be performed simultaneously.

[0041] For example, the control unit 201 first changes the image captured at time k−1 by gradually changing the magnification ratio {E n , E m , ,1, ,Rm , R n}(where E n >E m >···>1>···>R m >R n >0) to generate a scaled image. Then, the control unit 201 further calculates from each of the generated scaled images, -S n , -S m , 0, S m , S n The control unit 201 extracts a predetermined area shifted by a shift amount of k. The control unit 201 specifies a combination of a magnification ratio and a shift amount that minimizes the difference in pixel value between the predetermined area extracted based on the image captured at time k-1 and the predetermined area in the image captured at time k. In this way, it is possible to estimate the amount of movement when the vehicle 100 moves forward or backward in the horizontal direction at a speed equal to or greater than a certain speed.

[0042] In S606, the control unit 201 estimates the horizontal movement amount of the vehicle. Specifically, the control unit 201 executes the process of estimating the movement amount (horizontal movement amount) of the vehicle, which has been described with reference to Fig. 5. As described above, the control unit 201 acquires, for example, a captured image captured at a certain time k and a captured image captured at a time (for example, k-1) before the time k. The two captured images may be the captured images acquired in the process of S601.

[0043] The control unit 201 cuts out a predetermined area (also called a predetermined area of ​​a shifted image) that is shifted in the horizontal direction by a shift amount that differs stepwise from the captured image captured at time k-1. The control unit 201 also cuts out a predetermined area whose position is not shifted from the captured image captured at time k. The control unit 201 generates a difference image between the predetermined area cut out from the captured image captured at time k and the predetermined area cut out based on the captured image captured at time k-1, and calculates the sum of pixel values ​​of the binarized difference image. The control unit 201 specifies a shift amount that minimizes the sum of pixel values ​​of each binarized difference image. As described above, the specified shift amount is associated with the movement amount when the vehicle moves in the horizontal direction, so the control unit 201 can estimate the horizontal movement amount of the vehicle from the specified shift amount. The control unit 201 acquires the horizontal movement amount corresponding to the shift amount, for example, by referring to a lateral movement amount table pre-stored in the storage device 202.

[0044] In S607, the control unit 201 determines whether the speed level or the horizontal movement amount of the vehicle satisfies the notification condition. The notification condition may include, for example, a condition that the horizontal movement amount per predetermined unit time is equal to or greater than a threshold, or that the speed level of the vehicle is equal to or greater than a predetermined speed indicating excessive speed. For example, when the horizontal movement amount per predetermined unit time is equal to or greater than a threshold, the occupant can be notified that the vehicle is shaking significantly. Also, when the speed level of the vehicle is equal to or greater than a speed indicating excessive speed, the occupant can be notified that the vehicle is speeding. The speed indicating excessive speed may be set to a plurality of thresholds that are gradually different from each other, and a notification that is gradually different from each other may be made depending on the speed level.

[0045] The control unit 201 determines whether at least one of the notification conditions is satisfied based on the speed level estimated in S603 and the horizontal movement amount estimated in S606. If the control unit 201 determines that at least one of the notification conditions is satisfied, the process proceeds to S608, and otherwise the control unit 201 ends this series of operations.

[0046] In S608, the control unit 201 generates a notification via the output device 205 based on the determined notification condition. For example, the control unit 201 may output a voice (e.g., "speed exceeds XX km / s") or light an indicator corresponding to the notification condition determined to be satisfied in S606. If it is determined that a plurality of notification conditions are satisfied, the control unit 201 may refer to a priority order table in which the priorities of notification conditions are set in advance, and output a notification corresponding to the highest priority notification condition (or some of the highest priority notification conditions). When the control unit 201 finishes outputting the notification, it ends this series of operations.

[0047] As described above, in this embodiment, in order to estimate the movement amount of the attached vehicle, a first captured image captured at a first time of the vehicle's outside and a second captured image captured at a second time before the first time are used. The control unit 201 estimates the movement amount (speed level or horizontal movement amount) of the vehicle from the minimum difference between each of the image information of a predetermined area (predetermined area of ​​the changed image) of the image in which the second captured image is changed based on different predicted movement amounts (for example, speed level or horizontal movement amount) and the image information of the predetermined area cut out from the first captured image. In this way, since it is not premised on the existence of a specific object (for example, a white line) on the road, it is possible to estimate the movement amount of the vehicle even under conditions in which the white line is not properly captured, that is, under redundant conditions. Also, by simplifying the process in this way, it is possible to execute the process at high speed even with the computational resources of a drive recorder that operates alone.

[0048] (Embodiment 2) Next, a second embodiment will be described. In the above embodiment, the image capturing device 203 performs processing using an image captured in front of the vehicle 100. However, when another vehicle is traveling ahead of the lane in which the vehicle 100 is traveling, the vehicle area may not be appropriately estimated in speed level due to magnification change even though the vehicle area does not change as much as the background. If an occupant is notified using an inaccurate estimated speed level, the usability of the drive recorder may be reduced. For this reason, in this embodiment, the speed level is further estimated from the captured image captured in the rear and the captured image captured in the side, and the reliability of the speed level estimated from the captured image in front is determined. In this way, when the reliability of the estimated speed level is high, the speed level can be used for notification processing. This improves the accuracy of notification to the occupant.

[0049] The configuration of the drive recorder 101 according to this embodiment may be the same as that of the above-mentioned embodiment, but the movement amount estimation process is partially different. Therefore, the same configurations and processes are given the same reference numbers and the description is omitted, and the difference is mainly described.

[0050] A series of operations of the movement amount estimation process in the drive recorder 101 according to this embodiment will be described with reference to Fig. 7. Note that this process is realized, for example, by the control unit 201 of the drive recorder 101 executing a program stored in the storage device 202, as in the first embodiment.

[0051] The control unit 201 executes the processes of S601 and S602 in the same manner as in the first embodiment, and then in S701, estimates the speed level from the captured images of the front, rear, and side. The speed level of the front is estimated by the process of S603 described above.

[0052] For the rear, the control unit 201 acquires two captured images at the same time as the front, for example, an image captured at a certain time k and an image captured at a time (k-1) before the time k. However, when capturing an image of the outside world behind the vehicle from the drive recorder 101, the interior and window areas are captured, so other areas (such as the interior and people areas) may be masked so as to use the image of the outside world area. For example, it is sufficient to mask areas in which the variance of change is lower than a predetermined value in captured images captured sequentially over several seconds.

[0053] For the rearward captured image, the control unit 201 generates a plurality of magnified images by magnifying the captured image captured at time k-1 with gradually different magnification ratios, and cuts out a predetermined area having the same center from each of the magnified images. The control unit 201 also cuts out a predetermined area of ​​the same size from the captured image at time k. The control unit 201 generates a difference image between the predetermined area cut out from the captured image at time k and the predetermined area cut out from the magnified image, and calculates the sum of pixel values ​​of the binarized difference image. The control unit 201 specifies a magnification ratio that minimizes the sum from the sum of pixel values ​​of each binarized difference image. The magnification ratio specified from the rearward image is associated with the speed level when the vehicle moves forward or backward. However, the magnification ratio specified from the frontward image is opposite in terms of enlargement or reduction. This allows the control unit 201 to estimate the speed level of the vehicle from the magnification ratio specified from the rearward image. For convenience, the speed level estimated from the rearward image is called the rearward speed level. The control unit 201 acquires the rear speed level corresponding to the magnification ratio by, for example, referring to a table of speed levels for rear images stored in advance in the storage device 202 .

[0054] Furthermore, for the side, the control unit 201 acquires two captured images at the same time as the front, for example, a captured image captured at a certain time k and a captured image captured at a time (k-1) before the time k. However, when capturing an image of the outside world to the side of the vehicle from the drive recorder 101, the interior and window areas are also captured, so other areas may be masked to use the image of the outside world area. For example, it is sufficient to mask areas in which the variance of change is lower than a predetermined value in captured images captured sequentially over several seconds.

[0055] When estimating the speed level of the vehicle from the side captured image, the control unit 201 performs a process similar to the process of estimating the horizontal movement amount. That is, the control unit 201 cuts out a predetermined area that is shifted in the horizontal direction by a stepwise different shift amount from the captured image captured at time k-1. The control unit 201 also cuts out a predetermined area whose position is not shifted from the captured image at time k. The control unit 201 generates a difference image between the predetermined area cut out from the captured image at time k and the predetermined area cut out based on the captured image at time k-1, and calculates the sum of pixel values ​​of the binarized difference image. The control unit 201 specifies the shift amount that minimizes the sum of pixel values ​​of each binarized difference image from the sum of pixel values ​​of each binarized difference image. The shift amount specified from the side image is associated with the speed level of the vehicle. Therefore, the control unit 201 can estimate the speed level of the vehicle from the shift amount specified from the side image. For convenience, the speed level estimated from the side image is called the side speed level. The control unit 201 estimates the lateral speed levels of the right side and the left side of the vehicle, respectively. The control unit 201 acquires the lateral speed level corresponding to the shift amount identified from the lateral image, for example, by referring to a table of speed levels for the right side and left side images stored in advance in the storage device 202.

[0056] In S702, the control unit 201 determines whether the reliability of the speed level estimated from the forward image is equal to or higher than a threshold. For example, the control unit 201 calculates the difference between the speed level estimated from the forward image and other speed levels, and then calculates the sum of the differences between the two speed levels that are closest to each other (for example, the right lateral speed level and the left lateral speed level), and sets the calculated sum as the reliability of the forward speed level. In this case, for example, even if the speed level in one direction is an outlier (for example, the estimated rear speed level due to the presence of a rear vehicle), the reliability of the forward speed level can be calculated more reliably using the speed levels in the other directions. The control unit 201 determines whether the reliability of the forward speed level is equal to or higher than a threshold. below If it is determined that this is the case, the process proceeds to S703, and if not, this series of processes is terminated (it is determined that the current forward speed level cannot be used for notification).

[0057] In this embodiment, the reliability of the forward speed level is determined, but the speed for which the reliability is determined may be a speed level estimated from an image captured in another direction. For example, when the brightness of the forward image is higher than a predetermined value (when the forward image is captured in backlight due to the sun being in the traveling direction of the vehicle), the reliability of the speed level estimated from the rear image or the side image may be determined, and the speed level may be used in subsequent processing.

[0058] In S703, the control unit 201 determines whether the estimated forward speed level is equal to or greater than a predetermined speed threshold. If the control unit 201 determines that the forward speed level is equal to or greater than the predetermined speed threshold, the control unit 201 advances the process to S607 without performing a process of estimating the horizontal movement amount of the vehicle. On the other hand, if the control unit 201 determines that the forward speed level is smaller than the predetermined speed threshold, the control unit 201 advances the process to S606, where the control unit 201 performs a process of estimating the horizontal movement amount of the vehicle.

[0059] Thereafter, the control unit 201 executes the processes of S606 to S608 in the same manner as in the first embodiment, and ends this series of operations.

[0060] As described above, in this embodiment, four speed levels are estimated separately using the captured images of the front, rear, and sides, and the reliability of the speed level in one direction is calculated based on the speed levels in the other directions. In this way, the speed level can be calculated in a simple manner, and the use of the speed level in a direction with low reliability can be eliminated. This makes it possible to realize a notification using the speed level when the reliability is high.

[0061] (Embodiment 3) Next, a third embodiment will be described. In the second embodiment, the speed level is estimated from the captured image of the rear and the captured image of the side, and then the reliability of the speed level estimated from the captured image of the front is determined, and a speed level with low reliability is not adopted. In contrast, in the third embodiment, an object recognition process is executed to recognize a vehicle in the captured images in each direction, and the speed level is estimated based on the captured images in which the vehicle is not recognized.

[0062] In this embodiment, a case where a vehicle is recognized in a captured image is described as an example, but the recognition target is not limited to a vehicle and may include other moving objects. Also, the configuration of the drive recorder 101 according to this embodiment may be the same as that of the above-mentioned embodiment, but the movement amount estimation process is partially different. For this reason, the same reference numbers are used for the same configuration and process, and the description is omitted, and the differences are mainly described.

[0063] A series of operations of the movement amount estimation process in the drive recorder 101 according to this embodiment will be described with reference to Fig. 8. Note that this process is realized, for example, by the control unit 201 of the drive recorder 101 executing a program stored in the storage device 202, as in the above-described embodiment.

[0064] The control unit 201 executes the processes of S601 and S602 in the same manner as in the first embodiment, and then in S801, executes a known image recognition process on the captured images of the front, rear, and side to detect the vehicle in each captured image. This process is performed, for example, to eliminate the influence of other vehicles traveling in front or rear of the vehicle 100 and other vehicles traveling parallel to the vehicle 100. The recognition of the vehicle in each captured image may take into consideration the direction of the vehicle in the image. For example, image recognition may be performed in consideration of the rear of the vehicle in the image in the case of a captured image of the front, the front of the vehicle in the image in the case of a captured image of the rear, and the side of the vehicle in the image in the case of a captured image of the side.

[0065] In S802, the control unit 201 determines whether there is an image in which a vehicle was recognized in S801. If the vehicle was not recognized in any of the images, the control unit 201 advances the process to S804. On the other hand, if the vehicle was recognized in any of the images, the control unit 201 advances the process to S803. The speed level ahead is estimated by the process of S603 described above. Note that the control unit 201 may end this series of operations when the vehicle is recognized in all captured images.

[0066] In S803, the control unit 201 estimates the speed level using a captured image in which a vehicle is not recognized among the front image, the rear image, and the side image. The estimation of the speed level based on each image is similar to the process described above in S701.

[0067] In S804, the control unit 201 estimates the forward speed level using the forward image. The estimation of the forward speed level is the same as the process described above in S603. Note that, in this process, the process of estimating the forward speed level is performed as an example, but the speed levels in the rear and sides may be further estimated, and the reliability of the forward speed level may be obtained as in S702. In this case, if the reliability of the forward speed level is low, this series of operations may be terminated.

[0068] In S805, the control unit 201 uses the speed level estimated in S803 or S804 to determine whether the speed level is equal to or greater than a predetermined speed threshold. If the control unit 201 determines that the estimated speed level is equal to or greater than the predetermined speed threshold, the control unit 201 advances the process to S607 without performing a process of estimating the horizontal movement amount of the vehicle. On the other hand, if the control unit 201 determines that the estimated speed level is smaller than the predetermined speed threshold, the control unit 201 advances the process to S606 and executes a process of estimating the horizontal movement amount of the vehicle.

[0069] Thereafter, the control unit 201 executes the processes of S606 to S608 in the same manner as in the first embodiment, and ends this series of operations.

[0070] As described above, in this embodiment, when estimating four speed levels separately using the captured images of the front, rear, and sides, the captured images in which the vehicle is recognized by image recognition are not used for estimating the speed level. In this way, the speed level can be calculated in a simple manner, and the accuracy of the estimated speed level can be reduced due to the influence of vehicles traveling in front, rear, and sides.

[0071] <Summary of the embodiment> 1. The information processing device (e.g., 101) of the above embodiment is An information processing device (e.g., 101) for estimating the amount of movement of a moving object to which the information processing device is attached, An imaging means (e.g., 203) for acquiring an image of the outside world of the moving body; an estimation means (e.g., 201) for estimating a movement amount of a moving object based on a first captured image captured at a first time and a second captured image captured at a second time prior to the first time; The estimation means estimates the amount of movement of the moving object based on the difference between each piece of image information of a specified area of ​​a changed image obtained by changing the second captured image based on mutually different predicted amounts of movement and image information obtained by cutting out a specified area of ​​the first captured image (e.g., S603, S606, S701, S803, S804).

[0072] According to this embodiment, it is possible to estimate the behavior of a moving object in a simple and redundant manner. In addition, by simplifying the processing, the processing can be executed at high speed even with the computational resources of a small device such as a drive recorder that operates independently.

[0073] 2. The information processing device in the above embodiment is The amount of movement of the moving object is estimated based on the predicted amount of movement corresponding to the change image with the smallest difference. According to this embodiment, it is possible to estimate the behavior of a moving object in a simple and redundant manner based on changes in an image.

[0074] 3. The information processing device in the above embodiment is The predicted movement amount includes a horizontal movement amount of the moving object, The estimation means estimates the moving speed of the moving object based on the difference between each piece of image information of a specified area of ​​the shifted image, obtained by shifting the second captured image horizontally by different shift amounts, and image information obtained by cutting out a specified area of ​​the first captured image (e.g., S606).

[0075] According to this embodiment, it is possible to estimate the horizontal behavior of a moving object in a simple and redundant manner.

[0076] 4. The information processing device in the above embodiment is The amount of movement of the moving object is estimated based on the amount of shift corresponding to the shifted image that minimizes the difference. According to this embodiment, it is possible to estimate the horizontal behavior of a moving object in a simple and redundant manner based on the shift amount of the images.

[0077] 5. The information processing device in the above embodiment is The predicted movement amount includes a movement speed related to forward or backward movement of the moving object, The estimation means estimates the moving speed of the moving object based on the difference between image information of a specified area of ​​a scaled image obtained by enlarging or reducing the second captured image at different magnifications and image information obtained by cutting out a specified area of ​​the first captured image (e.g., S603, S701, S803, S804).

[0078] According to this embodiment, it is possible to estimate the forward and backward moving speeds of a moving object in a simple and redundant manner.

[0079] 6. The information processing device in the above embodiment is The amount of movement of the moving object is estimated based on the magnification ratio corresponding to the magnified image that minimizes the difference. According to this embodiment, it is possible to estimate the forward and backward moving speeds of a moving object in a simple and redundant manner by changing the magnification of an image.

[0080] 7. The information processing device in the above embodiment is The imaging means is capable of acquiring images obtained by imaging a plurality of directions of the outside world of the moving object (e.g., 203); The estimation means estimates the moving speed of the moving object using a first captured image and a second captured image captured in at least one of a plurality of directions (eg, S603, S701).

[0081] According to this embodiment, even in a situation where an appropriate moving speed cannot be estimated from an image captured in one direction, it is possible to estimate the moving speed in the forward and backward directions of a moving object in a simple and redundant manner from an image captured in another direction.

[0082] 8. The information processing device in the above embodiment is The method further includes a determination means (e.g., 201, S702) for determining the reliability of the moving speed of the moving object estimated by the estimation means, The determination means determines the reliability of the movement speed of a first moving body estimated using a first captured image and a second captured image captured in a first direction among a plurality of directions, using the movement speeds of a plurality of moving bodies estimated using the first captured image and the second captured image in each of a plurality of other directions.

[0083] According to this embodiment, it is possible to determine whether a moving speed estimated from an image captured in one direction is reliable by using an image captured in another direction. Thus, the speed level can be calculated in a simple manner, while the use of an estimated result with low reliability can be eliminated.

[0084] 9. The information processing device in the above embodiment is Further comprising a recognition means (e.g., 201, S801) for recognizing a moving object included in a captured image, The estimation means estimates the movement speed of the moving object by using a first captured image and a second captured image captured in a direction related to an captured image in which the moving object was not recognized by the recognition means, among the captured images captured in multiple directions (e.g., S803).

[0085] According to this embodiment, the speed level can be calculated in a simple manner, and it is possible to reduce the deterioration in accuracy of the estimated speed level caused by the influence of moving objects in front, behind, and to the sides.

[0086] 10. The information processing device in the above embodiment is The vehicle control device further includes an output unit that outputs a notification to a passenger in accordance with the amount of movement of the moving body estimated by the estimation unit.

[0087] According to this embodiment, it is possible to provide the occupant with a driving assistance function according to the amount of movement.

[0088] 11. The information processing device in the above embodiment is The first captured image and the second captured image are captured images of the area in front of or behind the moving object.

[0089] 12. The information processing method of the above embodiment (e.g., FIG. 6) An information processing method for estimating a movement amount of a moving object to which a sensor is attached, comprising the steps of: An imaging step of acquiring an image of an external environment of the moving body; an estimating step of estimating a movement amount of the moving object based on a first captured image captured at a first time and a second captured image captured at a second time prior to the first time, In the estimation process, the amount of movement of the moving body is estimated based on the difference between each piece of image information of a specified area of ​​a changed image in which the second captured image is changed based on mutually different predicted amounts of movement, and image information obtained by cutting out a specified area of ​​the first captured image.

[0090] According to this embodiment, it is possible to estimate the behavior of a moving object in a simple and redundant manner.

[0091] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0092] 100: vehicle, 101: drive recorder, 201: control unit, 202: storage device, 203: imaging device, 205: output device

Claims

1. An information processing device for estimating a movement amount of a moving object to which the information processing device is attached, An imaging means for acquiring an image of an external environment of the moving body; an estimation unit that estimates a movement amount of the moving object based on a first captured image captured at a first time and a second captured image captured at a second time prior to the first time, The information processing device is characterized in that the estimation means estimates the amount of movement of the moving body based on the difference between each piece of image information of a specified area of ​​a changed image in which the second captured image is changed based on mutually different predicted movement amounts and image information cut out of a specified area of ​​the first captured image.

2. The information processing apparatus according to claim 1 , wherein the movement amount of the moving object is estimated based on the predicted movement amount corresponding to the change image in which the difference is minimum.

3. the predicted movement amount includes a horizontal movement amount of the moving object, 3. The information processing device according to claim 1, wherein the estimation means estimates the horizontal movement amount of the moving object based on a difference between each piece of image information of the specified area of ​​a shifted image obtained by shifting the second captured image horizontally by different shift amounts and image information obtained by cutting out a specified area of ​​the first captured image.

4. The information processing apparatus according to claim 3 , further comprising: estimating a horizontal movement amount of the moving object based on the shift amount corresponding to the shifted image in which the difference is minimum.

5. the predicted movement amount includes a movement speed relating to forward or backward movement of the moving object, 5. The information processing device according to claim 1, wherein the estimation means estimates the moving speed of the moving body based on a difference between each of image information of the specified area of ​​a scaled image obtained by enlarging or reducing the second captured image at different scale factors and image information obtained by cutting out a specified area of ​​the first captured image.

6. 6. The information processing apparatus according to claim 5, wherein the moving speed of the moving object is estimated based on the magnification ratio corresponding to the magnified image in which the difference is minimum.

7. the imaging means is capable of acquiring captured images obtained by capturing images of a plurality of directions of an external environment of the moving body, 7. The information processing apparatus according to claim 5, wherein the estimation means estimates the moving speed of the moving object by using the first captured image and the second captured image captured in at least one of the plurality of directions.

8. The method further includes an evaluation unit for evaluating a reliability of the moving speed of the moving object estimated by the estimation unit, The evaluation means includes: calculating a difference between a moving speed of a first moving object estimated using the first captured image and the second captured image captured in a first direction among the plurality of directions and moving speeds of the plurality of moving objects estimated using the first captured image and the second captured image in each of the other plurality of directions; calculating a sum of a difference in a direction in which the calculated difference is smallest and a difference in a direction in which the calculated difference is second smallest as a reliability of the moving speed of the first moving body estimated in the first direction; The information processing apparatus according to claim 7 , further comprising: evaluating a reliability of the moving speed of the moving object depending on whether the reliability is equal to or less than a threshold value.

9. The imaging device further includes a recognition unit for recognizing a moving object included in the captured image, The information processing device according to claim 7 or 8, characterized in that the estimation means estimates the movement speed of the moving object by using the first captured image and the second captured image, which are captured in a direction related to an image in which the moving object is not recognized by the recognition means, among the captured images captured in the multiple directions.

10. 10. The information processing apparatus according to claim 1, further comprising an output unit that outputs a notification to a passenger in accordance with the amount of movement of the moving body estimated by the estimation unit.

11. 5. The information processing apparatus according to claim 3, wherein the first captured image and the second captured image are captured images of a front or rear of the moving object.

12. An information processing method for estimating a movement amount of a moving object to which a sensor is attached, comprising the steps of: An imaging step of acquiring an image of an external environment of the moving body; an estimation step of estimating a movement amount of the moving object based on a first captured image captured at a first time and a second captured image captured at a second time prior to the first time, This information processing method is characterized in that in the estimation process, the amount of movement of the moving body is estimated based on the difference between each piece of image information of a specified area of ​​a changed image in which the second captured image is changed based on mutually different predicted movement amounts, and image information cut out of a specified area of ​​the first captured image.

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images