Information processing device, vehicle behavior determination method, and vehicle behavior determination program

Abstract

To obtain a behavior of a vehicle from a moving image obtained by photographing an outside from an inside of the vehicle.SOLUTION: An information processing device 100 includes: an acquisition section 131 for acquiring a moving image obtained by photographing an outside from an inside of a vehicle; a comparison section 132 for comparing images of frames of a moving image acquired by the acquisition section between the frames for each predetermined area of the images; and a determination section 133 for determining a behavior of the vehicle based on a comparison result of the comparison section 132.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] ,

[0006] , ,

[0005] , ,

[0001] The present invention relates to an information processing apparatus, a vehicle behavior determination method, and a vehicle behavior determination program.

Background Art

[0002] Conventionally, motion of an object has been detected using differences between frames of a moving image. As a typical example, a technique for detecting an object by performing template matching has been proposed. For example, an image of a monitoring target area related to a station platform and a track is captured. Then, for the background image prepared for the monitoring target area and the captured image, template matching is performed for each of a plurality of partial areas in the image to calculate a correlation value between the images for each partial area. Then, based on the correlation value for each partial area, the presence situation of a vehicle such as a train in the monitoring target area is determined.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, a recording device that captures and records the outside from inside a vehicle such as a drive recorder is known.

[0005] However, the above conventional technique performs template matching to detect whether there is a moving object such as a vehicle in the captured image. In the above conventional technique, for example, the behavior of a vehicle cannot be obtained from a moving image captured from inside a vehicle by a recording device or the like.

[0006] The present invention aims to realize an information processing device, a vehicle behavior determination method, and a vehicle behavior determination program that can determine the behavior of a vehicle from video footage taken from inside the vehicle looking outwards. [Means for solving the problem]

[0007] The information processing device according to claim 1 is characterized by comprising: an acquisition unit that acquires a moving image of the outside taken from inside a vehicle; a comparison unit that compares the images of frames of the moving image acquired by the acquisition unit between frames for each predetermined region of the image; and a determination unit that determines the behavior of the vehicle based on the comparison result of the comparison unit.

[0008] The vehicle behavior determination method described in claim 8 is a vehicle behavior determination method executed by an information processing device, characterized by comprising: an acquisition step of acquiring a moving image of the outside taken from inside the vehicle; a comparison step of comparing the images of frames of the acquired moving image between frames for each predetermined region of the image; and a determination step of determining the behavior of the vehicle based on the comparison result.

[0009] The vehicle behavior determination program described in claim 9 is a vehicle behavior determination program that causes an information processing device to execute an acquisition procedure for acquiring a moving image of the outside taken from inside a vehicle, a comparison procedure for comparing images of frames of the acquired moving image between frames for each predetermined region of the image, and a determination procedure for determining the behavior of the vehicle based on the comparison result. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of an information processing system according to an embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing device according to the embodiment. [Figure 3] Figure 3 illustrates the comparison of images between frames according to the embodiment. [Figure 4] Figure 4 is a diagram illustrating template matching according to an embodiment. [Figure 5] Figure 5 illustrates the determination of the vehicle's behavior according to this embodiment. [Figure 6] Figure 6 is a diagram illustrating the determination of the vehicle's behavior according to the embodiment. [Figure 7] Figure 7 illustrates the movement of the image region within the frame due to the behavior of the vehicle VEx according to this embodiment. [Figure 8] Figure 8 illustrates the movement of the image region within the frame due to the behavior of the vehicle VEx according to this embodiment. [Figure 9] Figure 9 illustrates the movement of the image region within the frame due to the behavior of the vehicle VEx according to this embodiment. [Figure 10] Figure 10 is a diagram illustrating the determination of the vehicle's behavior according to this embodiment. [Figure 11] Figure 11 is a diagram illustrating the determination of the vehicle's behavior according to this embodiment. [Figure 12] Figure 12 is a flowchart showing the procedure for the vehicle behavior determination process that determines the behavior of a vehicle. [Figure 13] Figure 13 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0011] Below, an example of an embodiment for implementing the information processing device, vehicle behavior determination method, and vehicle behavior determination program (hereinafter referred to as "embodiment") will be described in detail with reference to the drawings. Note that the information processing device, vehicle behavior determination method, and vehicle behavior determination program are not limited by this embodiment. Furthermore, the same parts are denoted by the same reference numerals in the following embodiments, and redundant explanations are omitted.

[0012] [1. System Configuration] First, the configuration of the information processing system according to the embodiment will be explained using Figure 1. Figure 1 is a diagram showing an example of the information processing system according to the embodiment. Figure 1 shows information processing system 1 as an example of the information processing system according to the embodiment.

[0013] As shown in FIG. 1, the information processing system 1 may include an in-vehicle device 10 and an information processing device 100. Further, the in-vehicle device 10 and the information processing device 100 are communicably connected by wire or wirelessly via a network N. Also, the information processing system 1 shown in FIG. 1 may include any number of in-vehicle devices 10 and any number of information processing devices 100.

[0014] The in-vehicle device 10 may be a dedicated navigation device built in or externally attached to the vehicle VEx, or may be a recording device (drive recorder) installed in the vehicle VEx for crime prevention and anti-road rage measures.

[0015] Also, the in-vehicle device 10 may be composed of a navigation device and a recording device. As an example of this, the in-vehicle device 10 may be a composite device in which a navigation device and a recording device that are independent of each other are communicably connected to each other. Also, as another example, the in-vehicle device 10 may be one device having a navigation function and a recording function.

[0016] Also, the user can substitute it as the in-vehicle device 10 by introducing a predetermined application into a portable terminal device (for example, a smartphone, a tablet terminal, a notebook PC, a desktop PC, a PDA, etc.) that is used daily. For example, a portable terminal device installed with a predetermined navigation application and a predetermined recording application can be regarded as the in-vehicle device 10 here. When the portable terminal device is utilized as the in-vehicle device 10, for example, it is installed on the dashboard of the vehicle VEx during driving. <00,00096>

[0018] ​The information processing device 100 may acquire various types of data based on the sensor information detected by these sensors (for example, by analyzing the sensor information). For example, the information processing device 100 may acquire video data of the outside taken from inside the vehicle VEx by a camera. The information processing device 100 may also acquire location information from a GPS sensor. The information processing device 100 may acquire sensor information detected not only by sensors provided in the in-vehicle device 10, but also by sensors provided in the vehicle VEx itself.

[0019] The information processing device 100 is a device that performs information processing according to the embodiment. For example, the information processing device 100 compares images of frames of acquired video footage between frames for each predetermined region of the image, and determines the behavior of the vehicle VEx based on the comparison results. Furthermore, the information processing device 100 can also provide various driving assistance to the user based on the determined behavior of the vehicle VEx.

[0020] Here, if the in-vehicle device 10 is an edge computer that performs edge processing near the user, then the information processing device 100 may be, for example, a cloud computer that performs processing on the cloud side. In other words, the information processing device 100 may be a server device.

[0021] Furthermore, the following embodiment shows an example in which the information processing according to the embodiment is realized in the information processing system 1 by transmitting and receiving information between the in-vehicle device 10 and the information processing device 100. However, the information processing according to the embodiment may also be realized on the edge side, i.e., only on the in-vehicle device 10. In this case, the in-vehicle device 10 may be configured to behave like the information processing device 100, for example, by the vehicle behavior determination program according to the embodiment.

[0022] The in-vehicle device 10 transmits sensor information detected by the sensors to the information processing device 100. For example, the in-vehicle device 10 uses a camera to photograph the outside from inside the vehicle VEx. For example, the camera is positioned facing the front of the vehicle VEx and photographs the area in front of the vehicle VEx from inside the vehicle VEx. The in-vehicle device 10 transmits the video data of the outside photographed from inside the vehicle VEx by the camera to the information processing device 100. The in-vehicle device 10 also transmits location information indicating the current position of the vehicle VEx, detected by the GPS sensor, to the information processing device 100. The in-vehicle device 10 may transmit the video images and sensor information captured by the camera in real time, or it may transmit them at periodic intervals, such as at regular intervals.

[0023] The information processing device 100 stores video data and sensor information acquired from the vehicle VEx. The information processing device 100 performs information processing according to the embodiment in the following procedure to determine the behavior of the vehicle VEx from the stored video data.

[0024] [2. Configuration of the information processing device 100] An information processing device 100 according to an embodiment will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. As shown in Figure 2, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0025] (Regarding Communications Unit 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and performs information transmission and reception, for example, with the in-vehicle device 10.

[0026] (Regarding memory unit 120) The storage unit 120 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120 includes dynamic image data 121, a sensor information database 122, and a map information database 123.

[0027] The video data 121 is video data captured by a camera on the in-vehicle device 10. The sensor information database 122 is a database that stores sensor information detected by sensors on the in-vehicle device 10 or sensors on the vehicle VEx itself. The map information database 123 is a database that stores various types of map-related information.

[0028] (Regarding the control unit 130) The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs (for example, a vehicle behavior determination program according to this embodiment) stored in the memory device inside the information processing device 100, using RAM as the working area. Alternatively, the control unit 130 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0029] The control unit 130 includes an acquisition unit 131, a comparison unit 132, a determination unit 133, a storage unit 134, and an output unit 135, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 2, and other configurations are also acceptable as long as they perform the information processing described later.

[0030] (Regarding acquisition section 131) The acquisition unit 131 acquires various types of information from the in-vehicle device 10. For example, the acquisition unit 131 acquires video data of the outside taken from inside the vehicle VEx by the camera of the in-vehicle device 10. The acquisition unit 131 also acquires sensor information detected by sensors of the vehicle VEx (for example, sensors of the in-vehicle device 10 installed in the vehicle VEx, or sensors of the vehicle VEx itself). The acquisition unit 131 stores the acquired video data in video data 121. The acquisition unit 131 also stores the acquired sensor information in the sensor information database 122.

[0031] (Regarding comparison section 132) The comparison unit 132 performs various comparisons. For example, the comparison unit 132 compares the images of frames of the video acquired by the acquisition unit 131 between frames, for each predetermined region of the image.

[0032] A specific example of comparing images between frames will be explained. Figure 3 is a diagram illustrating the comparison of images between frames according to the embodiment. Figure 3 shows an example of an image 150 in a frame of a moving image. The comparison unit 132 divides the image 150 in the frame of the moving image into a predetermined number of regions. In Figure 3, the central part of the image 150, excluding the periphery, is divided vertically and horizontally in a predetermined matrix, thereby dividing the image 150 into a plurality of rectangular regions 160.

[0033] The comparison unit 132 determines the correlation or similarity between frames for each region 160 of the image 150 of the video frame, for example, using template matching, which will be described below as template matching, but is not limited to this. For example, the comparison unit 132 performs template matching on each region 160 of the image 150 of the video frame with the image of the next frame. Figure 4 is a diagram illustrating template matching according to an embodiment. Figure 4 shows the central region 160 to be compared in the images as the region of interest 160a. The comparison unit 132 first performs template matching on each region 160 of the image of the video frame with the region 160 at the same position in the image of the next frame. For example, the comparison unit 132 compares the region 160 designated as the region of interest 160a in Figure 4 of the image of the frame with the region 160 at the same position in the image of the next frame and determines whether they match or not. As a result of the comparison, the comparison unit 132 terminates template matching for the matching regions 160. On the other hand, for regions 160 that do not match as a result of the comparison, the comparison unit 132 compares the region 160 at the same position in the image of the next frame with the surrounding area to determine whether or not there is a matching area. For example, for region 160 designated as region 160a of interest as shown in Figure 4 of the image of the frame, the comparison unit 132 sets the range of the surrounding region 160, centered on region 160 at the same position in the image of the next frame, as the template matching search area. The comparison unit 132 moves the image of region 160 designated as region 160a of interest within the range of the search area, centered on region 160 at the same position in the image of the next frame, to determine whether or not there is a matching area. For regions 160 where there is no matching area, the comparison unit 132 terminates template matching. On the other hand, if there is a matching area, the comparison unit 132 identifies the matching area as the moved position of region 160 designated as region 160a of interest in the image of the next frame, and terminates template matching.

[0034] Furthermore, if the image frame of a moving image includes areas with few features, such as the sky or road surface, accurate matching of the featureless areas 160 is difficult even when performing template matching between frames because the features are so limited. Therefore, the comparison unit 132 may exclude the featureless areas 160 from the comparison. For example, the comparison unit 132 may calculate the standard deviation of the luminance values ​​of the pixels in each area 160 of the image frame and exclude areas 160 whose standard deviation is smaller than a predetermined value from the comparison. The predetermined value is set as the upper limit of the standard deviation that is considered to have few features. This allows for accurate matching of the same image area 160 between frames.

[0035] (Regarding the determination unit 133) The determination unit 133 performs various determinations. For example, the determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. For example, the determination unit 133 determines that the vehicle VEx is stopped based on the number of matching regions. For example, the determination unit 133 determines that the vehicle VEx is stopped if the number of matching regions satisfies a predetermined condition. The determination unit 133 also determines the direction of movement of the vehicle VEx based on the movement of each region of the image in the frame of the moving image.

[0036] Here, we will explain a specific example of determining the behavior of a vehicle VEx. First, we will explain a specific example of determining when a vehicle VEx is stopped, using Figures 5 and 6.

[0037] Figure 5 is a diagram illustrating the determination of the vehicle's behavior according to the embodiment. Figure 5 shows an image 151 taken from inside the parked vehicle VEx, looking forward to the front of the vehicle VEx. Image 151 shows buildings 151a and 151b in front of the parked vehicle VEx.

[0038] The comparison unit 132 divides the image 151 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the luminance values ​​of the pixels in each region 160 of the image 151. The comparison unit 132 excludes regions 160 with a standard deviation smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 with a standard deviation greater than or equal to the predetermined value. Figure 5 shows the regions 161 that match as a result of template matching. Figure 5 also shows regions 162 that were excluded from the comparison or for which there were no matching parts. In Figure 5, regions 160 such as the road surface are excluded from the comparison because they have few features.

[0039] Figure 6 is a diagram illustrating the determination of the vehicle's behavior according to the embodiment. Figure 6 shows an image 152 taken from inside the vehicle VEx while it is traveling on a road, showing the area in front of the vehicle VEx. Image 152 shows the road 152a on which the vehicle VEx is traveling, and buildings 152b around the road.

[0040] The comparison unit 132 divides the image 152 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the luminance values ​​of the pixels in each region 160 of the image 152. The comparison unit 132 excludes regions 160 with a standard deviation smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 with a standard deviation greater than or equal to the predetermined value. This allows for the exclusion of regions with few features that are unsuitable for template matching. In addition to the standard deviation of luminance values, exclusion may also be performed if feature points are not extracted or edges are not extracted. Figure 6 shows the regions 161 that match as a result of template matching. In Figure 6, since the vehicle VEx is in motion, regions 160 near the far end of the road 152a are shown as matching regions 161. Also in Figure 6, the movement position in the next frame when there is a part that matches region 160 as a result of template matching is shown as a matching region 163. Also in Figure 6, the direction of movement to the matching region 163 is indicated by an arrow 164. For example, in Figure 6, a matching region 163 is identified in region 160 near building 152b, and the direction of movement is indicated by an arrow 164. Figure 6 also shows regions 162 that were excluded from the comparison or where no matching parts were found. In Figure 6, since the vehicle VEx is in motion, many of the regions 160 become regions 162.

[0041] The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. For example, the determination unit 133 determines that the vehicle VEx is stopped based on the number of matching regions 161. For example, the determination unit 133 determines that the vehicle VEx is stopped if the number of matching regions 161 is greater than or equal to a predetermined threshold, or greater than or equal to a predetermined percentage of the number of regions 160 that were compared. The predetermined threshold or predetermined percentage is set to a value that allows the vehicle VEx to be considered stopped. For example, the predetermined percentage is set to either 30% or 60%. As a result, the determination unit 133 can determine that the vehicle VEx is stopped from the image 151 shown in Figure 5, and that the vehicle VEx is not stopped from the image 152 shown in Figure 6. Also, in Figure 5, there are 78 matching regions 161. On the other hand, in Figure 6, there are 4 matching regions 161. For example, by setting the threshold to any value between 30 and 78 (or a percentage between 30 and 60%), the determination unit 133 can determine from the image 151 shown in Figure 5 that the vehicle VEx is stopped, and from the image 152 shown in Figure 6 that the vehicle VEx is not stopped.

[0042] Next, we will explain a specific example of determining the direction of movement of the vehicle VEx using Figures 7 to 11. Figures 7 to 9 are diagrams illustrating the movement position of region 160 in the frame image due to the behavior of the vehicle VEx according to the embodiment. Figures 7 to 9 schematically show the movement position of region 160 in the N+1 frames of the N frame image due to the behavior of the vehicle VEx. In Figures 7 to 9, the vertical column of region 160 located in the center of the image is designated as column C. Also in Figures 7 to 9, the vertical column of region 160 one column to the right of column C is designated as column R1, and the vertical column of region 160 two columns to the right is designated as column R2. Also in Figures 7 to 9, the vertical column of region 160 one column to the left of column C is designated as column L1, and the vertical column of region 160 two columns to the left is designated as column L2.

[0043] Figure 7 shows the movement of region 160 when the vehicle VEx is moving in a straight line. When the vehicle VEx is moving in a straight line, region 160 moves radially from the center of the image. For example, in an N-frame image, in the N+1 frame, with column C as the center, the regions 160 of columns R1 and R2 move to the right, and the regions 160 of columns L1 and L2 move to the left. Regions 160 of columns C, R1, R2, L1, and L2 also move vertically. Furthermore, the amount of movement of region 160 in the N-frame image increases the further it is from the central column C. For example, in the N+1 frame, the region 160 of column L2 moves more than the region 160 of column L1.

[0044] Figure 8 shows the movement of region 160 when vehicle VEx is turning left. When vehicle VEx is turning left, region 160 moves to the right of the image, expanding both vertically and horizontally. For example, in an N-frame image, in N+1 frames, regions 160 in columns L2, L1, C1, R1, and R2 move to the right. Region 160 also moves vertically the further it is from the vertical center of the image. Furthermore, the further to the right region 160 is, the greater the amount of movement. For example, in N+1 frames, region 160 in column R2 moves further to the right than region 160 in column R1.

[0045] Figure 9 shows the movement of region 160 when vehicle VEx is turning right. When vehicle VEx is turning right, region 160 moves to the left of the image, expanding vertically. For example, in an N-frame image, in N+1 frames, regions 160 in columns L2, L1, C1, R1, and R2 move to the left. Region 160 also moves vertically the further it is from the vertical center of the image. Furthermore, when vehicle VEx is turning right, the amount of movement is greater for regions 160 on the left side of the example. For example, in N+1 frames, region 160 in column L2 moves further to the left than region 160 in column L1.

[0046] Therefore, the direction of movement of the vehicle VEx can be determined based on the results of the movement of each region 160 of the image in the video frame.

[0047] Figure 10 is a diagram illustrating the determination of the behavior of the vehicle VEx according to the embodiment. Figure 10 shows an image 153 taken from inside the vehicle VEx while it is turning left, showing the view in front of the vehicle VEx. Image 153 shows a pedestrian crossing 153a at an intersection, street trees 153b, a vehicle 153c, etc.

[0048] The comparison unit 132 divides the image 153 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the luminance values ​​of the pixels in each region 160 of the image 153. The comparison unit 132 excludes regions 160 whose standard deviation is smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 whose standard deviation is greater than or equal to the predetermined value. For example, the comparison unit 132 compares region 160 with region 160 at the same position in the image of the next frame and determines whether they match. The comparison unit 132 calculates a correlation value indicating the similarity of the regions 160, and determines that they match if the correlation value is greater than a threshold. The comparison unit 132 terminates template matching for regions 160 that match as a result of the comparison. On the other hand, for regions 160 that do not match as a result of the comparison, the comparison unit 132 compares the surrounding area with the region 160 at the same position in the image of the next frame as the center and determines whether there is a matching part. The comparison unit 132 calculates a correlation value indicating the similarity of region 160 to the surrounding area, and determines that the area with a correlation value greater than a threshold is a matching area. If a matching area is found, the comparison unit 132 identifies the matching area as the movement position of region 160 in the next frame's image. The threshold used for determining the match may be changed. For example, when comparing region 160 to the same position in the next frame's image, a higher threshold can be used to determine a match with higher accuracy. On the other hand, when comparing region 160 to the surrounding area of ​​the next frame, a lower threshold can be used to make it easier to identify the movement position. Figure 10 shows the matching area 161 as a result of template matching. In Figure 10, vehicle VEx is turning left, and the objects in the image are generally moving to the right, so there are few matching areas 161, and the lower right area 160 of image 153 is shown as the matching area 161. Also in Figure 10, the movement position in the next frame when there is a matching area with region 160 as a result of template matching is shown as the matching area 163. For example, in Figure 10, a matching area 163 is identified in area 160 near the pedestrian crossing 153a and the vehicle 153c. Figure 10 also shows areas 162 that were excluded from the comparison or where no matching parts were found.

[0049] Figure 11 is a diagram illustrating the determination of the behavior of a vehicle VEx according to an embodiment. Figure 11 shows an image 154 taken from inside the vehicle VEx while it is turning right, showing the view in front of the vehicle VEx. Image 154 shows a building 154a at the corner of the intersection, the vehicle 154b, the road to the right 154c, and the railway tracks 154d running alongside the road 154c.

[0050] The comparison unit 132 divides the image 154 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the luminance values ​​of the pixels in each region 160 of the image 154. The comparison unit 132 excludes regions 160 with a standard deviation smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 with a standard deviation greater than or equal to the predetermined value. Figure 11 shows the regions 161 that match as a result of template matching. In Figure 11, the vehicle VEx is turning right, and the objects in the image are generally moving to the left, so there are few matching regions 161, and the lower right region 160 of the image 154 is shown as a matching region 161. Also in Figure 11, the movement position in the next frame when there is a part that matches a region 160 as a result of template matching is shown as a matching region 163. For example, in Figure 11, a matching region 163 is identified in region 160 near building 154a. Furthermore, Figure 11 shows the regions 162 that were excluded from the comparison or for which there were no matching parts.

[0051] The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. For example, the determination unit 133 determines the direction of movement of the vehicle VEx based on the movement result of each region 160. For example, the determination unit 133 determines that the vehicle VEx is turning left if the number of regions moving to the right in the entire image of the video frame is greater than or equal to a predetermined threshold or a predetermined percentage. Also, the determination unit 133 determines that the vehicle VEx is turning right if the number of regions moving to the left in the entire image of the video frame is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or predetermined percentage is set to a value that allows the vehicle VEx to be considered to be turning right or left. For example, in Figure 10, the number of regions 160 moving to the right in the entire image of the video frame is 22. On the other hand, in Figure 11, the number of regions 160 moving to the left in the entire image of the video frame is 24. For example, by setting the threshold to a value of around 10 to 20, the determination unit 133 can determine from image 153 shown in Figure 10 that vehicle VEx is turning left, and from image 154 shown in Figure 10 that vehicle VEx is not turning right.

[0052] Here, as shown in Figures 7 to 9, the region 160 located in the left half of the image frame of the moving image moves to the left when going straight or turning right, and to the right when turning left. Similarly, the region 160 located in the right half of the image frame of the moving image moves to the right when going straight or turning left, and to the left when turning right.

[0053] Therefore, the determination unit 133 may determine a right turn based on the movement of a region 160 located in the left half of the image of the video frame to the right, and determine a left turn based on the movement of a region 160 located in the right half of the image of the video frame to the left. For example, the determination unit 133 determines that the vehicle VEx is turning left if the number of regions 160 moving to the right in the left half of the image of the video frame is greater than or equal to a predetermined threshold or a predetermined percentage. Also, the determination unit 133 determines that the vehicle VEx is turning right if the number of regions 160 moving to the left in the right half of the image of the video frame is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or predetermined percentage is set to a value that allows the vehicle VEx to be considered to be turning right or left. Alternatively, the determination unit may focus only on the amount of movement and determine a right or left turn if the amount of movement is greater than when moving straight.

[0054] Alternatively, the determination unit 133 may determine the behavior of the vehicle VEx by calculating the average movement for each column of the region 160. For example, the determination unit 133 calculates the average amount of movement and direction of movement for each column of the region 160. Based on the average amount of movement and direction of movement for each column of the region 160, the determination unit 133 determines the behavior of the vehicle VEx.

[0055] For example, in Figure 10, below image 153, the average amount of movement and direction of movement are shown by bars 170 for each vertical column of region 160.

[0056] Furthermore, in Figure 11, below Image 154, the average amount of movement and direction of movement are indicated by bars 170 for each vertical column of region 160.

[0057] The determination unit 133 determines that vehicle VEx is turning left if, in the entire frame of the video, the number of columns where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The determination unit 133 also determines that vehicle VEx is turning right if, in the entire frame of the video, the number of columns where the direction of movement is to the left and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or predetermined percentage is set to a value that allows vehicle VEx to be considered to be turning left or left. For example, by setting the threshold to a value of about 10 to 30, the determination unit 133 can determine from image 153 shown in Figure 10 that vehicle VEx is turning left, and from image 154 shown in Figure 10 that vehicle VEx is not turning right.

[0058] Alternatively, the determination unit 133 may determine the behavior of the vehicle VEx by calculating the average amount of movement for each column of region 160 in the left half and right half of the image of the video frame.

[0059] For example, in Figure 10, the number of columns determined to be left turns and the number of columns determined to be right turns are shown as "L:8, R:0" below Image 153. The number of columns determined to be left turns is the number of columns in the right half of Image 153 where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value. The number of columns determined to be right turns is the number of columns in the left half of Image 153 where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value.

[0060] Furthermore, Figure 11 shows the number of columns determined to be turning left and the number of columns determined to be turning right, labeled "L:0, R:8" below Image 154. The number of columns determined to be turning left is the number of columns in the right half of Image 154 where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value. The number of columns determined to be turning right is the number of columns in the left half of Image 154 where the direction of movement is to the left and the average amount of movement is greater than or equal to a certain value.

[0061] The determination unit 133 determines that vehicle VEx is turning left if, in the left half of the image of the frame of the moving image, the number of columns where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The determination unit 133 also determines that vehicle VEx is turning right if, in the right half of the image of the frame of the moving image, the number of columns where the direction of movement is to the left and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or predetermined percentage is set to a value that allows vehicle VEx to be considered to be turning right or left. For example, by setting the threshold to a value of about 5, the determination unit 133 can determine that vehicle VEx is turning left from image 153 shown in Figure 10, and that vehicle VEx is not turning right from image 154 shown in Figure 11. Alternatively, the determination unit 133 may determine whether a turn is right or left from the difference between the number of columns determined to be turning left and the number of columns determined to be turning right. For example, the determination unit 133 may determine that vehicle VEx is turning left if the difference between the number of columns determined to be turning left and the number of columns determined to be turning right is greater than or equal to a predetermined threshold or a predetermined percentage. Alternatively, the determination unit 133 may determine that vehicle VEx is turning right if the difference between the number of columns determined to be turning right and the number of columns determined to be turning left is greater than or equal to a predetermined threshold or a predetermined percentage. For example, suppose the number of columns determined to be turning left is 2. Suppose the number of columns determined to be turning right is 8. Suppose the threshold is 5. In this case, the difference between the number of columns determined to be turning right and the number of columns determined to be turning left is 6. Since the difference of 6 is greater than or equal to the threshold of 5, the determination unit 133 determines that it is turning right.

[0062] Furthermore, when a vehicle VEx is traveling alongside another vehicle at the same speed on a multi-lane road, the video image may show the other vehicle traveling alongside in a similar position, resulting in a large number of matching regions 161. This can lead to the system mistakenly determining that the vehicle VEx is stopped. Therefore, the determination unit 133 may determine that the vehicle VEx is stopped if a predetermined number of consecutive frames satisfy a predetermined condition regarding the number of matching regions 161. For example, the determination unit 133 determines that the vehicle VEx is stopped if, for example, five consecutive frames have a number of matching regions 161 that is equal to or greater than a predetermined threshold, or equal to or greater than a predetermined ratio of the number of regions 160 used for comparison. This prevents the system from mistakenly determining that the vehicle VEx is stopped when it is temporarily traveling alongside another vehicle at the same speed, and allows for accurate determination of whether the vehicle VEx is stopped. The determination unit 133 may also determine that the vehicle VEx has made a right or left turn if a predetermined number of consecutive frames indicate a right or left turn. The predetermined number can be variable, and by setting it to a small value, the behavior of the vehicle VEx can be determined in a short time.

[0063] (Regarding storage unit 134) The storage unit 134 stores the determination result made by the determination unit 133. For example, the storage unit 134 stores the behavior of the vehicle VEx determined by the determination unit 133 in the video data 121 or data associated with the video data 121, associating it with the frame and time of capture of the video in which the behavior was determined. Alternatively, the storage unit 134 may store the determination result made by the determination unit 133 in the map information database 123, associating it with the map location where the behavior was detected.

[0064] (Regarding output section 135) The output unit 135 performs various outputs. For example, the output unit 135 transmits various information to the in-vehicle device 10 based on the determination result by the determination unit 133. For example, if the determination unit 133 does not determine that a stop is required at a location stored in the map information database 123, the output unit 135 transmits a warning to the in-vehicle device 10.

[0065] As described above, the information processing device 100 according to this embodiment can determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards. For example, the information processing device 100 can determine the stopping, right turning, and left turning behavior of the vehicle VEx from the video footage. For example, the information processing device 100 can determine from the video footage behaviors of the vehicle VEx, such as momentary stops that cannot be determined by GPS location information. As a result, the information processing device 100 can also be used in the following ways. For example, by determining the behavior of the vehicle VEx from video footage captured by a recording device such as a drive recorder using the information processing device 100, accident verification can be performed. For example, it is possible to verify from the video footage whether or not the vehicle VEx made a stop. Furthermore, by determining the behavior of the vehicle VEx from video footage using the information processing device 100, it is possible to evaluate whether the driver is driving safely, and for example, insurance companies can use this to evaluate the driver's driving in telemastic insurance, etc.

[0066] [3. Processing Procedure] Next, we will explain the information processing procedure performed by the information processing device 100 using Figure 12. Figure 12 is a flowchart showing the procedure for the vehicle behavior determination process, which determines the behavior of the vehicle.

[0067] In the example shown in Figure 12, the acquisition unit 131 acquires various types of information from the in-vehicle device 10 (step S10). For example, the acquisition unit 131 acquires video data of the outside taken from inside the vehicle VEx by the camera of the in-vehicle device 10.

[0068] The comparison unit 132 compares the images of the video frames acquired by the acquisition unit 131 between frames, for each predetermined region of the image (step S11). For example, the comparison unit 132 divides the images of the video frames into multiple regions 160. The comparison unit 132 performs template matching between frames for each region 160 of the images of the video frames.

[0069] The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132 (step S12). For example, the determination unit 133 determines that the vehicle VEx is stopped if the number of matching regions 161 satisfies a predetermined condition. The determination unit 133 also determines the direction of movement of the vehicle VEx based on the result of the movement of each region 160 of the image in the frame of the moving image.

[0070] The storage unit 134 stores the determination result made by the determination unit 133 (step S13). For example, the storage unit 134 stores the behavior of the vehicle VEx determined by the determination unit 133 in the video data 121 or data associated with the video data 121, associating it with the video frame and shooting time of the video in which the behavior was determined.

[0071] The output unit 135 transmits various information to the in-vehicle device 10 based on the determination result from the determination unit 133 (step S14). For example, if the determination unit 133 does not determine that a stop is required at a location stored in the map information database 123, the output unit 135 transmits a warning to the in-vehicle device 10.

[0072] Thus, the information processing device 100 according to this embodiment can determine the behavior of the vehicle VEx from video footage taken from inside the vehicle VEx looking outwards.

[0073] In this embodiment, the vehicle behavior determination process described above was performed in the information processing device 100 to determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards. However, the invention is not limited to this. The in-vehicle device 10 may also perform the vehicle behavior determination process described above to determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards.

[0074] Furthermore, in the embodiment, the case where the shape of region 160 is rectangular was described as an example. However, it is not limited to this. The shape of region 160 may be other than rectangular. For example, the comparison unit 132 may divide the image of the video frame into equilateral triangles or regular hexagons and make region 160 into the shape of equilateral triangles or regular hexagons. Alternatively, for example, the comparison unit 132 may divide the image of the video frame into multiple types of polygons and make region 160 into multiple types of polygons. In addition, the comparison unit 132 may recognize the outlines of objects that appear in the image of the video frame and divide the image of the frame into regions 160 based on the outlines of the objects.

[0075] Furthermore, in the embodiment, the case of determining right turns and left turns as the direction of movement of the vehicle VEx was described as an example. However, it is not limited to this. By identifying the matching region 163 through template matching by the comparison unit 132, vertical movement can also be determined. The determination unit 133 may determine the behavior of the vehicle VEx in the vertical direction. This makes it possible to determine, for example, whether the vehicle VEx is going uphill or downhill.

[0076] Furthermore, in the embodiment, the video was described using the example of a camera positioned facing the front of the vehicle VEx, capturing the front of the vehicle VEx from inside the vehicle VEx. However, it is not limited to this. The video can be any video captured from inside the vehicle VEx looking outwards. For example, the video could be a camera positioned facing the rear of the vehicle VEx, capturing the rear of the vehicle VEx from inside the vehicle VEx. If the video is a video captured from inside the vehicle VEx looking outwards, it is possible to determine the vehicle VEx's behavior, such as when it stops. Also, if the video is a video captured from inside the vehicle VEx looking in front of, behind, or to the side of the vehicle VEx, it is possible to accurately determine the direction of movement of the vehicle VEx, such as turning right or left.

[0077] Furthermore, in the embodiment, as shown in Figure 4, the search area for template matching was described as a case where the area 160 designated as the area of ​​interest 160a was centered on the surrounding area 160. However, it is not limited to this. The comparison unit 132 may change the range of the search area according to the speed of the vehicle VEx. For example, the comparison unit 132 may widen the range of the search area to the left and right as the speed of the vehicle VEx increases.

[0078] Furthermore, when filming the outside from inside the vehicle VEx, a portion of the vehicle VEx may be captured in the video. For example, when filming the front of the vehicle VEx from inside the vehicle VEx, the hood of the vehicle VEx may be captured in the video. When a portion of the vehicle VEx is captured in the video, the region 160 of the vehicle VEx portion captured in the frame image will be the same image, and as a result of template matching, it will always be determined to be the matching region 161. Therefore, the information processing device 100 may exclude the always matching region 161 and determine the behavior of the vehicle VEx. For example, the determination unit 133 may exclude the region 160 that is always region 161 in multiple different behaviors and determine the behavior of the vehicle VEx. Also, if the portion of the vehicle VEx captured in the frame image is large, the region 161 capturing the area around the vehicle VEx will be small, which may reduce the accuracy of determining the behavior of the vehicle VEx. Therefore, the information processing device 100 may output a warning to the in-vehicle device 10 when the portion of the vehicle VEx captured in the frame image is large. For example, the output unit 135 may output a warning to the in-vehicle device 10 prompting adjustment of the camera's field of view if the number of consistently matching regions 161 exceeds an acceptable value. Alternatively, the target rectangular region may be disabled.

[0079] [4. Summary] The information processing device 100 according to this embodiment includes an acquisition unit 131, a comparison unit 132, and a determination unit 133. The acquisition unit 131 acquires moving images of the outside taken from inside the vehicle VEx. The comparison unit 132 compares the images of frames of the moving images acquired by the acquisition unit 131 between frames, for each predetermined region 160 of the image. The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. As a result, the information processing device 100 can determine the behavior of the vehicle VEx from moving images of the outside taken from inside the vehicle VEx.

[0080] Furthermore, the comparison unit 132 performs template matching on each region 160 of the image in the video frame with the region 160 at the same position in the image of the next frame. The determination unit 133 determines that the vehicle VEx has stopped based on the number of matching regions 161. The determination unit 133 also determines that the vehicle VEx has stopped if the number of matching regions 161 satisfies a predetermined condition. As a result, the information processing device 100 can determine whether the vehicle VEx has stopped from a video image taken from inside the vehicle VEx looking outwards.

[0081] Furthermore, the determination unit 133 determines that the vehicle VEx has stopped if a predetermined number of consecutive frames satisfy a predetermined condition regarding the number of matching regions 161. This allows the information processing device 100 to accurately determine whether the vehicle VEx has stopped.

[0082] Furthermore, the comparison unit 132 calculates the standard deviation of the luminance values ​​of the pixels in each region 160 of the frame image, and excludes regions 160 whose standard deviation is smaller than a predetermined value from the comparison. As a result, the information processing device 100 can accurately match the same image regions 160 between frames.

[0083] Furthermore, the comparison unit 132 performs template matching on the image of the next frame for each region 160 of the video frame to determine the movement position of each region 160 within the image of the next frame. The determination unit 133 determines the direction of movement of the vehicle VEx based on the movement results of each region 160. As a result, the information processing device 100 can determine the direction of movement of the vehicle VEx from video footage taken from inside the vehicle VEx looking outwards.

[0084] The determination unit 133 determines a right turn based on the movement of region 160 located in the left half of the image frame to the right, and determines a left turn based on the movement of region 160 located in the right half of the image to the left. As a result, the information processing device 100 can accurately determine whether the vehicle VEx is turning right or left from a video image taken from inside the vehicle VEx looking outwards.

[0085] [5. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having the configuration shown in Figure 13. Figure 13 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0086] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0087] The HDD1400 stores programs executed by the CPU1100, as well as data used by such programs. The communication interface1500 receives data from other devices via a predetermined communication network and sends it to the CPU1100, and transmits data generated by the CPU1100 to other devices via the predetermined communication network.

[0088] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0089] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0090] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0091] [6. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0092] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0093] Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

[0094] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention. [Explanation of Symbols]

[0095] 1. Information Processing System 10 Onboard equipment 100 Information Processing Devices 120 Storage section 121 Moving image data 122 Sensor Information Database 123 Map Information Database 130 Control Unit 131 Acquisition Department 132 Comparison Section 133 Judgment section 134 Storage Unit 135 Output section 160, 161, 162 areas 163 Matching Area VEx Vehicle

Claims

1. An acquisition unit that acquires video footage of the outside taken from inside the vehicle, A determination unit that, for each predetermined region of the image of the frame of the moving image acquired by the acquisition unit, determines the correspondence between the image and the image of the next frame, identifies the movement position in the next frame for each region, determines a left turn based on the movement to the right of the region located on the left side of the image of the frame of the moving image, and determines a right turn based on the movement to the left of the region located on the right side of the image, An information processing device characterized by comprising:

2. The determination unit determines the average amount of movement and direction of movement for each column in the region, and determines that the vehicle is turning left if, in the left half of the image of the frame of the moving image, the number of columns in which the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage, and determines that the vehicle is turning right if, in the right half of the image of the frame of the moving image, the number of columns in which the direction of movement is to the left and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The information processing apparatus according to feature 1.

3. A method for determining vehicle behavior performed by an information processing device, The acquisition process involves obtaining video footage of the exterior from inside the vehicle, The acquired video frame image is used to determine the correspondence between the image of the next frame and a predetermined region of the image, the movement position in the next frame for each region is identified, a left turn is determined based on the movement to the right of the region located on the left side of the video frame, and a right turn is determined based on the movement to the left of the region located on the right side of the image. A method for determining vehicle behavior, characterized by including the following:

4. The procedure for acquiring video footage of the exterior from inside the vehicle, A determination procedure that involves determining the correspondence between the image of the acquired video frame, for each predetermined region of the image, and the image of the next frame, identifying the movement position in the next frame for each region, determining a left turn based on the movement to the right of the region located on the left side of the video frame, and determining a right turn based on the movement to the left of the region located on the right side of the image, A vehicle behavior determination program to be executed by an information processing device.

Images