Flickering area detection method, program, flickering area detection device, and autonomous mobile body

Abstract

To reduce failure to detect a flickering area from images.SOLUTION: A flickering area detection method includes: acquisition step S20 of acquiring multiple images captured at multiple time points that are different from each other; association step S30 of associating pixels representing an object captured in each of the multiple images between the multiple images; calculation step S40 of calculating a difference value that is a difference between a pixel value of a first pixel included in a first image, which is one of the multiple images, and a pixel value of a second pixel that is included in a second image among the multiple images different from the first image and is associated with the first pixel in the association step S30; and detection step S50 of detecting a flickering area included in the multiple images based on the difference value. The pixels associated in the association step S30 include pixels included in the flickering area.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a blinking area detection method, a program, a blinking area detection device, and an autonomous moving body. [Background technology]

[0002] Research into autonomous driving of moving objects such as automobiles has been actively conducted. One of the technologies required for autonomous driving of moving objects is surrounding environment recognition. One method for surrounding environment recognition is surrounding environment recognition based on captured images of the surrounding environment. In autonomous driving, in order to understand the intentions of the drivers of surrounding vehicles and traffic conditions, it is particularly necessary to recognize flashing areas such as turn signals of surrounding vehicles, traffic signals, and red warning lights of emergency vehicles, among other image-based surrounding environment recognition.

[0003] For example, Patent Document 1 discloses a technology for detecting a vehicle's turn indicators and the like based on an image captured by a camera. In the technology described in Patent Document 1, first, a vehicle captured in an image is detected, and then the turn indicators included in the vehicle are detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 1,069,1962 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 may not be able to detect the vehicle itself, for example, when a large portion of the vehicle captured in the image is hidden behind another vehicle. As a result, even if the turn signal is captured in the image, the turn signal may not be detected. Furthermore, the technology described in Patent Document 1 may not be able to detect the vehicle when the distance to the vehicle is far, that is, when the image of the vehicle captured in the image is too small. In this case, the turn signal of the vehicle cannot be detected.

[0006] Therefore, an object of the present invention is to reduce the number of blinking areas not detected from an image. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, a flashing area detection method according to one aspect of the present invention includes an acquisition step of acquiring a plurality of images each captured at a plurality of different time points; a matching step of matching pixels representing objects imaged in each of the plurality of images between the plurality of images; a calculation step of calculating a difference value which is the difference between the pixel value of a first pixel included in a first image, which is one of the plurality of images, and the pixel value of a second pixel included in a second image, different from the first image, of the plurality of images, which is matched to the first pixel in the matching step; and a detection step of detecting a flashing area included in the plurality of images based on the difference value, wherein the pixels matched in the matching step include pixels included in the flashing area.

[0008] In order to achieve the above object, a program according to one aspect of the present invention is a program for causing a computer to execute the above blinking area detection method.

[0009] In addition, in order to achieve the above-mentioned object, a flashing area detection device according to one embodiment of the present invention comprises an acquisition unit that acquires a plurality of images each captured at a plurality of different time points; a matching unit that matches pixels representing objects captured in each of the plurality of images between the plurality of images; a calculation unit that calculates a difference value that is the difference between the pixel value of a first pixel included in a first image, which is one of the plurality of images, and the pixel value of a second pixel included in a second image, different from the first image, of the plurality of images, and matched to the first pixel by the matching unit; and a detection unit that detects flashing areas included in the plurality of images based on the difference value, wherein the pixels matched by the matching unit include pixels included in the flashing area.

[0010] In order to achieve the above object, an autonomous moving body according to one aspect of the present invention includes the above blinking area detection device and an imaging unit that captures the plurality of images.

[0011] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0012] According to the present invention, it is possible to reduce failure to detect blinking areas from an image. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of a configuration of an autonomous moving body according to an embodiment; [Figure 2] FIG. 2 is a diagram showing a first example of an image acquired by an acquisition unit according to an embodiment. [Figure 3] FIG. 10 is a diagram showing a second example of an image acquired by the acquisition unit according to the embodiment. [Figure 4]10A and 10B are diagrams illustrating an example of an optical flow calculated by a correspondence unit according to the embodiment. [Figure 5] FIG. 10 is a diagram showing, in gray scale, the SV value of each pixel in an image captured at time tn. [Figure 6] FIG. 10 is a diagram showing, in gray scale, the SV value of each pixel of a predicted image at time tn of an image captured at time tn-1. [Figure 7] This is an image at time tn that is composed of pixels with the maximum SV value. [Figure 8] This is an image at time tn that is composed of pixels with the smallest SV value. [Figure 9] 10A and 10B are diagrams illustrating difference values ​​of pixels calculated by a calculation unit according to the embodiment. [Figure 10] 10 is a graph showing the relationship between the difference in SV values ​​between a first pixel and a second pixel and the probability that the first pixel and the second pixel are included in a blinking region according to an embodiment. [Figure 11] 1 is a probability map showing the probability that each pixel in an image is included in a blinking region. [Figure 12] FIG. 10 is a diagram illustrating a third example of an image acquired by the acquisition unit. [Figure 13] 13 is a probability map showing the probability that each pixel of the image shown in FIG. 12 is included in a blinking area. [Figure 14] 1 is a flowchart illustrating a flashing area detection method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0016] In addition, each drawing is a schematic diagram and is not necessarily an exact illustration. In addition, the same components are denoted by the same reference numerals in each drawing.

[0017] (Embodiment) A blinking area detection method, a blinking area detection device, and an autonomous moving body according to an embodiment will be described.

[0018] [1. Configuration of the blinking area detection device and autonomous moving body] The configuration of an autonomous moving body according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an autonomous moving body V10 according to this embodiment.

[0019] The autonomous moving body V10 is a moving body that performs automatic driving, and includes an imaging unit 10 and a blinking area detection device 100. In this embodiment, the autonomous moving body V10 further includes a travel control unit 70 and a memory unit 80. The autonomous moving body V10 may further include a sensor such as a LiDAR (Light Detection and Ranging) sensor. The autonomous moving body V10 is not particularly limited as long as it is a moving body. In this embodiment, the autonomous moving body V10 is a vehicle that performs automatic driving. Hereinafter, the autonomous moving body V10 will also be referred to as the host vehicle.

[0020] The imaging unit 10 captures images of the surroundings of the autonomous moving body V10. The imaging unit 10 captures multiple images at multiple different time points. In this embodiment, the imaging unit 10 acquires images at predetermined time intervals. For example, the imaging unit 10 captures a first image captured at a first time point and a second image captured at a second time point different from the first time point. In this embodiment, the imaging unit 10 is a camera that captures color images in front of the autonomous moving body V10 every 0.1 seconds.

[0021] The blinking area detection device 100 is a device that detects blinking areas included in multiple images captured by the imaging unit 10 based on the multiple images. The blinking area detection device 100 detects, for example, areas corresponding to vehicle turn signals, traffic signals, red warning lights of emergency vehicles, and the like, around the autonomous moving body V10 as blinking areas. The blinking period in the blinking area is not particularly limited. In this embodiment, the blinking period in the blinking area is 0.5 seconds or more and 1 second or less. For example, the blinking period of a vehicle's turn signal is regulated by safety standards to 0.5 seconds or more and 1 second or less. Therefore, the blinking area detection device 100 according to this embodiment can detect the vehicle's turn signal as a blinking area. In this embodiment, the blinking area detection device 100 includes an acquisition unit 20, an association unit 30, a calculation unit 40, and a detection unit 50.

[0022] The acquisition unit 20 acquires a plurality of images captured at a plurality of different time points. In this embodiment, the acquisition unit 20 sequentially acquires a plurality of color images captured by the imaging unit 10. The acquisition unit 20 acquires one image at each of the plurality of time points. Note that the acquisition unit 20 may acquire all of the images captured by the imaging unit 10, or may acquire only a portion of the images.

[0023] The range of time points over which the multiple images are captured (i.e., the interval from the first to the last of the multiple time points over which the multiple images are captured) is set appropriately according to the blinking cycle of the blinking area to be detected. For example, the range may be 0.1 seconds or more, or 0.5 seconds or more. The range may also be 1 second or less.

[0024] In this embodiment, the acquisition unit 20 acquires the latest image captured by the imaging unit 10 (at time t n (n: integer)) and five images captured by the imaging unit 10 immediately before that. n The image captured at time t n Time t 0.1 seconds before n-1 The image captured at time t n Time t 0.2 seconds before n-2 The image captured at time t n Time t 0.3 seconds before n-3 The image captured at time t n Time t 0.4 seconds before n-4 The image captured at time t n Time t 0.5 seconds before n-5 In this embodiment, the acquisition unit 20 acquires the images sequentially each time the imaging unit 10 captures each of the plurality of images. n When acquiring an image captured at time t n-1 The previously captured image may be acquired by the acquisition unit 20 and then stored in the storage unit 80 or the like as an image that has been processed by the association unit 30, which will be described later.

[0025] The acquisition unit 20 acquires pixel values ​​of each of a plurality of pixels included in each image. In this embodiment, the acquisition unit 20 acquires at least values ​​corresponding to lightness and saturation as pixel values. Here, an example of an image acquired by the acquisition unit 20 will be described with reference to Figs. 2 and 3. Figs. 2 and 3 are diagrams showing a first example and a second example of an image acquired by the acquisition unit 20 according to this embodiment, respectively. Note that although Figs. 2 and 3 show monochrome images, the image acquired by the acquisition unit 20 may also be a color image. The image shown in Fig. 2 is captured at time t n The image shown in FIG. 3 is an image captured by the image capturing unit 10 at time t n Time t 0.1 seconds before n-12 and 3, the positions of some of the objects, such as vehicles, are different between Fig. 2 and Fig. 3, and the lighting states of the turn signals provided on the objects, which are vehicles, are different.

[0026] The association unit 30 is a processing unit that associates pixels representing an object captured in each of the multiple images acquired by the acquisition unit 20, between the multiple images. That is, the association unit 30 associates each pixel representing an object captured in one of the multiple images acquired by the acquisition unit 20 with each pixel representing the object captured in the other images. In this embodiment, the association unit 30 associates all pixels that can be associated between the multiple images, among the pixels representing an object captured in each of the multiple images. The pixels associated by the association unit 30 include pixels included in a blinking area.

[0027] For example, the association unit 30 associates a first pixel included in a first image, which is one of the multiple images, with a second pixel included in a second image, which is different from the first image, among the multiple images. The first and second images capture the same object, and a point indicated by a first pixel on the object corresponds to a point indicated by a second pixel on the object. In other words, the first pixel and the second pixel are pixels indicating the same point on the object. The association unit 30 similarly associates pixels other than the first pixel in the first image with pixels in each image other than the first image. Note that a pixel corresponding to a pixel included in the first image does not necessarily have to be included in the other images. Similarly, a pixel included in each image other than the first image does not necessarily have to be included in the other images.

[0028] Furthermore, when a plurality of objects are captured in each image, the association unit 30 associates pixels representing each of the plurality of objects among the plurality of images.

[0029] In this embodiment, the matching unit 30 uses optical flow. Here, optical flow is a vector indicating the amount and direction of movement of pixels representing the same object in two images captured consecutively in time during a video (continuous images). In this embodiment, the Farneback method, which is one method of optical flow, is used. In the Farneback method, the pixel values ​​of each pixel and its neighboring area are approximated by a second-order polynomial, and the coefficients are compared between the two images to estimate the amount of movement with high accuracy. The matching unit 30 uses the Farneback algorithm to calculate dense optical flow from one image to another among multiple images. In other words, the movement of an object, such as a vehicle, captured in each image is calculated for all pixels in each image. In this embodiment, the matching unit 30 calculates the movement of an object, such as a vehicle, captured in each image for all pixels in each image at time t n The image captured at time t n-1 Dense optical flow is calculated from the captured images.

[0030] An example of the optical flow calculated by the association unit 30 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the optical flow calculated by the association unit 30 according to this embodiment. In Fig. 4, the motion vector of each pixel is shown in grayscale. In Fig. 4, the closer a pixel is to white, the greater the movement.

[0031] From the optical flow calculated in this way and each image, the position of the object captured in each image at a predetermined time point is predicted. That is, the association unit 30 predicts the position of the object captured in each of the first and second images at a predetermined time point using the Farneback algorithm. The association unit 30 may use, for example, a time t n The position of the object at time t is predicted, and a predicted image is generated in which the position of the object is moved to the predicted position. n-1 The image taken at time t nIn generating a predicted image in the above, the optical flow (x-axis component of the optical flow in the image F) corresponding to each pixel position (u, v) is calculated as shown in the following equation (1). x (u,v) and y-axis component F y (u,v)) and time t n-1 Information I at each position (u,v) in n-1 (u,v) and time t n The predicted information P(u,v) at the position (u,v) at is calculated.

[0032]

number

[0033] In this embodiment, the optical flow corresponding to the position (u, v) of each pixel is calculated at time t n The image captured at time t n-1 The optical flow is calculated from the image captured at time t n-1 For images captured at times other than t, we use equation (1) to calculate the time t n A predicted image can be generated.

[0034] The information of each pixel included in the predicted image calculated as described above includes at least lightness and saturation as pixel values. Here, the information of each pixel including lightness and saturation does not necessarily mean that the information includes lightness and saturation themselves, but also means that the information includes information corresponding to each of lightness and saturation.

[0035] In this embodiment, the association unit 30 calculates the product of lightness and saturation (hereinafter also referred to as SV value) as the pixel value of each pixel of the predicted image. The distribution of SV values ​​in each predicted image will be explained using Figs. 5 and 6. Fig. 5 shows the distribution of SV values ​​at time t n 6 is a diagram showing the SV value of each pixel in the image captured at time t n-1 The image taken at time t n5 and 6 are diagrams showing, in grayscale, the SV value of each pixel of the predicted image in 2013. In Fig. 5 and Fig. 6, the closer a pixel is to white, the larger the SV value. As shown in Fig. 5 and Fig. 6, the SV value is large in areas where vehicle turn signals, traffic lights, etc. are lit.

[0036] The calculation unit 40 is a processing unit that calculates a difference value, which is the difference between the pixel value of a first pixel included in a first image, which is one of the multiple images acquired by the acquisition unit 20, and the pixel value of a second pixel included in a second image, which is different from the first image among the multiple images, and which is associated with the first pixel by the association unit 30. The pixel value used for the calculation in the calculation unit 40 is not particularly limited. The pixel value used for the calculation in the calculation unit 40 may include a brightness or an SV value. In this embodiment, the pixel value used for the calculation in the calculation unit 40 is the SV value.

[0037] The calculation unit 40 is not particularly limited in the method of selecting the first pixel and the second pixel used to calculate the difference value. In this embodiment, the pixel with the largest SV value is selected as the first pixel from among the corresponding pixels in the plurality of images. In this embodiment, for example, n A pixel corresponding to one pixel included in the image captured at time t n-1 From time t n-5 There are up to five such pixels in the five images captured up to time t. The calculation unit 40 selects the pixel with the largest SV value as the first pixel from among these up to six corresponding pixels. Here, the image including the first pixel is the first image. The calculation unit 40 also selects the pixel with the smallest SV value from among the up to six corresponding pixels as the second pixel. Here, the image including the second pixel is the second image. In this way, the calculation unit 40 selects from the multiple images the pixel with the largest SV value and the pixel with the smallest SV value at each point of each object captured in each image. Such pixels with the largest or smallest SV value will be explained using Figs. 7 and 8. Figs. 7 and 8 respectively show images at time t consisting of the pixel with the largest SV value and the pixel with the smallest SV value. n7 and 8 are images at time t n The image captured at time t n The image taken before time t n 7 and 8, the pixel with the maximum SV value and the pixel with the minimum SV value are extracted and arranged. In this embodiment, the calculation unit 40 calculates a difference value, which is the difference between the SV value shown in FIG. 7 and the SV value shown in FIG. 8, for each pixel shown in FIG. 7 and FIG. 8. The difference value calculated in this way will be explained using FIG. 9. FIG. 9 is a diagram showing the difference value of each pixel calculated by the calculation unit 40 according to this embodiment. The difference value calculated in this way is calculated at time t n From time t n-5 This shows the amount of change in SV value during the period from 1 to 2000. It is assumed that pixels with a large amount of change in SV value correspond to the blinking area.

[0038] The detection unit 50 is a processing unit that detects blinking areas included in multiple images based on the difference values ​​calculated by the calculation unit 40. In this embodiment, the probability that the first pixel and the second pixel are included in the blinking area is calculated based on the difference values, and whether the first pixel and the second pixel are included in the blinking area is determined based on the probability. A method for calculating the probability in the detection unit 50 according to this embodiment will be described with reference to FIGS. 10 and 11. FIG. 10 is a graph that defines the relationship between the difference value of the SV values ​​of the first pixel and the second pixel according to this embodiment and the probability that the first pixel and the second pixel are included in the blinking area. FIG. 11 is a probability map that shows the probability that each pixel of the image is included in the blinking area. In FIG. 11, the probability is shown in grayscale.

[0039] In this embodiment, a BBF (Binary Bayes Filter) is used to generate a probability map indicating the probability that each pixel is included in a blinking area, as shown in Fig. 11. In calculating the probability distribution using the BBF, the following equation (2) is used.

[0040]

number

[0041] In equation (2), time t n The logarithmic odds at are expressed by the following equation (3):

[0042]

number

[0043] In addition, in equation (2), the probability distribution of the state changing with time is expressed by the following equation (4).

[0044]

number

[0045] Here, time t n Prior probability distribution at t n-1 The posterior probability distribution at is expressed by the following equation (5).

[0046]

number

[0047] Also, at time t n The probability estimated from the prior probability distribution at is represented by x.

[0048] In this embodiment, the difference value calculated by the calculation unit 40 is converted into a probability using the sigmoid function shown in Fig. 10, and is input to the above equation (2). By performing such calculations for each pixel, a probability map such as that shown in Fig. 11 is obtained. The detection unit 50 detects, as a blinking region, an area in the probability map such as that shown in Fig. 11 where the probability is equal to or greater than a predetermined threshold.

[0049] In this embodiment, taking into consideration the influence of the logarithmic odds and probability values ​​on the detection results, the ranges of the logarithmic odds and probability values ​​are set to the ranges expressed by the following equations (6) and (7).

[0050]

number

[0051]

number

[0052] The detection unit 50 may identify the detected blinking region. In other words, the detection unit 50 may identify what is blinking in the blinking region. A method for identifying a blinking region will be described with reference to Figs. 12 and 13. Fig. 12 is a diagram showing a third example of an image acquired by the acquisition unit 20. Fig. 13 is a probability map showing the probability that each pixel of the image shown in Fig. 12 is included in the blinking region. The probability map shown in Fig. 13 is generated by the detection unit 50 in the same way as Fig. 11.

[0053] 13, blinking area R0 and blinking area D0 are detected. The detection unit 50 can identify each blinking area based on its characteristics. For example, the detection unit 50 can identify the blinking area based on the shape, position, etc. of the blinking area.

[0054] In this embodiment, the detection unit 50 identifies a flashing area as either a turn signal, a traffic signal, or a red warning light. In the example shown in FIG. 13, the flashing area R0, which is made up of multiple adjacent flashing areas, can be identified as the red warning light of an emergency vehicle. The detection unit 50 can also identify a single flashing area D0 shown in FIG. 13 as a turn signal based on its height from the ground, its speed of movement, and the like. The speed of movement of the flashing area can be calculated by detecting the flashing area at a predetermined time interval. The detection unit 50 can also identify a flashing area that is high above the ground and does not move as a traffic signal.

[0055] The driving control unit 70 is a processing unit that controls the driving of the vehicle. In this embodiment, the driving control unit 70 controls driving based on information such as the flashing area detected by the flashing area detection device 100. The driving control unit 70 controls the traveling direction of the vehicle based on, for example, the position of the vehicle and map information stored in the storage unit 80 in addition to the information on the flashing area. The driving control unit 70 may also use information on the surrounding environment acquired by a camera, LiDAR, etc.

[0056] The storage unit 80 stores information for detecting a blinking area, information for estimating the position of the vehicle, etc. For example, the storage unit 80 stores map information including information about the road on which the vehicle is driving, etc. The storage unit 80 may also store information for autonomous driving.

[0057] [2. Flashing area detection method] Next, a blinking area detection method according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the blinking area detection method according to this embodiment. The following describes the blinking area detection method using blinking area detection device 100 (and imaging unit 10) according to this embodiment.

[0058] 14, first, the imaging unit 10 captures an image of the periphery of the autonomous moving body V10 (imaging step S10). In this embodiment, the imaging unit 10 captures a plurality of images at a plurality of different time points. Each of the plurality of images captured by the imaging unit 10 is an image in front of the autonomous moving body V10.

[0059] Next, the acquisition unit 20 of the blinking area detection device 100 acquires a plurality of images captured at a plurality of different time points (acquisition step S20). In this embodiment, the acquisition unit 20 acquires a plurality of images captured in the imaging step S10 from the imaging unit 10.

[0060] Next, the association unit 30 associates pixels representing objects captured in each of the multiple images acquired in the acquisition step S20 between the multiple images (association step S30). The pixels associated in the association step S30 include pixels included in the blinking area. In this embodiment, optical flow is used in the association step S30. More specifically, in the association step S30, the position of the object captured in each of the multiple images at a predetermined time point is predicted using the Farneback algorithm.

[0061] Next, the calculation unit 40 calculates a difference value, which is the difference between the pixel value of a first pixel included in a first image, which is one of the multiple images acquired in the acquisition step S20, and the pixel value of a second pixel included in a second image, which is different from the first image among the multiple images and which is associated with the first pixel in the association step S30 (calculation step S40). The pixel value used for the calculation in the calculation step S40 may include a brightness or an SV value. In this embodiment, the pixel value used for the calculation in the calculation unit 40 is the SV value.

[0062] Next, the detection unit 50 detects a blinking area included in the multiple images based on the difference value calculated in the calculation step S40 (detection step S50). In this embodiment, in the detection step S50, the probability that the first pixel and the second pixel are included in the blinking area is calculated based on the difference value. In the detection step S50, the detection unit 50 may identify the blinking area. For example, the detection unit 50 may identify that the blinking area is any one of a turn signal, a traffic signal, or a red warning light.

[0063] Next, the process returns to the imaging step S10, and each step from the imaging step S10 to the detection step S50 is repeated. Note that, although the flowchart shown in Fig. 14 shows an example in which each step is executed sequentially, multiple steps may be executed simultaneously. For example, the imaging step S10 may be executed while each step from the acquisition step S20 to the detection step S50 is being executed.

[0064] [3. Effects, etc.] The effects of the blinking area detection device 100, the autonomous moving body V10, and the blinking area detection method according to this embodiment will be described.

[0065] The flashing area detection device 100 of this embodiment includes an acquisition unit 20 that acquires multiple images captured at multiple different time points, a matching unit 30 that matches pixels representing objects imaged in each of the multiple images between the multiple images, a calculation unit 40 that calculates a difference value that is the difference between the pixel value of a first pixel included in a first image, which is one of the multiple images, and the pixel value of a second pixel included in a second image, different from the first image, of the multiple images, and matched to the first pixel by the matching unit, and a detection unit 50 that detects flashing areas included in the multiple images based on the difference value, and the pixels matched by the matching unit 30 include pixels included in the flashing area.

[0066] In this manner, the flashing area detection device 100 according to the present embodiment associates pixels representing objects captured in each of the multiple images across the multiple images. The pixels associated by the association unit 30 include pixels included in a flashing area. For example, even when an object cannot be recognized as a vehicle, pixels representing objects that may include a flashing area are associated across the multiple images. Therefore, even when, for example, part of a vehicle is hidden behind another object and cannot be recognized as part of a vehicle, or when the image of the vehicle is too small in the image and cannot be recognized as a vehicle, it is possible to detect a flashing area, such as a turn signal, included in part of the vehicle. In this manner, the flashing area detection device 100 according to the present embodiment can detect a flashing area without identifying the object, thereby reducing missed detections of flashing areas.

[0067] An autonomous moving body V10 according to this embodiment includes a blinking area detection device 100 and an imaging unit 10 that captures a plurality of images.

[0068] According to such an autonomous moving body V10, the same effects as those of the blinking area detection device 100 can be achieved.

[0069] The flashing area detection method of this embodiment includes an acquisition step S20 of acquiring a plurality of images each captured at a plurality of different time points; a correspondence step S30 of matching pixels representing objects imaged in each of the plurality of images between the plurality of images; a calculation step S40 of calculating a difference value which is the difference between the pixel value of a first pixel included in a first image, which is one of the plurality of images, and the pixel value of a second pixel included in a second image, different from the first image, among the plurality of images, and matched to the first pixel in the correspondence step S30; and a detection step S50 of detecting a flashing area included in the plurality of images based on the difference value, wherein the pixels matched in the correspondence step S30 include pixels included in the flashing area.

[0070] Such a blinking area detection method provides the same effects as those of the blinking area detection device 100.

[0071] In the blinking area detection method according to the present embodiment, the pixel values ​​used in calculation step S40 may include brightness.

[0072] By calculating the difference between pixel values ​​including brightness in this way, it is possible to detect pixels with large changes in brightness among multiple pixels. Here, since the brightness changes greatly over time in a blinking area, using such pixel values ​​makes it possible to detect a blinking area more accurately.

[0073] In the blinking area detection method according to the present embodiment, the pixel values ​​used in calculation step S40 may include SV values.

[0074] By calculating the difference between pixel values ​​including the SV value in this way, it is possible to detect pixels with large changes in brightness and saturation between multiple pixels. Here, since the brightness and saturation change significantly over time in a blinking area, using such pixel values ​​makes it possible to detect blinking areas more accurately.

[0075] In the association step S30 of the blinking area detection method according to this embodiment, optical flow may be used.

[0076] This makes it possible to calculate the movement of the object captured in each of the multiple images.

[0077] In the association step S30 of the blinking area detection method according to this embodiment, the position at a given time of an object captured in each of the multiple images may be predicted using the Farneback algorithm.

[0078] This allows a predicted image indicating the position of the object at a predetermined time to be generated based on each of the multiple images. By generating a predicted image for each of the multiple images in this manner, the positions of the object in each of the multiple predicted images can be aligned.

[0079] In the blinking area detection method according to the present embodiment, the blinking period in the blinking area may be not less than 0.5 seconds and not more than 1 second.

[0080] This makes it possible to detect key flashing areas necessary for autonomous driving, such as vehicle turn signals.

[0081] In the detection step S50 of the flashing area detection method according to the present embodiment, the flashing area may be identified. For example, in the detection step S50, the flashing area may be identified as being one of a turn signal, a traffic signal, or a red warning light.

[0082] This makes it possible to grasp traffic conditions more accurately based on the blinking areas.

[0083] In the detecting step S50 of the blinking area detecting method according to this embodiment, the probability that the first pixel and the second pixel are included in the blinking area may be calculated based on the difference value.

[0084] (Variations, etc.) Although the flashing area detection method according to one aspect of the present invention has been described above based on the embodiment, the present invention is not limited to the embodiment. As long as it does not deviate from the spirit of the present invention, various modifications that a person skilled in the art can make to the embodiment may also be included within the scope of the present invention.

[0085] For example, in the above embodiment, the blinking region detection device 100 does not include the imaging unit 10, but the blinking region detection device 100 may include the imaging unit 10.

[0086] The following embodiments may also be included within the scope of one or more aspects of the present invention.

[0087] (1) Some of the components included in the above-mentioned flashing area detection device may be a computer system consisting of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the RAM or hard disk unit. The microprocessor operates in accordance with the computer program to achieve its functions. Here, the computer program is composed of a combination of multiple instruction codes that indicate commands to a computer to achieve a predetermined function.

[0088] (2) Some of the components included in the above-described flashing area detection device may be configured as a single system LSI (Large Scale Integration). A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on a single chip, and specifically, is a computer system configured including a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.

[0089] (3) Some of the components included in the above-mentioned flashing area detection device may be configured as an IC card or a standalone module that can be attached to each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may include the above-mentioned ultra-multifunctional LSI. The IC card or the module achieves its functions when the microprocessor operates according to a computer program. The IC card or the module may be tamper-resistant.

[0090] (4) Furthermore, some of the components included in the above-described flashing area detection device may be the computer program or the digital signal recorded on a computer-readable recording medium, such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray (registered trademark) Disc), semiconductor memory, etc. Alternatively, they may be the digital signal recorded on such a recording medium.

[0091] In addition, some of the components included in the above-mentioned flashing area detection device may transmit the computer program or the digital signal via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc.

[0092] (5) The present invention may be embodied as the above-described blinking area detection method. It may also be embodied as a computer program for causing a computer to implement the above-described blinking area detection method, or as a digital signal comprising the computer program. Furthermore, the present invention may be embodied as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the computer program is recorded.

[0093] (6) The present invention may also be a computer system having a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating in accordance with the computer program.

[0094] (7) The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network, etc.

[0095] (Addendum) Furthermore, the above description discloses the following techniques.

[0096] (Technology 1) A blinking area detection method including: an acquisition step of acquiring a plurality of images captured at a plurality of different time points; a correspondence step of matching pixels representing an object imaged in each of the plurality of images between the plurality of images; a calculation step of calculating a difference value that is the difference between the pixel value of a first pixel included in a first image, which is one of the plurality of images, and the pixel value of a second pixel included in a second image, different from the first image, of the plurality of images, and matched to the first pixel in the correspondence step; and a detection step of detecting a blinking area included in the plurality of images based on the difference value, wherein the pixels matched in the correspondence step include pixels included in the blinking area.

[0097] (Technology 2) The blinking area detection method according to Technology 1, wherein the pixel values ​​include brightness.

[0098] (Technology 3) The blinking area detection method according to Technology 1 or 2, wherein the pixel value includes a product of brightness and saturation.

[0099] (Technology 4) The blinking area detection method according to any one of Technologies 1 to 3, wherein the association step uses optical flow.

[0100] (Technology 5) The blinking area detection method according to Technology 4, wherein in the association step, a position at a predetermined time of the object captured in each of the plurality of images is predicted using a Farneback algorithm.

[0101] (Technology 6) The blinking area detection method according to any one of Technologies 1 to 5, wherein the blinking period in the blinking area is 0.5 seconds or more and 1 second or less.

[0102] (Technology 7) A flickering area detection method according to any one of Technologies 1 to 6, wherein the detecting step identifies the flickering area.

[0103] (Technology 8) A flashing area detection method according to any one of Technologies 1 to 7, wherein in the detection step, the flashing area is identified as being either a turn signal, a traffic signal, or a red warning light.

[0104] (Technology 9) A blinking area detection method according to any one of techniques 1 to 8, wherein in the detection step, a probability that the first pixel and the second pixel are included in the blinking area is calculated based on the difference value.

[0105] (Technology 10) A program for causing a computer to execute the blinking area detection method according to any one of techniques 1 to 9.

[0106] (Technology 11) A blinking area detection device comprising: an acquisition unit that acquires a plurality of images captured at a plurality of different time points; a correspondence unit that associates pixels representing objects captured in each of the plurality of images between the plurality of images; a calculation unit that calculates a difference value that is the difference between the pixel value of a first pixel included in a first image that is one of the plurality of images and the pixel value of a second pixel that is included in a second image that is different from the first image and that is associated with the first pixel by the correspondence unit; and a detection unit that detects blinking areas included in the plurality of images based on the difference value, wherein the pixels associated by the correspondence unit include pixels included in the blinking area.

[0107] (Technology 12) An autonomous moving body including the blinking area detection device according to Technology 11 and an imaging unit that captures the plurality of images. [Industrial Applicability]

[0108] The flashing area detection method according to one aspect of the present invention can be applied to, for example, vehicles for autonomous driving. [Explanation of symbols]

[0109] 10. Imaging unit 20 Acquisition Department 30 Mapping section 40 Calculation Unit 50 Detector 70 Driving control unit 80 Storage section 100 Flashing area detection device V10 autonomous vehicle

Claims

1. an acquisition step of acquiring a plurality of images captured at a plurality of different time points; a matching step of matching pixels representing an object captured in each of the plurality of images among the plurality of images; a calculation step of calculating a difference value that is a difference between a pixel value of a first pixel included in a first image that is one of the plurality of images and a pixel value of a second pixel that is included in a second image that is different from the first image and that is associated with the first pixel in the association step; a detection step of detecting a blinking area included in the plurality of images based on the difference value, The pixels associated in the association step include pixels included in the blinking area. Blinking area detection method.

2. The pixel value includes brightness. The method for detecting a blinking area according to claim 1 .

3. The pixel value includes the product of lightness and saturation. The method for detecting a blinking area according to claim 1 .

4. In the matching step, optical flow is used. The blinking area detection method according to any one of claims 1 to 3.

5. In the step of associating, a Farneback algorithm is used to predict the position of the object captured in each of the plurality of images at a predetermined time. The method for detecting a blinking area according to claim 4 .

6. The blinking period in the blinking region is between 0.5 seconds and 1 second. The blinking area detection method according to any one of claims 1 to 3.

7. In the detecting step, the flashing area is identified. The blinking area detection method according to any one of claims 1 to 3.

8. In the detecting step, the flashing area is identified as being either a turn signal, a traffic signal, or a red warning light. The blinking area detection method according to any one of claims 1 to 3.

9. In the detecting step, a probability that the first pixel and the second pixel are included in the blinking region is calculated based on the difference value. The blinking area detection method according to any one of claims 1 to 3.

10. A method for causing a computer to execute the blinking area detection method according to any one of claims 1 to 3. program.

11. an acquisition unit that acquires a plurality of images captured at a plurality of different time points; a correspondence unit that associates pixels representing an object captured in each of the plurality of images among the plurality of images; a calculation unit that calculates a difference value that is a difference between a pixel value of a first pixel included in a first image that is one of the plurality of images and a pixel value of a second pixel that is included in a second image that is different from the first image and that is associated with the first pixel by the association unit; a detection unit that detects a blinking area included in the plurality of images based on the difference value; The pixels associated by the association unit include pixels included in the blinking region. Flashing area detection device.

12. The blinking area detection device according to claim 11; an imaging unit that captures the plurality of images; Autonomous mobile body.

Images