Image processing device, image processing method and program

Abstract

To improve the accuracy of detecting a light-emitting state of a traffic light at a low cost.SOLUTION: An image processing apparatus mounted on a vehicle includes a control unit. The control unit is configured to project, on a virtual plane in a real space, central coordinates of a rectangular region extracted as a traffic light through image recognition on a camera image. The control unit is configured to sample pixels of the camera image, the pixels corresponding to sampling points of a virtual rectangular region, on the virtual rectangular region of a determined size centered on the projected central coordinates, to generate a signal image. The control unit is configured to perform a predetermined color space adjustment on the signal image to extract pixels corresponding to color components indicating a light-emitting state of the traffic light, from the signal image subjected to the color space adjustment. The control unit is configured to determine a light-emitting state of the traffic light on the basis of positional relationships between the extracted pixels.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The disclosed embodiments relate to an image processing device, an image processing method, and a program. [Background technology]

[0002] Conventionally, a technology has been proposed for determining whether a vehicle has run a red light, based on the light status (green, yellow, red, etc.) of a traffic light detected from information received by an optical beacon or an image captured by a camera, and the vehicle's operating status (e.g., braking) (see Patent Document 1, for example).

[0003] In these technologies, when determining the illumination state of a traffic light using an image captured by a camera, it is necessary to detect the traffic light itself in the image. In such cases, a known method is to use a machine learning algorithm such as deep learning to extract the traffic light to be detected as a rectangular area called a bounding box.

[0004] Within the extracted rectangular area, the shape of the luminous area (circle, ellipse) is detected. Then, for example, by identifying the position of the detected shape within the rectangular area, the luminous color of the traffic light can be determined. For example, if the luminous area is located to the left of the rectangular area, it can be determined that the light is green. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-069051 Summary of the Invention [Problem to be solved by the invention]

[0006] However, there is room for further improvement in the conventional technology in terms of improving the accuracy of determining the light emission state of a traffic light at low cost.

[0007] For example, when detecting the shape of the aforementioned luminous area, the Hough transform is a standard method for detecting a "circular" shape. However, this method is known to have high processing costs, making it difficult to implement in environments with limited computing resources, such as in-vehicle devices.

[0008] Furthermore, if the accuracy of the extracted position of the rectangular area itself is low, determining the light emission state of the traffic light based solely on the position of the light emission area within the rectangular area may result in an erroneous determination.

[0009] One aspect of the embodiment has been made in consideration of the above, and aims to provide an image processing device, an image processing method, and a program that can improve the accuracy of determining the illumination state of traffic lights at low cost. [Means for solving the problem]

[0010] According to one aspect of the embodiment, an image processing device is mounted on a vehicle and includes a control unit. The control unit projects the center coordinates of a rectangular area extracted as a traffic light by image recognition of a camera image onto a virtual plane in real space. The control unit generates a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of ​​a predetermined size centered on the projected center coordinates. The control unit also performs a predetermined color space adjustment on the traffic light image and extracts pixels corresponding to each color component indicating the traffic light state from the traffic light image after the color space adjustment. The control unit also determines the traffic light state based on the positional relationship of each extracted pixel. [Effects of the Invention]

[0011] According to one aspect of the embodiment, it is possible to improve the accuracy of determining the light emission state of a traffic signal at low cost. [Brief explanation of the drawings]

[0012] [Figure 1]FIG. 1 is a diagram (part 1) outlining an image processing method according to an embodiment. [Figure 2] FIG. 2 is a diagram (part 2) outlining the image processing method according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of the configuration of the image processing apparatus according to the embodiment. [Figure 4] FIG. 4 is an explanatory diagram (part 1) of the virtual rectangular area generation process. [Figure 5] FIG. 5 is an explanatory diagram (part 2) of the virtual rectangular area generation process. [Figure 6] FIG. 6 is an explanatory diagram (part 3) of the virtual rectangular area generation process. [Figure 7] FIG. 7 is an explanatory diagram of the signal image generation process. [Figure 8] FIG. 8 is an explanatory diagram of the color space adjustment process. [Figure 9] FIG. 9 is an explanatory diagram (part 1) of the color feature extraction process. [Figure 10] FIG. 10 is an explanatory diagram (part 2) of the color feature extraction process. [Figure 11] FIG. 11 is an explanatory diagram (part 1) of the position feature extraction process. [Figure 12] FIG. 12 is an explanatory diagram (part 2) of the position feature extraction process. [Figure 13] FIG. 13 is an explanatory diagram (part 3) of the position feature extraction process. [Figure 14] FIG. 14 is an explanatory diagram (part 4) of the position feature extraction process. [Figure 15] FIG. 15 is an explanatory diagram (part 5) of the position feature extraction process. [Figure 16] FIG. 16 is an explanatory diagram (part 1) of the determination process. [Figure 17] FIG. 17 is an explanatory diagram (part 2) of the determination process. [Figure 18] FIG. 18 is an explanatory diagram (part 3) of the determination process. [Figure 19] FIG. 19 is a flowchart showing a processing procedure executed by the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of an image processing device, an image processing method, and a program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0014] In the following description, the image processing device 10 according to the embodiment is assumed to be an in-vehicle device mounted on a vehicle V. The image processing device 10 is a device capable of detecting an object in an image based on an image captured by a camera 3 mounted on the vehicle V, such as a red light violation determination device or a drive recorder. In the present embodiment, the object is assumed to be a traffic light 300.

[0015] First, an overview of an image processing method according to an embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram (part 1) outlining the image processing method according to an embodiment. Fig. 2 is a diagram (part 2) outlining the image processing method according to an embodiment.

[0016] As already mentioned, when determining the light emission state of the traffic light 300 using the image captured by the camera 3, it is necessary to detect the traffic light 300 itself in the image. In such a case, for example, a machine learning algorithm such as deep learning is used to extract the traffic light 300 to be detected as a rectangular area called a bounding box.

[0017] The shape of the light emitting area is detected within the extracted rectangular area. Then, for example, by identifying the position of the detected shape within the rectangular area, the light color of traffic light 300 can be determined.

[0018] For example, as shown in the left diagram of FIG. 1, when a light emitting area is detected near the left side of the rectangular area BB1, it can be correctly determined as a green light.

[0019] Here, when detecting the shape of a light-emitting area, the Hough transform is a standard method for detecting a "circular" shape. However, this method is known to have high processing costs, and is difficult to implement in environments with limited computing resources, such as in-vehicle devices.

[0020] Furthermore, if the accuracy of the extracted position of the rectangular area itself is low, determining the light emitting state of traffic light 300 based solely on the position of the light emitting area within the rectangular area may result in an erroneous determination.

[0021] As an example, as shown in the right figure of Figure 1, if an illuminated area is detected near the center of a rectangular area BB2 extracted at a position shifted to the left of traffic light 300, there is a possibility that the light will be mistakenly determined to be a yellow light even though it is actually a green light.

[0022] Therefore, in the image processing method of the embodiment, the control unit 12 of the image processing device 10 (see Figure 3) projects the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of the camera image onto a virtual plane in real space, generates a signal image by sampling pixels of the camera image corresponding to the sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates, performs a predetermined color space adjustment on the signal image, extracts pixels corresponding to each color component that indicates the light emission state of the traffic light 300 from the signal image after the color space adjustment, and determines the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

[0023] 2, the control unit 12 receives a camera image captured by the camera 3 as an input, extracts a rectangular area (step S1), generates a virtual rectangular area (step S2), generates a signal image (step S3), and adjusts the color space (step S4). Based on the signal image after the color space adjustment, the control unit 12 then extracts a first feature amount (step S5), extracts a second feature amount (step S6), and determines the light emission state (step S7), and outputs the determination result. The first feature amount is a color feature amount. The second feature amount is a position feature amount.

[0024] Step S1, which is outside the dashed rectangle, is an existing technique. In step S1, the control unit 12 extracts the traffic light 300 in the image as a rectangular area BB using a DNN (Deep Neural Network) model trained using a machine learning algorithm.

[0025] In step S2, the control unit 12 converts the center coordinates of the rectangular area BB into world coordinates. The world coordinates are points projected onto a virtual plane at the height of the traffic light (for example, 5 m). This world coordinate is regarded as the center point of the traffic light 300 in real space, and a rectangle having a fixed width in the vertical and horizontal directions from this center point in real space is generated as a "virtual rectangular area." A specific example of step S2 will be described later using Figures 4 to 6.

[0026] In step S3, the control unit 12 generates a signal image corresponding to the virtual rectangular area. The control unit 12 samples the virtual rectangular area as an image of a fixed size (e.g., 25 x 25 pixels) and generates the signal image by referencing pixels on the camera image corresponding to each sampling point. Since the size of the signal image is always the same, a nearby traffic light 300 and a distant traffic light 300 will appear the same, including the sense of scale. By performing steps S2 and S3, the traffic light 300 to be determined can be treated as image information of the same size regardless of its size, position, or vertical and horizontal orientation in the camera image. This contributes to reducing processing costs. A specific example of step S3 will be described later using FIG. 7.

[0027] Furthermore, in step S4, the control unit 12 adjusts (converts) the signal image to a predetermined color space in accordance with the extraction of the first feature amount in the subsequent step S5. In this embodiment, color space adjustment is performed in a so-called YUV format, but the representation format of color information is not limited. For example, color space adjustment may be performed in an HSV format. Differences in characteristics when using the YUV format and the HSV format will be described later with reference to FIG. 8.

[0028] In step S5, the control unit 12 receives the signal image after color space adjustment, sets color flags (hereinafter referred to as "color flags") corresponding to each color component of the traffic light 300 for pixels that satisfy the corresponding conditions for each color component (blue, yellow, red, arrow, and black) of the traffic light 300 indicated by predetermined YUV thresholds (lower and upper limits), and extracts the color information. A specific example of step S5 will be described later with reference to FIGS. 9 and 10.

[0029] Note that the "flag" referred to here refers to an area that stores a truth value indicating the state of each pixel in image processing and a value equivalent to that truth value. In this embodiment, "setting" a flag means storing a value equivalent to a "true value" in the flag. On the other hand, "lowering" a flag means storing a value equivalent to a "false value" in the flag. "Setting" can also be rephrased as "turning on." "Lowering" can also be rephrased as "turning off."

[0030] In step S6, the control unit 12 combines color flags, and when the color flags have a specific positional relationship with each other, sets a position flag (hereinafter referred to as a "position flag" as appropriate) for the pixel and extracts it as position information. A specific example of step S6 will be described later using FIGS. 11 to 15.

[0031] In step S7, the control unit 12 determines the light emission state of the traffic light 300 based on the processing result of step S6. In making this determination, the control unit 12 first determines whether the position flags are concentrated within a predetermined range, and if so, sets a corresponding area flag (hereinafter referred to as an "area flag" as appropriate). The control unit 12 then calculates the sum of each area flag within the signal image, and determines the light emission state of the traffic light 300 based on this sum. By making a determination on an area basis based on the area flag, it is possible to reduce erroneous determinations due to spike-like noise. Specific examples of step S7 will be described later using Figures 16 to 18.

[0032] As described above, the image processing method of the embodiment is an image processing method executed by the image processing device 10 mounted on the vehicle V, and includes projecting the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of a camera image onto a virtual plane in real space, generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates, performing a predetermined color space adjustment on the signal image, extracting pixels corresponding to each color component that indicates the light emission state of the traffic light 300 from the signal image after the color space adjustment, and determining the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

[0033] Therefore, according to the image processing method of the embodiment, it is possible to improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

[0034] An example of the configuration of the image processing device 10 to which the image processing method according to the above-described embodiment is applied will now be described in more detail.

[0035] Fig. 3 is a block diagram showing an example of the configuration of an image processing device 10 according to an embodiment. Note that Fig. 3 shows only components necessary for explaining the features of this embodiment, and omits descriptions of general components.

[0036] In other words, the components shown in Figure 3 are conceptual functional components and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0037] In addition, in the description using FIG. 3, the description of components that have already been described may be simplified or omitted.

[0038] 3, the image processing device 10 includes a storage unit 11 and a control unit 12. The image processing device 10 is also connected to a camera 3 and an output unit 5.

[0039] The camera 3 is mounted on the vehicle V and is provided so as to be able to capture an image of the area in front of the vehicle V. Note that the camera 3 may be, for example, a 360-degree camera that is able to capture an image of the entire periphery of the vehicle V.

[0040] The output unit 5 is an output device for outputting data. The output unit 5 is realized by, for example, a display, a speaker, etc. The output unit 5 may also be an external device that executes information processing based on the data output by the image processing device 10.

[0041] The camera 3 and the output unit 5 are connected to the image processing device 10 via an in-vehicle network such as a Controller Area Network (CAN). The camera 3 and the output unit 5 may also be connected to the image processing device 10 wirelessly via Bluetooth (registered trademark), Wi-Fi (registered trademark), UWB (Ultra Wide Band), or the like.

[0042] The image processing device 10 is a computer mounted on the vehicle V, and as described above, is, for example, a red light ignition determination device. The image processing device 10 executes at least steps S1 to S7 described with reference to FIG.

[0043] The storage unit 11 is realized by a storage device such as a RAM (Random Access Memory) or a flash memory, and in the example of FIG. 3, stores an image recognition AI 11a and processing parameter information 11b.

[0044] The image recognition AI 11a is an AI (Artificial Intelligence) model for image recognition. Specifically, the image recognition AI 11a is a DNN model or the like that is trained using a machine learning algorithm. After being read into the image recognition unit 12a (described later) as a DNN model, the image recognition AI 11a is configured to be able to detect various objects contained in an image captured by the camera 3 when the image is input to the image recognition unit 12a. In this embodiment, the image recognition AI 11a is configured to be able to detect at least a traffic light 300 as an object.

[0045] The processing parameter information 11b is information on various parameters used in each process executed by a virtual rectangular area generation unit 12b, a signal image generation unit 12c, a color space adjustment unit 12d, a color feature extraction unit 12e, a position feature extraction unit 12f, and a determination unit 12g, which will be described later.

[0046] The processing parameter information 11b includes the various flags mentioned above. The flags may be realized by registers or may be represented by program variables. Furthermore, the stored values ​​of the flags do not have to be limited to "0" or "1." Therefore, the size of the flags does not have to be limited to 1 bit.

[0047] The control unit 12 is a controller, and is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing a program according to the embodiment stored in the storage unit 11 using RAM as a work area. The control unit 12 can also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0048] The control unit 12 has an image recognition unit 12a, a virtual rectangular area generation unit 12b, a signal image generation unit 12c, a color space adjustment unit 12d, a color feature extraction unit 12e, a position feature extraction unit 12f, and a determination unit 12g, and realizes or executes the functions and actions of information processing described below.

[0049] The image recognition unit 12a acquires a captured image captured by the camera 3 and inputs the captured image to the image recognition AI 11a that has read the image. Furthermore, the image recognition unit 12a acquires an image recognition result output from the image recognition AI 11a as a result of inputting the captured image, and outputs the acquired image recognition result to the virtual rectangular area generation unit 12b. The image recognition result includes the traffic light 300 extracted as a rectangular area BB. That is, the image recognition unit 12a executes step S1 described above. The camera 3 captures a moving image. In response to this, the image recognition unit 12a extracts a rectangular area BB from each frame of the moving image.

[0050] The virtual rectangular area generating unit 12b executes step S2 described above. The virtual rectangular area generating process executed by the virtual rectangular area generating unit 12b will now be described in more detail with reference to Figs. 4 to 6. Fig. 4 is an explanatory diagram (part 1) of the virtual rectangular area generating process. Fig. 5 is an explanatory diagram (part 2) of the virtual rectangular area generating process. Fig. 6 is an explanatory diagram (part 3) of the virtual rectangular area generating process.

[0051] The virtual rectangular area generation unit 12b converts the center coordinates of the rectangular area BB into world coordinates, regards the world coordinates as the center point of the traffic light 300 in real space, and generates a rectangle having a fixed width in the vertical and horizontal directions in real space from this center point as a virtual rectangular area.

[0052] The virtual rectangular area generation unit 12b first assumes that the height of the traffic light 300 is more or less the same, as shown in Figure 4, and then converts the camera coordinates indicating the center of the rectangular area BB into world coordinates on a virtual plane (hereinafter referred to as the "virtual plane") that assumes the height of the traffic light 300 using camera parameters.

[0053] Specifically, the virtual rectangular area generation unit 12b calculates a virtual vehicle position indicating the position of the vehicle V projected onto a virtual plane of the same height in time-series space based on a rectangular area BB indicating the position of the traffic light 300 in the image 201 captured by the camera 3 of the vehicle V.

[0054] Then, the virtual rectangular area generating unit 12b calculates the coordinates of the traffic light 300 in a virtual plane obtained by fixing the coordinate value of the vertical axis in a coordinate system represented by three mutually perpendicular axes including a vertical axis as world coordinates.

[0055] As shown in FIG. 4, a position in real space is represented by coordinate values ​​of the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis is a vertical axis. The Y-axis is parallel to the traveling direction of the vehicle V in FIG. 4. The X-axis is perpendicular to the Z-axis and Y-axis.

[0056] Here, the center of the rectangular area BB is set to a center point 221. The virtual rectangular area generating unit 12b converts the two-dimensional coordinates (camera coordinates) of the center point 221 in the image 201 into three-dimensional world coordinates using the camera parameters. Then, the distance to the center point 221 in the world coordinates is calculated as the distance (Dist) from the vehicle V to the traffic light 300.

[0057] The camera parameters include internal and external parameters of the camera 3. The internal parameters include dot pitch, focal length, distortion (elevation image height TBL), etc. The external parameters include the coordinate values ​​of the X-axis, Y-axis, and Z-axis of the shooting position of the camera 3, and values ​​(PAN, ROLL, TILT) that represent the attitude of the camera 3.

[0058] Furthermore, as a parameter separate from the camera parameters, the virtual rectangular area generating unit 12b refers to a predetermined height from the road surface of the traffic light 300. The height of the traffic light 300 is set within the range of 5.0 m to 5.5 m, for example.

[0059] The camera parameters and the height of the traffic light 300 are stored in, for example, the processing parameter information 11b described above.

[0060] By converting to world coordinates in this way, there is an advantage that the distance to the traffic light 300 can be calculated inexpensively using a monocular camera.

[0061] Then, the virtual rectangular area generating unit 12b regards the world coordinates as the center point of the traffic light 300 in the real space, and generates a rectangle having a fixed width in the vertical and horizontal directions from this center point in the real space as the virtual rectangular area VB.

[0062] As a result, as shown in Figure 5, in the case of rectangular area BB, the rectangular area BB1 of the closer traffic light 300 is larger and the rectangular area BB2 of the farther traffic light 300 is smaller, but in the case of virtual rectangular area VB, the sizes of the virtual rectangular areas VB1 and VB2 are the same regardless of the distance from the traffic light 300.

[0063] The virtual rectangular area generation unit 12b determines the size of the virtual rectangular area VB based on the size (e.g., 1250 mm × 450 mm) of a standard traffic light 300 in real space. In this embodiment, the virtual rectangular area generation unit 12b determines both horizontal traffic lights and vertical traffic lights, and therefore generates the virtual rectangular area VB using a square as a reference shape, one side of which is the long side (e.g., 1250 mm) of the traffic light 300 in real space, as shown in Fig. 6. Furthermore, as shown in the same figure, the virtual rectangular area generation unit 12b uses the above-mentioned reference shape expanded by a predetermined expansion rate (e.g., 1.1 times) to accommodate variations in the size and position of the input rectangular area BB.

[0064] Returning to the explanation of Fig. 3, the signal image generating unit 12c executes step S3 described above. Here, the signal image generating process executed by the signal image generating unit 12c will be explained in more detail with reference to Fig. 7. Fig. 7 is an explanatory diagram of the signal image generating process.

[0065] The signal image generation unit 12c generates a signal image corresponding to the virtual rectangular area VB. As already described, the signal image generation unit 12c samples the virtual rectangular area VB as an image of the same size (e.g., 25×25 pixels) and generates the signal image by referring to pixels on the camera image corresponding to each sampling point.

[0066] To refer to pixels on the camera image, the inverse conversion of the above-mentioned conversion from camera coordinates to world coordinates is performed. As for the sampling direction, as shown in FIG. 7, horizontal traffic lights are scanned from the upper left to the lower right, and vertical traffic lights are scanned from the lower left to the upper right. In other words, the signal image generation unit 12c generates a signal image by sampling in a sampling direction in which the pixels of the camera image are sequentially scanned along the longitudinal direction of the traffic light 300 in the camera image. As a result, as shown in the same figure, the signal image after sampling looks the same whether it is a horizontal traffic light or a vertical traffic light, including the sense of scale and orientation within the image.

[0067] Therefore, the traffic light 300 to be judged can always be treated as image information of the same size regardless of its size, position, or vertical / horizontal orientation in the camera image, which helps to reduce processing costs.

[0068] Furthermore, by reducing the size of the signal image (that is, increasing the sampling interval in real space), the processing cost in each subsequent process can be reduced.

[0069] Returning to the explanation of Fig. 3, the color space adjustment unit 12d executes step S4 described above. Here, the color space adjustment process executed by the color space adjustment unit 12d will be further explained with reference to Fig. 8. Fig. 8 is an explanatory diagram of the color space adjustment process.

[0070] The color space adjustment unit 12d adjusts the generated signal image to a predetermined color space in accordance with the color feature extraction process performed by the color feature extraction unit 12e at the subsequent stage. In this embodiment, the color space adjustment unit 12d performs color space adjustment in the YUV format as already described.

[0071] As shown in Fig. 8, the color space adjustment unit 12d can perform color space adjustment using the HSV format in addition to the YUV format. As shown in the figure, the YUV format uses a rectangular coordinate system for color representation, with U and V representing colors, which can make color separation difficult in some cases, but has the advantage of low processing costs. On the other hand, the HSV format uses a polar coordinate system for color representation, with H and S representing colors, which is a sector representation, making color separation easier than the YUV format, but has the advantage of high processing costs.

[0072] In this embodiment, the YUV format is adopted with emphasis on processing cost, but it is possible to select the HSV format or other formats as appropriate depending on the processing cost, such as whether the calculation resources of the image processing device 10 are acceptable.

[0073] Returning to the explanation of FIG. 3, the color feature extraction unit 12e executes step S5 described above. Here, the color feature extraction process executed by the color feature extraction unit 12e will be explained in more detail with reference to FIGS. 9 and 10. FIG. 9 is an explanatory diagram (part 1) of the color feature extraction process. Also, FIG. 10 is an explanatory diagram (part 2) of the color feature extraction process.

[0074] The color feature extraction unit 12e takes the signal image after color space adjustment as input, and extracts it as color information by setting color flags corresponding to each color component for pixels that satisfy the corresponding conditions for each color component of the traffic light 300 indicated by predetermined YUV thresholds.

[0075] Specifically, for example, in the processing parameter information 11b, as shown in Figure 9, the relevant conditions for each color component of the traffic light 300, namely blue, yellow, red, arrow, and black, are specified by predetermined lower and upper limits of Y, U, and V.

[0076] The color feature extraction unit 12e extracts pixels that satisfy the relevant conditions for each color component of the traffic light 300 in the signal image after color space adjustment. Then, the color feature extraction unit 12e sets a color flag corresponding to the relevant color component for each extracted pixel.

[0077] Then, as shown in FIG. 10, for example, in the case of a blue light, at least each pixel that satisfies the corresponding condition for the blue component and each pixel that satisfies the corresponding condition for the black component are extracted from the signal image, and a green light color flag is set for the former, and a non-emission signal color flag is set for the latter.

[0078] Similarly, for example, in the case of a yellow signal, at least each pixel that satisfies the corresponding condition for the yellow component and each pixel that satisfies the corresponding condition for the black component are extracted from the signal image, and a yellow signal color flag is set for the former, and a non-emission signal color flag is set for the latter.

[0079] Similarly, for example, in the case of a red traffic light, at least each pixel that satisfies the corresponding condition for the red component and each pixel that satisfies the corresponding condition for the black component are extracted from the signal image, and a red signal color flag is set for the former, and a non-emission signal color flag is set for the latter.

[0080] Similarly, for example, in the case of an arrow signal, at least each pixel that satisfies the corresponding condition for the red component and each pixel that satisfies the corresponding condition for the arrow component are extracted from the signal image, and a red signal color flag is set for the former, and an arrow signal color flag is set for the latter.

[0081] Similarly, for example, in the case of a non-emission signal, at least pixels that satisfy the corresponding conditions for the black component are extracted from the signal image, and a non-emission signal color flag is set for these.

[0082] Returning to the description of Fig. 3, the position feature amount extraction unit 12f executes the above-mentioned step S6. Here, the position feature amount extraction process executed by the position feature amount extraction unit 12f will be described in more detail with reference to Figs. 11 to 15.

[0083] FIG. 11 is an explanatory diagram (part 1) of the position feature amount extraction process. FIG. 12 is an explanatory diagram (part 2) of the position feature amount extraction process. FIG. 13 is an explanatory diagram (part 3) of the position feature amount extraction process. FIG. 14 is an explanatory diagram (part 4) of the position feature amount extraction process. FIG. 15 is an explanatory diagram (part 5) of the position feature amount extraction process.

[0084] The position feature extraction unit 12f combines the color flags extracted by the position feature extraction unit 12f, and when the color flags have a specific positional relationship with each other, sets a position flag for that pixel and extracts it as position information.

[0085] As shown in FIG. 11, if there is a black component indicated by the non-emission signal color flag to the right of the blue component indicated by the green signal color flag, and if there is also a black component further to the right, the position feature extraction unit 12f estimates it as a green signal, and sets a green signal position flag for each pixel for which such a green signal color flag has been set.

[0086] Furthermore, as shown in FIG. 12, if there is a black component to the right of the yellow component indicated by the yellow light color flag and also a black component to the left, the position feature extraction unit 12f estimates it as a yellow light and sets a yellow light position flag for each pixel for which the yellow light color flag has been set.

[0087] Furthermore, as shown in FIG. 13, if there is a black component to the left of the red component indicated by the red light color flag, and if there is another black component further to the left, the position feature extraction unit 12f estimates it as a red light, and sets a red light position flag for each pixel for which the red light color flag has been set.

[0088] Furthermore, as shown in FIG. 14, if there is a black component to the right of the black component indicated by the non-emission signal color flag and if there is another black component further to the right, the position feature extraction unit 12f estimates it as a non-emission signal and sets a non-emission signal position flag for each pixel for which such a non-emission signal color flag has been set.

[0089] Furthermore, as shown in FIG. 15, for an arrow signal, if there is an arrow component indicated by an arrow signal color flag within a predetermined range below the center of gravity of the collection of red light position flags extracted in FIG. 13, the position feature extraction unit 12f estimates that this is the position of an arrow signal, and sets an arrow signal position flag for each pixel for which such arrow signal color flag is set.

[0090] Returning to the description of Fig. 3, the determination unit 12g executes the above-mentioned step S7. Here, the determination process executed by the determination unit 12g will be described in more detail with reference to Figs.

[0091] Fig. 16 is an explanatory diagram (part 1) of the determination process. Fig. 17 is an explanatory diagram (part 2) of the determination process. Fig. 18 is an explanatory diagram (part 3) of the determination process. Fig. 17 is a flowchart showing the processing procedure of the first determination process executed by the determination unit 12g, and Fig. 18 is a flowchart showing the processing procedure of the second determination process executed by the determination unit 12g.

[0092] The determination unit 12g determines the light emission state of the traffic light 300 based on the result of extraction of the position information by the position feature amount extraction unit 12f.

[0093] When determining the light emission state of the traffic light 300, the determining unit 12g first determines whether or not the position flags are gathered within a predetermined range, and if they are gathered, sets an area flag accordingly.

[0094] As shown in Figure 16, for the green light position flag, yellow light position flag, red light value flag, and arrow signal position flag, the judgment unit 12g sets the confirmation range to, for example, 2 x 2 pixels of the signal image, and sets each signal area flag if the number of flags in the confirmation range is 3 or more.

[0095] Furthermore, the determining unit 12g sets the non-light emitting signal position flag to a checking range of, for example, 3×3 pixels of the signal image, and sets a non-light emitting signal area flag if the number of flags in the checking range is 6 or more.

[0096] Then, the determination unit 12g calculates the sum of the area flags in the signal image, and determines the light emission state of the traffic light 300 based on the calculated sum.

[0097] Specifically, for green, yellow, red and non-light emitting signals, the judgment unit 12g judges whether any of the sums of the green light area flag, yellow light area flag or red light area flag is greater than or equal to a predetermined value (here, for example, 1) as shown in FIG. 17 (step S11).

[0098] Here, if any of the sums of the green light area flag, yellow light area flag, or red light area flag is 1 or more (step S11, Yes), the judgment unit 12g determines the light emission state (green light, yellow light, or red light) corresponding to the flag with the largest sum among the green light area flag, yellow light area flag, or red light area flag as the judgment result (step S12), and terminates the first judgment process.

[0099] In addition, if the sum of the green light area flags, yellow light area flags, or red light area flags is less than 1 (step S11, No), the judgment unit 12g judges whether the sum of the non-luminous signal area flags is greater than or equal to a predetermined value (here, for example, 2) (step S13).

[0100] If the sum of the non-luminous signal area flags is 2 or more (step S13, Yes), the determination unit 12g determines that the signal is a non-luminous signal (step S14) and ends the first determination process. If the sum of the non-luminous signal area flags is less than 2 (step S13, No), the determination unit 12g determines that the determination is impossible (step S15) and ends the first determination process.

[0101] Furthermore, for the arrow signal, the judgment unit 12g judges whether the sum of the arrow signal area flags is equal to or greater than a predetermined value (here, for example, 1) as shown in FIG. 18, independently of the first judgment process of FIG. 17 (step S21).

[0102] If the sum of the arrow signal area flags is 1 or more (step S21, Yes), the determination unit 12g sets the arrow signal flag (step S22) and ends the second determination process. If the sum of the arrow signal area flags is less than 1 (step S21, No), the determination unit 12g does not set the arrow signal flag (step S23) and ends the second determination process.

[0103] Returning to the explanation of Fig. 3, the determination unit 12g then outputs the determination results of the first determination process and the second determination process to the output unit 5. If the image processing device 10 is a red light running determination device, the determination unit 12g determines whether or not the vehicle V has run a red light based on the image recognition result of the image recognition unit 12a and the determination results of the first determination process and the second determination process.

[0104] In such a case, the determination unit 12g estimates the behavior of the vehicle V based on the trajectory of each object image-recognized by the image recognition unit 12a, for example. The determination unit 12g also estimates the trajectory of the traffic light 300 relative to the vehicle V based on the image-recognized traffic light 300 and the distance to the traffic light 300 calculated by the virtual rectangular area generation unit 12b.

[0105] In addition, the judgment unit 12g judges whether the vehicle V has ignored a traffic light based on the estimated behavior of the vehicle V, the trajectory of the traffic light 300, and the illumination state of the traffic light 300, which is the judgment result of the first judgment process and the second judgment process.

[0106] For example, if the traffic light 300, which indicates whether or not there is a right of way in the direction of travel of the vehicle V, is red, and the vehicle V continues to pass for a predetermined time or longer at a predetermined speed or faster, the determination unit 12g determines that the vehicle V has run a red light. Then, the determination unit 12g outputs the determination result to the output unit 5.

[0107] In addition, when the image processing device 10 constitutes part of the function of a drive recorder, the determination unit 12g may notify the determination result that the vehicle V has ignored a traffic light as an event and cause the drive recorder to record the event.

[0108] Next, a processing procedure executed by the image processing device 10 according to the embodiment will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the processing procedure executed by the image processing device 10 according to the embodiment. The processing procedure shown in Fig. 19 is repeated, for example, while the vehicle V is in operation.

[0109] 19, the control unit 12 of the image processing device 10 acquires a camera image from the camera 3 (step S101). Then, the control unit 12 extracts a rectangular area BB of the traffic light 300 by image recognition of the acquired camera image (step S102).

[0110] Then, the control unit 12 generates a virtual rectangular area VB based on the rectangular area BB (step S103), and generates a signal image based on the virtual rectangular area VB (step S104).

[0111] Then, the control unit 12 adjusts the generated signal image to a predetermined color space (step S105), and extracts a first feature amount (color) from the signal image after the color space adjustment (step S106).

[0112] The control unit 12 then extracts a second feature (position) based on the extracted first feature (step S107). The control unit 12 then executes the process of determining the light emission state of the traffic light 300 shown in Fig. 17 and Fig. 18 based on the extracted second feature (step S108). The control unit 12 then outputs the determination result (step S109) and ends the process.

[0113] As described above, the image processing device 10 according to the embodiment is an image processing device mounted on a vehicle V and includes a control unit 12. The control unit 12 projects the central coordinates of a rectangular area BB, extracted as a traffic light 300 by image recognition of a camera image, onto a virtual plane in real space. The control unit 12 then generates a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected central coordinates. The control unit 12 then performs a predetermined color space adjustment on the traffic light image and extracts pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment. The control unit 12 then determines the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

[0114] Therefore, the image processing device 10 according to the embodiment can improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

[0115] Furthermore, the control unit 12 generates a virtual rectangular area VB using a square as a reference shape, the square having the long side of a standard traffic light 300 in real space as one side.

[0116] Therefore, the image processing device 10 according to the embodiment can determine both horizontal traffic lights and vertical traffic lights.

[0117] Furthermore, the control unit 12 generates a virtual rectangular area VB by expanding the reference form at a predetermined expansion rate.

[0118] Therefore, the image processing apparatus 10 according to the embodiment can accommodate variations in the size and position of the input rectangular area BB.

[0119] Furthermore, the control unit 12 generates the signal image by sampling in a sampling direction in which the pixels of the camera image are sequentially scanned along the longitudinal direction of the traffic light 300.

[0120] Therefore, according to the image processing device 10 of the embodiment, the signal image after sampling appears the same whether it is a horizontal traffic light or a vertical traffic light, including the sense of scale and orientation within the image. This allows the traffic light 300 to be determined to be treated as image information with the same sense of size regardless of its size, position, or vertical and horizontal orientation within the camera image, which contributes to reducing processing costs.

[0121] Furthermore, the control unit 12 extracts, from the signal image after the color space adjustment, pixels that satisfy the corresponding conditions for each color component of the traffic light 300 indicated by the predetermined threshold value in the color expression format of the color space adjustment.

[0122] Therefore, according to the image processing device 10 according to the embodiment, it is possible to extract pixels that satisfy the corresponding conditions for each color component of the traffic light 300 in accordance with the color expression format of the color space adjustment.

[0123] Furthermore, the control unit 12 uses the YUV format as the color representation format.

[0124] Therefore, the image processing device 10 according to the embodiment can perform image processing at reduced processing costs, which can be performed even in an environment with limited computing resources, such as an in-vehicle device.

[0125] In addition, the control unit 12 estimates the position of the arrow signal as the position of the pixel that satisfies the corresponding condition for the color component of the arrow signal within a specified range below the center of gravity of each pixel that satisfies the corresponding condition for the red component of the traffic light 300.

[0126] Therefore, according to the image processing device 10 according to the embodiment, the position of the arrow signal can be estimated based on the relative positional relationship with the center of gravity position of each pixel that satisfies the corresponding condition for the red component.

[0127] In addition, when determining the light emission state of traffic light 300 based on the positional relationship of each pixel extracted as corresponding to each color component of traffic light 300, control unit 12 determines whether each pixel for each color component is concentrated within a predetermined range, and if so, sets an area flag for each color component, and determines the light emission state of traffic light 300 based on the sum of each of the area flags set within the signal image.

[0128] Therefore, the image processing device 10 according to the embodiment can suppress erroneous determinations due to spike noise.

[0129] The image processing method according to the embodiment is an image processing method executed by an image processing device 10 mounted on a vehicle V, and includes projecting the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of a camera image onto a virtual plane in real space. The image processing method according to the embodiment also includes generating a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates. The image processing method according to the embodiment also includes performing a predetermined color space adjustment on the traffic light image and extracting pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment. The image processing method according to the embodiment also includes determining the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

[0130] Therefore, according to the image processing method of the embodiment, it is possible to improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

[0131] The program according to the embodiment also causes the image processing device 10 (corresponding to an example of a "computer") to project the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of a camera image onto a virtual plane in real space. The program according to the embodiment also causes the image processing device 10 to generate a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates. The program according to the embodiment also causes the image processing device 10 to perform a predetermined color space adjustment on the traffic light image and extract pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment. The program according to the embodiment also causes the image processing device 10 to determine the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

[0132] Therefore, according to the program of the embodiment, it is possible to improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

[0133] In the above-described embodiment, the illumination state of the traffic light 300 and whether the traffic light has been ignored are determined based on image recognition, but it is of course possible to appropriately combine sensor data from various sensors mounted on the vehicle V. For example, the behavior of the vehicle V may be estimated using sensor values ​​from a steering sensor or an acceleration sensor, and the vehicle speed may be obtained using sensor values ​​from a speed sensor.

[0134] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0135] 3 Camera 5 Output section 10 Image processing device 11 Storage section 11a Image recognition AI 11b Processing parameter information 12 Control Unit 12a Image recognition unit 12b Virtual rectangular area generation unit 12c Signal image generation unit 12d color space adjustment section 12e Color feature extraction unit 12f Position feature extraction unit 12g Judgment part BB rectangular area V vehicle VB Virtual rectangular area

Claims

1. An image processing device mounted on a vehicle, the image processing device having a control unit, The control unit The center coordinates of a rectangular area extracted as a traffic light by image recognition of the camera image are projected onto a virtual plane in real space. generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of ​​a determined size centered on the projected central coordinates; performing a predetermined color space adjustment on the signal image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light from the signal image after the color space adjustment; determining the light emission state of the traffic light based on the positional relationship of each extracted pixel; Image processing device.

2. The control unit The virtual rectangular area is generated using a square as a reference shape, the square having the long side of a standard traffic light in real space as one side. The image processing device according to claim 1 .

3. The control unit generating the virtual rectangular area by expanding the reference shape at a predetermined expansion rate; The image processing device according to claim 2 .

4. The control unit generating the signal image by sampling in a sampling direction in which pixels of the camera image are sequentially scanned along the longitudinal direction of the traffic light; 4. The image processing device according to claim 1, 2 or 3.

5. The control unit extracting, from the signal image after the color space adjustment, pixels that satisfy a corresponding condition for each color component of the traffic light indicated by a predetermined threshold value in a color representation format of the color space adjustment; 5. The image processing device according to claim 1.

6. The control unit The YUV format is used as the color representation format. The image processing device according to claim 5 .

7. The control unit The position of a pixel that satisfies the corresponding condition of the color component of an arrow signal within a determined range below the center of gravity of each pixel that satisfies the corresponding condition of the red component of the traffic light is estimated to be the position of the arrow signal.

7. The image processing device according to claim 5 or 6.

8. The control unit When determining the light emission state of the traffic light based on the positional relationship of each pixel extracted as corresponding to each color component of the traffic light, it is determined whether or not each pixel for each color component is concentrated within a predetermined range, and if so, an area flag for each color component is set, and the light emission state of the traffic light is determined based on the sum of each of the set area flags within the signal image. The image processing device according to any one of claims 1 to 7.

9. An image processing method executed by an image processing device mounted on a vehicle, comprising: projecting the center coordinates of a rectangular area extracted as a traffic light by image recognition of the camera image onto a virtual plane in real space; generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of ​​a determined size centered on the projected central coordinates; performing a predetermined color space adjustment on the signal image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light from the signal image after the color space adjustment; determining a light emission state of the traffic light based on the positional relationship of each extracted pixel; An image processing method comprising:

10. Projecting the center coordinates of a rectangular area extracted as a traffic light by image recognition of the camera image onto a virtual plane in real space; generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of ​​a determined size centered on the projected central coordinates; performing a predetermined color space adjustment on the signal image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light from the signal image after the color space adjustment; determining the light emission state of the traffic light based on the positional relationship of each extracted pixel; A program that causes a computer to execute the following.

Images