Image processing device and image processing method
The image processing device transforms images to match specific conditions, addressing the challenge of varied environmental effects in object detection by enhancing model performance across different conditions.
Patent Information
- Application Number
- JP2021030975
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-26
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2041-02-26
AI Technical Summary
Existing object detection models struggle with accurately detecting objects in images taken under varying environmental conditions due to the need for extensive and costly data collection, and performance drops when applied to conditions different from their training data.
An image processing device and method that converts images taken under non-specific conditions into equivalent images under specific conditions using image processing techniques, such as brightness normalization and color adjustments, allowing a single model to perform well across various conditions.
Enables the use of a single high-performance estimation model for images taken under diverse conditions by transforming them to match training data, reducing the need for multiple models and improving estimation accuracy.
Smart Images

Figure 0007590734000004 
Figure 0007590734000005 
Figure 0007590734000006
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device and an image processing method. [Background technology]
[0002] There is an object detection technology for an image using machine learning such as deep learning. In this technology, for example, an estimation model is generated that estimates an object contained in an image from an image, which is learned using a pair of an image and the name of an object contained in the image as training data. In order to generate a high-performance estimation model corresponding to images taken under various environmental conditions, it is desirable that the training data includes images taken under various environmental conditions. In addition, in order to generate a high-performance estimation model, it is desirable to use as many images as training data. Examples of environmental conditions include conditions due to the time of day, location, weather, angle of view, etc. when an image is taken. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-187360 A [Patent Document 2] JP 2020-91453 A [Patent Document 3] JP 2007-333458 A [Patent Document 4] JP 2009-289189 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, it is costly to comprehensively collect images taken under various environmental conditions. For example, in outdoor daytime images and nighttime images taken at the same location, even if the shapes of objects (man-made objects such as road signs, vehicles, and buildings, natural objects such as plants, animals, people, topography, and rocks) reflected in the images are the same, the colors and brightness of the objects may be different. Even if the shapes of objects are the same, if the colors and brightness of the objects are different, the objects may not be detected accurately. Therefore, when generating an estimation model, it is necessary to prepare and learn a large number of images of objects with different colors and brightness as training data.
[0005] In addition, in general, an estimation model using only images taken under a certain environmental condition (e.g., daytime) as training data performs better for images taken under the certain environmental condition than an estimation model using images taken under various environmental conditions as training data. However, it is difficult for an estimation model using only images taken under a certain environmental condition as training data to perform estimation for images taken under other environmental conditions (e.g., nighttime). Therefore, in this case, in order to handle images taken under various environmental conditions, an estimation model must be prepared for each environmental condition, which is costly. Therefore, it is required to estimate images taken under various environmental conditions using an estimation model that has high performance and corresponds to one (or a smaller number) specific environmental condition.
[0006] An object of the present invention is to provide an image processing device and an image processing method for converting an image into an image captured under specific conditions. [Means for solving the problem]
[0007] In order to solve the above problems, the present invention employs the following solutions. That is, the first aspect is A converted image is obtained by performing image processing on a first image captured under a first condition. an image processing unit that generates the image processing unit performs the image processing until an absolute value of a difference between a first value calculated based on the converted image and a second value calculated based on a plurality of second images captured under second conditions different from the first conditions becomes less than a threshold value. Image processing device.
[0008] The disclosed aspects may be realized by a program being executed by an information processing device. That is, the disclosed configuration may be specified as a program for causing an information processing device to execute the processes executed by each of the means in the above-mentioned aspects, or a recording medium readable by the information processing device on which the program is recorded. The disclosed configuration may also be specified by a method in which the information processing device executes the processes executed by each of the above-mentioned means. The disclosed configuration may also be specified as a system including an information processing device that executes the processes executed by each of the above-mentioned means. The information processing device is, for example, a computer. The computer may also be called a personal computer or a server. Effect of the Invention
[0009] According to the present invention, it is possible to provide an image processing device and an image processing method for converting an image into an image captured under specific conditions. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3] FIG. 3 is a diagram illustrating an example of an overall operational flow of image processing in the image processing apparatus of the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of an operational flow of the first image processing. [Diagram 5] FIG. 5 is a diagram illustrating an example of an operational flow of the second image processing. [Figure 6] FIG. 6 is a diagram illustrating an example of an operation flow of the third image processing. [Figure 7] FIG. 7 is a diagram illustrating an example of an operational flow of the fourth image processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The configurations of the following embodiments are merely examples, and the present invention is not limited to the configurations of the embodiments.
[0012] <Embodiment> (Configuration example) FIG. 1 is a diagram showing an example of the configuration of an image processing device according to this embodiment. The image processing device 100 includes an image processing unit 102, an input unit 104, an output unit 106, a communication unit 108, and a storage unit 110. The image processing device 100 performs image processing to convert an image (original image) into an image captured under certain environmental conditions. Examples of the environmental conditions include conditions due to the time of day, location, weather, angle of view, and road congestion (traffic jam) when capturing an image. The environmental conditions may be a combination of these. The environmental conditions are not limited to these. The environmental conditions are an example of a condition. The original image to be subjected to image processing is, for example, an image captured by a camera mounted on a vehicle traveling on a road.
[0013] The original image is, for example, an image taken under environmental conditions other than the specific environmental conditions. The converted image is, for example, an image that can be used as an image of training data when generating an estimation model that is trained (machine learning) using data including an image taken under the specific environmental conditions as training data. The estimation model (trained estimation model) is, for example, an estimation model that is trained using a pair of an image taken under a specific environmental condition and an object included in the image as training data, and that estimates the name of an object included in the image from the image. The converted image is, For example, the image may be used as an image for which the name of an object included in the image is estimated by the estimation model (an image to be estimated). The specific environmental condition is, for example, that the image was taken during the daytime. In this case, the environmental condition other than the specific environmental condition is, for example, that the image was taken during the nighttime.
[0014] The image processing device 100 converts an image captured under environmental conditions other than the specific environmental conditions into an image equivalent to the image included in the training data of an estimation model that has been trained (machine learned) using a pair of an image captured under the specific environmental conditions and the name of an object included in the image as training data. That is, the image processing device 100 performs image processing that brings an image captured under environmental conditions other than the specific environmental conditions closer to the image characteristics of an image included in the training data of the estimation model (image characteristics of an image captured under the specific environmental conditions). As the machine learning, a well-known machine learning may be applied. Here, the estimation model is, for example, a model that estimates an object included in an image from an image captured under the specific environmental conditions. The estimation model here may be an estimation model that has been trained using an image captured under the specific environmental conditions as training data. The estimation model is also referred to as a target estimation model. The image characteristics are, for example, values calculated based on an image, such as the luminance of a pixel of an image, and the R value of the pixel of an image in the RGB color space. Here, the image processing device 100 converts an image captured under environmental conditions other than the specific environmental conditions into an image equivalent to an image (image captured under the specific environmental conditions) included in the teacher data of the target estimation model. For example, the converted image and the target estimation model can be used to estimate the name of an object included in the converted image. The converted image may also be used as teacher data for improving the performance of the target estimation model. That is, the image processing device 100 performs image processing on the shooting data (photographed image) of the target to be used as the estimation model to approximate the image characteristics of the image captured under the shooting conditions (specific environmental conditions) of the teacher data, thereby generating an image to be used as the teacher data. As a result, the image processing device 100 performs a predetermined image processing on the image captured under environmental conditions other than the specific environmental conditions, and can use the image processed image as an image of the teacher data of the estimation model in which the image captured under the specific environmental conditions is used as the teacher data. The image processing device 100 may also perform object estimation (estimation of the name of an object included in an image) using the estimation model, using the shooting data (photographed image) that has been subjected to image processing to approximate the image characteristics of the teacher data in the estimation model.This enables the image processing device 100 to perform object estimation for images captured under environmental conditions other than the specific environmental conditions, using an estimation model that uses images captured under the specific environmental conditions as training data.
[0015] Here, the image processing device 100 performs image processing to convert an image captured in a nighttime period (environmental conditions other than the specific environmental conditions) into an image equivalent to an image captured in a daytime period (specific environmental conditions) (also simply referred to as an image captured in a daytime period). For example, the nighttime period is from sunset to sunrise, and the daytime period is from sunrise to sunset. The nighttime period may be from a predetermined time (e.g., 2 hours) after sunset to a predetermined time (e.g., 2 hours) before sunrise, and the daytime period may be from a predetermined time (e.g., 2 hours) after sunrise to a predetermined time (e.g., 2 hours) before sunset. Regardless of the sunrise time and the sunset time, the daytime period may be from 10:00 to 16:00, and the nighttime period may be from 20:00 to 4:00 the next morning, etc. The nighttime period may be any period other than the daytime period. The nighttime period and the daytime period are not limited to those listed here. Also, the period from the end time of the evening time period to the start time of the morning time period may be defined as a morning time period, and the period from the end time of the daytime time period to the start time of the evening time period may be defined as an evening time period.
[0016] The image processing unit 102 acquires an image to be stored in the storage unit 110. The image processing unit 102 performs image processing on the acquired image to convert it into an image equivalent to an image under specific environmental conditions (also simply referred to as an image taken under specific environmental conditions). When the specific environmental conditions are images taken during the daytime, for example, the image processing may include brightness normalization, red color reduction, etc. .
[0017] It is assumed that each pixel in an image is represented by a pixel value in the RGB (Red Green Blue) color space. It is assumed that the image has X pixels in the horizontal direction and Y pixels in the vertical direction. For example, each pixel indicated by coordinates (x, y) in the image is represented by an R value representing red, a G value representing green, and a B value representing blue. For example, x and y are integers from 1 to X and from 1 to Y, respectively. The R value, G value, and B value are examples of pixel values. The R value, G value, and B value are represented by integers from 0 to 255, for example. The image may also be represented by pixel values in other formats (color spaces) in which each pixel in the image can be converted into a pixel value in the RGB color space. Examples of other formats (color spaces) include the YCbCr color space, the HSV color space, the HLS color space, and the CMYK color space.
[0018] The input unit 104 is an input means for accepting information input by a user, etc. The input unit 104 is an input device such as a keyboard, a pointing device, or the like.
[0019] The output unit 106 is an output means for outputting information such as display information to a user, etc. The output unit 106 is, for example, a display device such as a display. The output unit 106 displays images converted by image processing, etc.
[0020] The communication unit 108 is a communication interface that communicates with other information processing devices etc. via a communication network etc. The communication unit 108 receives images etc. from other information processing devices etc. and stores them in the storage unit 110.
[0021] The storage unit 110 stores an image in association with the date and time when the image was captured. The image may be considered to be generated when the image is captured, and the date and time when the image was captured may be the date and time when the image was generated. The image may be associated with the environmental conditions when the image was captured (date and time of capture, location of capture, weather at the time of capture, angle of view at the time of capture, etc.). The storage unit 110 stores an image captured under environmental conditions other than the specific environmental conditions, which is an image to be processed. The storage unit 110 also stores various data, values, information, etc. used in the image processing device 100. The storage unit 110 also stores an image (an image captured under specific environmental conditions) included in the teacher data of the target estimation model, or information on the image (average value, standard deviation, etc. of brightness values (Y values) of images captured under specific environmental conditions). The images included in the teacher data of the target estimation model are at least two or more images. The images, etc. stored in the storage unit 110 are acquired from another information processing device, etc., via the communication unit 108, for example. The storage unit 110 stores values, data, images generated by the image processing unit 102, and the like.
[0022] FIG. 2 is a diagram showing an example of the hardware configuration of an information processing device. The information processing device shown in FIG. 2 has the configuration of a general computer. The image processing device 100 is realized by an information processing device 90 as shown in FIG. 2. The information processing device 90 in FIG. 2 has a processor 91, a memory 92, a storage unit 93, an input unit 94, an output unit 95, and a communication control unit 96. These are connected to each other by a bus. The memory 92 and the storage unit 93 are computer-readable recording media. The hardware configuration of the computer is not limited to the example shown in FIG. 2, and components may be omitted, replaced, or added as appropriate.
[0023] The information processing device 90 has a processor 91 which loads a program stored in a recording medium into a working area of a memory 92 and executes it, and each component, etc. is controlled through the execution of the program, thereby realizing functions consistent with a specified purpose.
[0024] The processor 91 may be, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). Signal Processor).
[0025] The memory 92 includes, for example, a random access memory (RAM) and a read only memory (ROM). The memory 92 is also called a main storage device.
[0026] The storage unit 93 includes a non-volatile storage medium. The non-volatile storage medium included in the storage unit 93 is, for example, an EPROM (Erasable Programmable ROM) or a hard disk drive (HDD The storage unit 93 is a removable medium, i.e., a portable storage device. The removable medium may include a recording medium. The removable medium is, for example, a Universal Serial Bus (USB) memory, or a disk recording medium such as a Compact Disc (CD) or a Digital Versatile Disc (DVD). The storage unit 93 is also called a secondary storage device.
[0027] The storage unit 93 stores various programs, various data, and various tables in a readable and writable recording medium. An operating system (OS), various programs, various tables, etc. are stored in the storage unit 93. Information stored in the storage unit 93 may be stored in the memory 92. Information stored in the memory 92 may be stored in the storage unit 93.
[0028] The operating system is software that mediates between software and hardware, manages memory space, manages files, and manages processes and tasks. The operating system includes a communication interface. The communication interface is a program that exchanges data with other external devices connected via the communication control unit 96. External devices include, for example, other computers and external storage devices.
[0029] The input unit 94 includes a keyboard, a pointing device, a wireless remote control, a touch panel, etc. The input unit 94 can also include an input device for video or images such as a camera, and an input device for audio such as a microphone.
[0030] The output unit 95 is a liquid crystal display (LCD), an electroluminescence (EL) panel, The output unit 95 may include a display device such as a display panel, a CRT (Cathode Ray Tube) display, a PDP (Plasma Display Panel), etc., and an output device such as a printer, etc. The output unit 95 may also include an audio output device such as a speaker.
[0031] The communication control unit 96 connects to other devices and controls communication between the information processing device 90 and the other devices. The communication control unit 96 is, for example, a LAN (Local Area Network) interface board, a wireless communication circuit for wireless communication, or a communication circuit for wired communication. The LAN interface board and the wireless communication circuit are connected to a network such as the Internet.
[0032] A computer that realizes the image processing device 100 realizes the functions of an image processing unit 102, an input unit 104, an output unit 106, and a communication unit 108 by a processor loading a program stored in an auxiliary storage device into a main storage device and executing the program. On the other hand, a storage unit 110 is provided in a storage area of the main storage device or the auxiliary storage device.
[0033] (Example of operation) Overall Operation FIG. 3 is a diagram showing an example of the overall operation flow of image processing by the image processing device 100 of this embodiment. Here, the target estimation model is an estimation model trained using training data including images taken during daytime hours (specific environmental conditions). The image processing device 100 performs predetermined image processing on images taken during time periods other than daytime hours (environmental conditions other than the specific environmental conditions, here assumed to be nighttime hours) to train data for the target estimation model. The image processing device 100 converts the image into an image equivalent to the image contained therein (an image taken during the daytime). Here, it is assumed that the storage unit 110 of the image processing device 100 stores an image taken during the nighttime (an image to be processed) that is to be subjected to image processing.
[0034] In S101, the image processing unit 102 of the image processing device 100 acquires from the storage unit 110 an image to be subjected to image processing (an image captured during the nighttime hours).
[0035] In S102, the image processing unit 102 extracts a pixel having the maximum R value among the R value, G value, and B value in the RGB color space from each pixel of the image. The image here is the image acquired in S101 when the process proceeds from S101 to S102, and is the image converted in the image processing immediately before S109 when the process proceeds from S109 to S102. The image is regarded as the original image. The image processing unit 102 counts the number of extracted pixels. The image processing unit 102 calculates the ratio of the number of counted pixels to the number of all pixels of the original image. The image processing unit 102 judges whether the calculated ratio is equal to or greater than a first threshold value. The first threshold value is stored in the storage unit 110 in advance. If the calculated ratio is equal to or greater than the first threshold value (S102; YES), the process proceeds to S103. If the calculated ratio is less than the first threshold value (S102; NO), the process proceeds to S104.
[0036] In S103, the image processing unit 102 performs a second image processing for reducing red color of the original image, and converts the original image. The second image processing will be described later.
[0037] In S104, the image processing unit 102 calculates the Y value of the YCbCr color space for each pixel of the original image. The conversion from the RGB color space to the YCbCr color space is performed by a known conversion formula (calculation formula). The YCbCr color space is expressed by a Y value (luminance), a Cb value, and a Cr value. The image processing unit 102 extracts the maximum Y value and the minimum Y value from the calculated Y values. The image processing unit 102 calculates the difference between the maximum Y value and the minimum Y value. The image processing unit 102 determines whether the calculated difference is equal to or greater than a second threshold value. The second threshold value is stored in advance in the storage unit 110. If the calculated difference is equal to or greater than the second threshold value (S104; YES), the process proceeds to S105. If the calculated difference is less than the second threshold value (S104; NO), the process proceeds to S106.
[0038] In S105, the image processing unit 102 performs a third image processing for normalizing the brightness of the illumination light component of the original image, and converts the original image. The third image processing will be described later.
[0039] In S106, the image processing unit 102 extracts pixels of the original image whose Y value in the YCbCr color space is less than a third threshold. The third threshold is stored in advance in the storage unit 110. The image processing unit 102 counts the number of extracted pixels. The image processing unit 102 calculates the ratio of the counted number of pixels to the number of all pixels of the original image. The image processing unit 102 determines whether the calculated ratio is equal to or greater than a fourth threshold. The fourth threshold is stored in advance in the storage unit 110. If the calculated ratio is equal to or greater than the fourth threshold (S106; YES), the process proceeds to S107. If the calculated ratio is less than the fourth threshold (S106; NO), the process proceeds to S108.
[0040] In S107, the image processing unit 102 performs a fourth image processing for normalizing the brightness of the reflectance component of the original image, and converts the original image. The fourth image processing will be described later.
[0041] In S108, the image processing unit 102 performs a first image processing for normalizing the brightness of the entire original image, thereby converting the original image. The first image processing will be described later.
[0042] In S109, the image processing unit 102 performs various image processing on the original image to convert it into an image. It is determined whether the converted image is equivalent to an image taken during the daytime by comparing the converted image with images (all images taken during the daytime) included in the training data of the target estimation model stored in the storage unit 110. An image equivalent to an image taken during the daytime is an image whose pixel values are similar to the pixel values of an image taken during the daytime.
[0043] The image processing unit 102 calculates, for example, for each image (image a, image b, image c, ···) taken in the daytime, the average value of the Y values in the YCbCr color space of all the pixels of the image (average value Ya of image a, average value Yb of image b, average value Yc of image c, ···). Further, the image processing unit 102 calculates the average value AVG1 (average value of Ya, Yb, Yc, ··· (average value of the Y values of all the images taken in the daytime)) and the standard deviation σY (standard deviation of Ya, Yb, Yc, ···) of the average values of the Y values calculated for each image. Also, the image processing unit 102 calculates the average value AVG2 of the Y values of all the pixels of the converted image in the YCbCr color space. The image processing unit 102 calculates the absolute value of the difference between the average value AVG2 of the Y values of the converted image and the average value AVG1 of the Y values of all the images taken in the daytime (absolute value of AVG2 - AVG1). If the calculated absolute value of the difference (absolute value of AVG2 - AVG1) is less than the calculated standard deviation σY, the pixel values of the converted image are similar to the pixel values of the images taken in the daytime. Therefore, when the calculated absolute value of the difference (absolute value of AVG2 - AVG1) is less than the calculated standard deviation σY, the image processing unit 102 determines that the converted image is an image equivalent to the images taken in the daytime. On the other hand, when the calculated absolute value of the difference (absolute value of AVG2 - AVG1) is greater than or equal to the calculated standard deviation σY, the image processing unit 102 determines that the converted image is not an image equivalent to the images taken in the daytime. Instead of the standard deviation σY as the threshold value, a value that is X times the standard deviation σY (for example, 0 < X < 3) or a predetermined fixed value may be used as the threshold value.
[0044] The image processing unit 102 may calculate in advance the average Y value (Ya, Yb, Yc,...), the average value AVG1 of the average Y values, and the standard deviation σY of each image taken during the daytime, and store them in the storage unit 110. The image processing unit 102 may also acquire the average Y value (Ya, Yb, Yc,...), the average value AVG1 of the average Y values, and the standard deviation σY of each image taken during the daytime from another information processing device or the like via the communication unit 108, and store them in the storage unit 110. At this time, the image processing unit 102 acquires the average Y value (Ya, Yb, Yc,...), the average value AVG1 of the average Y values, and the standard deviation σY of each image taken during the daytime from the storage unit 110 during the processing of S109.
[0045] Instead of the Y value in the YCbCr color space, other pixel values (for example, R value, G value, B value in the RGB color space), the sum of R value, G value, and B value, etc. may be used. Also, multiple values may be used. That is, for example, when the differences between multiple values (values of the index) are all less than a threshold, it is determined that the converted image is equivalent to an image taken in the daytime (image included in the teacher data of the target estimation model). Also, the image processing unit 102 may determine whether or not the converted image is equivalent to an image taken in the daytime (image included in the teacher data of the target estimation model) by another method. Luminance, red, green, blue, etc. are examples of indexes. Y value, R value, G value, B value, the sum of R value, G value, and B value, etc. are examples of index values.
[0046] If it is determined that the converted image is not equivalent to an image taken during the daytime (S109; NO), the process returns to S102. If it is determined that the converted image is equivalent to an image taken during the daytime (S109; YES), the process ends. As a result, an image taken at night is converted into an image equivalent to an image taken during the daytime. In addition, until the converted image becomes equivalent to an image taken during the daytime, The process is repeated.
[0047] Of steps S102, S104, and S106, steps 1 to 3 may be omitted. If S102 is omitted, the process proceeds to S104. If S104 is omitted, the process proceeds to S106. If S106 is omitted, the process proceeds to S108. The order of steps S102, S104, and S106 may be reversed.
[0048] <First image processing operation> Fig. 4 is a diagram showing an example of an operational flow of the first image processing. The first image processing is the processing of S108 in the operational flow of Fig. 3. The first image processing is processing for normalizing the brightness of the entire image of the original image.
[0049] In S1101, the image processing unit 102 of the image processing device 100 converts each pixel of the original image from the RGB color space to the HSV color space. The conversion from the RGB color space to the HSV color space is performed by a well-known conversion formula (calculation formula). The HSV color space is expressed by hue (Hue (H value)), saturation (Saturation·Chroma (S value)), and brightness (Value·Brightness (V value)). Here, the V value of the pixel at the coordinates (x, y) of the original image is defined as I(x, y). Brightness is also called luminance.
[0050] In S1102, the image processing unit 102 calculates the average value I a , mark Standard deviation σ I Calculate.
[0051] In S1103, the image processing unit 102 calculates I(x, y), I a , σ I Using Calculate the corrected V value for all pixels. Let I'(x,y) be the corrected V value of the pixel at coordinates (x,y). The corrected V value I'(x,y) of the pixel at coordinates (x,y) is expressed as follows:
number
[0052] Here, N a is the average value (average of Aa, Ab, Ac, ...) of the V values in the HSV color space of all pixels for each image (image a, image b, image c, ...) taken during the daytime (average value of Aa, Ab, Ac, ...). σ N is the standard deviation (standard deviation of Aa, Ab, Ac, ...) of the average V value in the HSV color space of all pixels for each image (image a, image b, image c, ...) taken during the daytime (the average value Aa of image a, the average value Ab of image b, the average value Ac of image c, ...).
[0053] In S1104, the image processing unit 102 converts each pixel into the RGB color space using the H value, S value, and corrected V value I'(x, y) of each pixel of the original image. The conversion from the HSV color space to the RGB color space is performed by a well-known conversion formula (calculation formula). The image processing unit 102 generates a converted image based on the pixel value of each converted pixel in the RGB color space, and stores it in the storage unit 110. This normalizes the brightness of each pixel of the original image. In general, an image taken at night is darker (has a smaller Y value) than an image taken during the day. Therefore, by normalizing the brightness, the image taken at night can be made closer to the image taken during the day.
[0054] <Second image processing operation> Fig. 5 is a diagram showing an example of the operation flow of the second image processing. The second image processing is the process of S103 in the operation flow of Fig. 3. The second image processing is a process of reducing red in the original image and then normalizing the brightness of the entire image.
[0055] In S1201, the image processing unit 102 of the image processing device 100 extracts, from each pixel of an image, a pixel whose R value is maximum among the R value, G value, and B value in the RGB color space. The image processing unit 102 replaces the R value of the extracted pixel with the average value of the G value and the B value ((G value+B value) / 2).
[0056] In S1202, the image processing unit 102 calculates the H value in the HSV color space for the pixel replaced in S1201, using the replaced R value (=(G value+B value) / 2) and the original G value and B value.
[0057] In S1203, the image processing unit 102 calculates the S value and V value in the HSV space for the pixel replaced in S1201, using the original R value, G value, and B value (the R value, G value, and B value before replacement).
[0058] In S1204, the image processing unit 102 converts the pixels replaced in S1201 into R values, G values, and B values in the RGB color space using the H values in the HSV color space calculated in S1202 and the S values and V values in the HSV space calculated in S1203. Also, the pixels not replaced in S1201 retain their original R values, G values, and B values. The image processing unit 102 generates a new image using the converted R values, G values, and B values of the pixels replaced in S1201 and the original R values, G values, and B values of the pixels not replaced in S1201, and stores the new image in the storage unit 110.
[0059] In steps S1205 to S1208, the image processing unit 102 performs brightness normalization processing on the new image generated in step S1204. The processing from step S1205 to step S1208 is similar to the processing from step S1101 to step S1104 in FIG. 4, and therefore the description will be omitted. The image processing unit 102 stores the converted image in the storage unit 110. This makes it possible to reduce red color while maintaining the hue of the original image and perform brightness normalization. This makes it possible to reduce the influence of the red light of the vehicle included in the image. The red light of the vehicle is turned on during the night but not during the day, so by reducing the influence of the red light, the image can be made closer to an image taken during the day.
[0060] <Third image processing operation> Fig. 6 is a diagram showing an example of the operation flow of the third image processing. The third image processing is the process of S105 in the operation flow of Fig. 3. The third image processing is a process for normalizing the brightness of the illumination light component of the original image.
[0061] In S1301, the image processing unit 102 of the image processing device 100 converts the pixel values (R value, G value, B value) of each pixel of the original image into pixel values (Y value, Cb value, Cr value) in the YCbCr color space.
[0062] In S1302, the image processing unit 102 calculates the Y value L of the illumination light component and the Y value R of the reflectance component for each pixel from the Y value in the YCbCr color space based on the SSR (Single Scale Retinex) theory. Here, if the Y value of the coordinates (x, y) is I(x, y), the Y value of the illumination light component is L(x, y), and the Y value of the reflectance component is R(x, y), then it can be expressed as follows:
number
[0063] From these equations, the Y value L of the illumination light component and the Y value R of the reflectance component of each pixel can be obtained.
[0064] In S1303, the image processing unit 102 calculates the average value I a , mark Standard deviation σ I Calculate.
[0065] In S1304, the image processing unit 102 calculates I(x, y), I a , σ I Using the original image The Y value L of the corrected illumination light component is calculated for all pixels. The Y value L of the corrected illumination light component of the pixel at coordinates (x, y) is defined as I'(x, y). The Y value L of the corrected illumination light component of the pixel at coordinates (x, y) is expressed as follows:
number
[0066] Here, N ais the average value (average value of Aa, Ab, Ac, ...) of the Y values in the YCbCr color space of all pixels for each image (image a, image b, image c, ...) taken during the daytime (average value of Aa, Ab, Ac, ...). σ N is the standard deviation (standard deviation of Aa, Ab, Ac, ...) of the average Y values in the YCbCr color space of all pixels for each image (image a, image b, image c, ...) taken during the daytime (the average value Aa of image a, the average value Ab of image b, the average value Ac of image c, ...). Note that the V value in the HSV color space and the Y value in the YCbCr color space are the same value. The reason for performing correction to the illumination light component here is to maintain the local contrast of the image.
[0067] In S1305, the image processing unit 102 calculates a corrected Y value using the corrected Y value of the illumination light component (I'(x,y)) calculated in S1304 and the Y value R(x,y) of the reflectance component calculated in S1302. The corrected Y value of the pixel at the coordinates (x,y) is calculated by I'(x,y)·R(x,y).
[0068] In S1306, the image processing unit 102 converts each pixel into the RGB color space using the Cb and Cr values calculated in S1301 and the corrected Y value calculated in S1305. The conversion from the YCbCr color space to the RGB color space is performed using a well-known conversion formula (calculation formula). The image processing unit 102 generates a converted image based on the pixel values in the RGB color space of each converted pixel, and stores the image in the storage unit 110. This normalizes the brightness of the illumination light component of each pixel in the original image. The third image processing corrects the brightness of pixels with strong illumination light such as high-luminance neon lights and street lights. The third image processing can make an image taken at night, for example in a city with many street lights, closer to an image taken during the day.
[0069] <Fourth image processing operation> Fig. 7 is a diagram showing an example of the operation flow of the fourth image processing. The third image processing is the process of S107 in the operation flow of Fig. 3. The fourth image processing is a process for converting the luminance of the original image into a reflectance component.
[0070] In S1401, the image processing unit 102 of the image processing device 100 converts the pixel values (R value, G value, B value) of each pixel of the original image into pixel values (Y value, Cb value, Cr value) in the YCbCr color space.
[0071] In S1402, the image processing unit 102 calculates the Y value L of the illumination light component and the Y value R of the reflectance component from the Y value in the YCbCr color space for each pixel based on the SSR (Single Scale Retinex) theory. The calculation of the Y value L of the illumination light component and the Y value R of the reflectance component is similar to S1302. The image processing unit 102 sets the Y value of the reflectance component as the luminance of the image to be converted.
[0072] In S1403, the image processing unit 102 converts each pixel into the RGB color space using the luminance (=Y value of reflectance component) calculated in S1402 and the Cb value and Cr value calculated in S1401. The conversion from the YCbCr color space to the RGB color space is performed by a well-known conversion formula (calculation formula). The image processing unit 102 generates a converted image based on the pixel value of the converted pixel in the RGB color space of each pixel, and stores it in the storage unit 110. This converts the luminance of each pixel of the original image into a reflectance component. The fourth image processing can generate an image in which the contour information of the original image is emphasized. The image generated by the fourth image processing may be used as an image of teacher data or an image to be estimated for an estimation model in which an image obtained by performing the fourth image processing on an image taken during the daytime is used as teacher data.
[0073] (others) Here, the specific environmental condition is a daytime period, but the specific environmental condition is not limited to this and may be, for example, a nighttime period, a sunny weather, a rainy weather, a cloudy weather, a shooting location inside a tunnel, a shooting location outdoors, etc. A combination of multiple environmental conditions may be used as a specific environmental condition.
[0074] (Actions and Effects of the Embodiments) The image processing device 100 converts an image captured under environmental conditions other than the specific environmental conditions into an image equivalent to the image included in the teacher data of an estimation model that has been trained (machine learned) using a pair of an image captured under the specific environmental conditions and the name of an object included in the image as teacher data. The estimation model is, for example, an estimation model that estimates the name of an object included in an image from an image, which has been trained using a pair of an image captured during daytime hours and the name of an object included in the image as teacher data. The image processing device 100 converts an image captured during a time period other than daytime hours (for example, nighttime hours) into an image equivalent to an image captured during daytime hours. The converted image can be used as an image of teacher data for the estimation model or an image (image to be estimated) from which the name of an object is estimated by the estimation model. According to the image processing device 100, by converting an image captured during nighttime hours into an image captured during daytime hours, the image captured during nighttime hours can be used as an image of teacher data or an image to be estimated by the estimation model, in an estimation model that has been trained using an image captured during daytime hours. An estimation model trained using images captured under specific environmental conditions has higher performance than an estimation model trained using images captured under various environmental conditions. According to the image processing device 100, an estimation model with higher performance can be used by converting an image captured under a non-specific environmental condition into an image captured under the specific environmental condition.
[0075] The image processing device 100 performs brightness normalization processing, brightness normalization processing after reducing red color, brightness normalization processing of illumination light components, and processing of converting brightness to reflectance components according to the state of the original image (image taken at night). The image processing device 100 can perform appropriate image conversion processing according to the state of the original image to convert it into an image taken at daytime. The image processing device 100 performs red color reduction and brightness normalization processing for an image in which the ratio of pixels with the largest R value among R values, G values, and B values is equal to or greater than a threshold value. This makes it possible to reduce the red color of vehicle brake lights and the like included in an image taken at night and correct the brightness. The image processing device 100 extracts illumination light components from an image in which the difference between the maximum and minimum values of brightness (Y value) is equal to or greater than a threshold value, and performs brightness normalization processing. This makes it possible to suppress the brightness of areas with strong illumination light (high-brightness neon signs and street lights) included in an image taken at night. The image processing device 100 performs a process of replacing the luminance with the luminance of the reflectance component for an image in which the ratio of pixels whose luminance (Y value) is smaller than the threshold value is equal to or greater than the threshold value, thereby generating an image in which edge information is emphasized.
[0076] Conventionally, in order to estimate an image taken during the nighttime using an estimation model, it was required to prepare an estimation model that learned images taken during the nighttime as teacher data. In addition, in order to estimate an image taken during the daytime using an estimation model, it was required to prepare an estimation model that learned images taken during the daytime as teacher data. That is, it was required to prepare an estimation model for each specific environmental condition. In addition, there was a problem that the estimation accuracy was low when an estimation model that learned images taken during the daytime as teacher data was used to estimate an image taken during the nighttime. Furthermore, in order to deal with images taken during the nighttime and images taken during the daytime with one estimation model, it was required to prepare an estimation model that learned images taken during the nighttime and images taken during the daytime as teacher data. The image processing device 100 of this embodiment converts images taken during the nighttime into images taken during the daytime. As a result, by using the converted image, it is possible to estimate images taken during the nighttime without reducing the estimation accuracy using an estimation model that learned images taken during the daytime as teacher data. According to the image processing device 100, if an estimation model trained using images taken during the daytime as training data is prepared, it is possible to estimate images taken during the nighttime and images taken during the daytime. When training the estimation model, it is sufficient to prepare training data including images taken during the daytime, and the amount of training data can be reduced compared to estimation models corresponding to the nighttime and daytime. In other words, the types and variations of training data can be reduced.
[0077] Computer-readable recording medium A program for causing a computer or other machine or device (hereinafter, referred to as a computer, etc.) to realize any of the above functions can be recorded on a recording medium readable by the computer, etc. Then, the computer, etc. can provide the function by reading and executing the program from the recording medium.
[0078] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc. Such a recording medium may be provided with elements that constitute a computer, such as a CPU and memory, and the CPU may be made to execute the program.
[0079] Among such recording media, those that are removable from a computer etc. include, for example, flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, DATs, 8mm tapes, memory cards and the like.
[0080] Further, examples of recording media fixed to a computer include a hard disk and a ROM.
[0081] Although the embodiments of the present invention have been described above, these are merely examples, and the present invention is not limited to these. Various modifications based on the knowledge of those skilled in the art are possible without departing from the spirit of the claims. [Explanation of symbols]
[0082] 100 Image processing device 102 Image processing section 104 Input section 106 Output section 108 Communications Department 110 Storage section 90 Information processing equipment 91 Processor 92 Memory 93 Memory section 94 Input section 95 Output section 96 Communications Department
Claims
1. An image processing unit that performs image processing on a first image captured under a first condition to generate a converted image, the image processing unit performs the image processing until an absolute value of a difference between a first value calculated based on the converted image and a second value calculated based on a plurality of second images captured under second conditions different from the first conditions becomes less than a threshold value; the first value is an average value of the luminance of all pixels included in the converted image, the second value is an average value of the luminance of all images included in the plurality of second images, and the luminance of the image is an average value of the luminance of all pixels included in the image; Image processing device.
2. The image processing includes a process of correcting a luminance of each pixel of the first image based on a luminance of each pixel of the plurality of second images. The image processing device according to claim 1 .
3. the image processing includes a process of replacing an R value of a pixel having a maximum R value among the R values, G values, and B values with an average value of the G values and the B values when each pixel of the first image is represented in an RGB color space by R values, G values, and B values; 3. The image processing device according to claim 1 or 2.
4. the image processing includes a process of calculating a luminance of an illumination light component and a luminance of a reflectance component of each pixel from a luminance of each pixel of the first image, correcting the luminance of the illumination light component based on a luminance of each pixel of the plurality of second images, and correcting the luminance of each pixel of the first image from the corrected luminance of the illumination light component and the luminance of the reflectance component. The image processing device according to claim 1 .
5. The image processing includes: calculating the luminance and reflectance of the illumination light component of each pixel from the luminance of each pixel of the first image; and replacing the brightness of each pixel of the first image with the brightness of the reflectance component. The image processing device according to claim 1 .
6. The first condition is that the image is taken during a nighttime period, and the second condition is that the image is taken during a daytime period. The image processing device according to claim 1 .
7. The computer performing image processing on the first image captured under the first condition to generate a converted image; performing the image processing until an absolute value of a difference between a first value calculated based on the converted image and a second value calculated based on a plurality of second images captured under second conditions different from the first conditions becomes less than a threshold value; the first value is an average value of the luminance of all pixels included in the converted image, the second value is an average value of the luminance of all images included in the plurality of second images, and the luminance of the image is an average value of the luminance of all pixels included in the image; Image processing methods.
8. The image processing includes a process of replacing the R value of a pixel having the maximum R value among the R values, G values, and B values with an average value of the G value and the B value when each pixel of the first image is represented in an RGB color space by R values, G values, and B values. The image processing method according to claim 7.
9. A method for generating a teacher image of teacher data in an estimation model, comprising the steps of: The computer performing image processing on the first image captured under the first condition to generate a converted image; When performing the image processing to obtain the training data, the image processing is performed until an absolute value of a difference between a first value calculated based on the converted image and a second value calculated based on a plurality of second images taken under a second condition different from the first condition becomes less than a threshold value, the first value is an average value of the luminance of all pixels included in the converted image, the second value is an average value of the luminance of all images included in the plurality of second images, and the luminance of the image is an average value of the luminance of all pixels included in the image; A teacher image generating method comprising the steps of:
10. A computer comprising: performing image processing on the first image captured under the first condition to generate a converted image; When performing the image processing to obtain training data, the image processing is performed until an absolute value of a difference between a first value calculated based on the converted image and a second value calculated based on a plurality of second images taken under a second condition different from the first condition becomes less than a threshold value, the first value is an average value of luminance of all pixels included in the converted image, the second value is an average value of luminance of all images included in the plurality of second images, and the luminance of the image is an average value of luminance of all pixels included in the image; An object estimation method, comprising: performing object estimation using an estimation model trained by machine learning using the training data.
Citation Information
Patent Citations
Peripheral obstacle detector
JP2007333458A
Apparatus and program for creating learning model, and apparatus and program for detecting object
JP2009289189A
Control device, image capturing apparatus and control method thereof
JP2020091453A
Imaging device
JP2020187360A
Operation processing device, object identifying system, learning method, automobile, and lighting appliance for vehicle
WO2019194256A1
