Image processing method, image processing device, image processing system, and program

Abstract

This invention provides an image processing method that enables highly accurate acquisition of information about a subject from an image. [Solution] The image processing method includes an acquisition step of acquiring first information relating to the color of a subject, a first database of color models, and second information relating to ambient light; a determination step of determining the state of ambient light using the second information; and a modification step of changing at least one of the processing relating to the first information or the processing relating to the first database based on the determination result of the state of ambient light.

Description

Technical Field

[0004] , , ,

[0006] , , , ,

[0005] , , , , ,

[0003] ,

[0007] , ,<000003�>, , ,

[0001] The present invention relates to an image processing method, an image processing apparatus, an image processing system, and a program.

Background Art

[0002] Patent Document 1 discloses a method of obtaining a SPAD (Soil & Plant Analyzer Development) value from the spectroscopic measurement results of light reflected by a subject. Patent Document 2 also discloses a method of obtaining a SPAD value using an optimization operation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the method disclosed in Patent Document 1, when the weather or the sun position changes, the mixing ratio of the reflected light and the transmitted light from the subject changes, and it is difficult to obtain information about the subject with high accuracy.

[0005] Patent Document 2 does not describe separating reflection characteristic data or transmission characteristic data according to the weather. Therefore, the appearance of the image changes due to the shadow formed by the influence of direct sunlight, and it is difficult to obtain information about the subject with high accuracy.

[0006] Therefore, an object of the present invention is to provide an image processing method capable of accurately obtaining information about a subject from an image.

Means for Solving the Problems

[0007] An image processing method as one aspect of the present invention is characterized by comprising: an acquisition step of acquiring first information relating to the color of a subject, a first database of color models, and second information relating to ambient light; a determination step of determining the state of the ambient light using the second information; and a modification step of changing at least one of the processing relating to the first information or the processing relating to the first database based on the determination result of the state of the ambient light.

[0008] Other objects and features of the present invention are described in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing method that can acquire information about a subject from an image with high accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of the image processing system in Example 1. [Figure 2] This figure shows how the image processing system in Example 1 is used to capture an image of a subject under outdoor ambient light. [Figure 3] This is a flowchart showing the method for generating color information about a subject in Example 1. [Figure 4] This is a flowchart showing the method for determining the leaf color correlation value in Example 1. [Figure 5] This figure shows an image related to the masking process in Example 1. [Figure 6] This figure shows the histogram smoothing process in Example 1. [Figure 7] This figure shows an example of a color model database in Example 1. [Figure 8] This figure shows an example of the histogram and the database of the color model in Example 1. [Figure 9]It is a flowchart showing a method for determining the leaf color correlation value in Example 2. [Figure 10] It is a diagram showing an example in which the mask processing, histogram, and database of the model related to color in Example 2 are described together. [Figure 11] It is a diagram showing an example in which the mask processing, histogram, and database of the model related to color in Example 2 are described together. [Figure 12] It is a diagram showing an example in which the mask processing, histogram, and database of the model related to color in Example 2 are described together. [Figure 13] It is a flowchart for calculating the leaf color correlation value in Example 3. [Figure 14] It is a diagram showing an example in which the mask processing, histogram, and database of the model related to color in Example 3 are described together. [Figure 15] It is a diagram showing an example in which the mask processing, histogram, and database of the model related to color in Example 3 are described together.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted.

[0012] (Example 1) In this embodiment, an image processing method for obtaining numerical information (information related to the concentration of a substance) that is correlated with the concentration of a substance contained in a subject from an image (spectral image) acquired by a camera (RGB camera) having an imaging unit will be described. The concentration of the substance contained in the subject is correlated with, for example, a leaf color value or a SPAD value. The substance is, for example, chlorophyll contained in a plant leaf. In the following description, "numerical information that is correlated with the concentration of the substance contained in the subject" is also referred to as a "leaf color correlation value". Also, in this embodiment, the case where the subject is a rice leaf will be described, but the present invention is not limited thereto. For example, it may be a rice stem, spike, flower, and fruit. Also, it may be a plant other than rice (e.g., wheat).

[0013] FIG. 1 is a block diagram of an image processing system 100. The image processing system 100 includes an imaging unit (camera) 101, an image processing unit (image processing apparatus) 102, a control unit 103, a storage unit 104, a communication unit 105, a display unit 106, and an ambient light information acquisition unit 110. The imaging unit 101 includes an imaging optical system 101a and an imaging element 101b. The imaging element 101b photoelectrically converts an optical image (subject image) formed via the imaging optical system 101a and outputs an image (image data) to the image processing unit 102.

[0014] The storage unit 104 stores a database (first database) of a model related to colors to be described later, and a database (second database) of light information of ambient light (second information related to ambient light). The light information of ambient light is information related to the brightness of ambient light and changes according to the brightness.

[0015] The image processing unit 102 includes an input means (acquisition means) 102a, a determination means (ambient light state determination means) 102b, and a modification means 102c. The input means 102a inputs (acquires) color information about the subject (first information about the subject's color), a first database of color models, and ambient light light information (second information about ambient light). The determination means 102b determines the state of the ambient light using the light information. The modification means 102c modifies at least one of the processing related to color information or the processing related to the first database based on the result of determining the state of the ambient light (determination result).

[0016] The image processing system (imaging system) 100 may be located inside the camera. Alternatively, some functions of the image processing unit 102 or the storage unit 104 may be configured to be implemented on a computer (user PC) or cloud computing system located away from the camera. In this case, the camera will only have some of the functions of the image processing system 100, including the imaging unit 101.

[0017] Figure 2 shows the process of imaging a subject 140 under outdoor ambient light using the image processing system 100. The ambient light information acquisition unit 110 includes an ambient light sensor 111 that acquires the illuminance of light incident on the subject 140 for predetermined wavelengths. The wavelength range that the ambient light sensor 111 can acquire is the wavelength range of visible light, specifically a wavelength range of about 400 nm to 700 nm (wavelength range of blue light (B light), green light (G light), and red light (R light)). The ambient light sensor 111 may be, for example, a photoelectric conversion element such as a CCD or CMOS sensor on which red, green, and blue color filters are provided. The range shown by the dotted line in Figure 2 is the field of view range 171 of the imaging unit 101.

[0018] Next, with reference to Figure 3, a method for generating color information about a subject to be input to the image processing unit 102 of this embodiment will be described. Figure 3 is a flowchart showing a method for generating color information about a subject to be input to the image processing unit 102, and specifically, it is a flow for correcting the effect of ambient light from the image information captured by the imaging unit 101. Ambient light refers to external light such as sunlight. Since the color balance (or color temperature) of the light source may change depending on the time of shooting and the weather, it is preferable to correct the effect of ambient light (correct the white balance) in order to obtain accurate color information of the subject 140. Note that each step in Figure 3 is mainly performed by each part of the image processing system 100.

[0019] First, in step S201, the imaging unit 101 captures the subject 140 in response to a signal from the control unit 103 and acquires a color image (RGB image). The image processing unit 102 acquires the image captured by the imaging unit 101.

[0020] In step S202, the ambient light information acquisition unit 110 (ambient light sensor 111) acquires (detects) ambient light information (ambient light information, ambient light data) at the time the image was captured, in response to a signal from the control unit 103. In this embodiment, ambient light information includes information on the brightness (intensity) of light from the surrounding light source (mainly sunlight) and information on the balance of colors (or color temperature) contained in the light. The ambient light sensor 111 is pre-calibrated to have the same spectral sensitivity characteristics as the imaging unit 101. In this embodiment, the processing of steps S201 and S202 is performed almost simultaneously, but it is not limited to this, and steps S201 and S202 may be performed at different timings.

[0021] Next, in step S210, the image processing unit 102 performs ambient light correction processing based on the image information obtained in step S201 and the color balance information from the ambient light information obtained in step S202. The image that has undergone ambient light correction processing (white balance correction processing) in this step is the input image used in the image processing method of this embodiment, and corresponds to the color information of the subject.

[0022] Next, we will explain how to determine the leaf color correlation value using the input image, referring to Figure 4. Figure 4 is a flowchart showing the method for determining the leaf color correlation value.

[0023] First, in step S301, the image processing unit 102 receives the input image. Next, in step S302, the image processing unit 102 performs leaf region masking. In leaf region masking, instead of processing the entire input image in the subsequent processing, masking is performed to process a specific region. This is preferable because it can improve accuracy by excluding unnecessary elements such as soil and obstacles that do not need to be processed.

[0024] Specific examples will be explained with reference to Figures 5(A) to (C). Figure 5(A) is an example image obtained by capturing the subject 140 with the imaging unit 101 and completing the series of steps shown in Figure 3. Figure 5(B) is an example image obtained by extracting only the portion enclosed by the black frame in Figure 5(A) when the image of the subject 140 is input as in Figure 5(A). Figure 5(C) is a mask image for extracting only the leaf region from the image in Figure 5(B).

[0025] Step S302, leaf region masking, is the process of extracting only the leaf region (the white area in the mask image in Figure 5(C)) from the image in Figure 5(B). Leaf region masking can be performed by first extracting a specific part from the input image and then applying the leaf region masking to that extracted image, or by processing the entire image at once. Possible methods for performing leaf region masking include setting an arbitrary threshold for leaf color (green) and extracting only the green parts of the leaves, or using machine learning to detect leaves by considering not only color but also leaf shape. The method is not limited to the above, as long as it can remove background soil and obstacles from the input image and extract only the leaf portion.

[0026] In this step, the leaf region is extracted from the input image, and subsequent steps are performed on the extracted leaf region image. If the input image has a very large number of pixels, the input image may be reduced in size before performing the leaf region masking. This method of masking, which leaves only the necessary regions, is preferable because it enables higher accuracy. The image obtained by masking the input image in this way, and extracting only the leaf region from the input image, is called the leaf region image.

[0027] Next, in step S303, the image processing unit 102 calculates a histogram for the leaf region image (color information of the subject). The leaf region image is a color image having at least three channels of RGB (red, green, blue). For example, the leaf region image (color information) is image information of a plant acquired using a camera having at least three RGB channels. The leaf region image is also numerical information that correlates with at least the concentration of substances contained in the subject.

[0028] If we directly create a histogram for the pixel values ​​of each RGB channel, it will be three-dimensional. Since it is difficult to analyze in three dimensions, it is preferable to convert the histogram to two dimensions and obtain a plane histogram consisting of bins (intervals or classes) based on color information. Here, a "bin" is also called an interval or class, and it is the numerical range that a single bar (column) in the histogram graph possesses.

[0029] In this embodiment, it is preferable to create a two-dimensional histogram on a plane with the horizontal axis B / G and the vertical axis R / G, where the R channel and B channel are normalized by the G channel. This is preferable because it allows for differentiation of whether the green color of the leaves is predominantly red or predominantly blue, facilitating analysis.

[0030] Next, in step S304, the image processing unit 102 performs a smoothing process on the histogram. The result of calculating the histogram in step S303 often has several peaks and a jagged shape depending on the input image. Since calculating representative values ​​in this state may result in low accuracy, it is preferable to perform a smoothing process at this stage. Specifically, the histogram can be smoothed by applying a Gaussian filter or median filter with arbitrary coefficients. Other existing techniques capable of smoothing may also be used.

[0031] Figures 6(A) to 6(D) show the histogram smoothing process. Figure 6(A) is the histogram before smoothing. In Figure 6(A), the horizontal axis is B / G and the vertical axis is R / G, representing a two-dimensional histogram. This is the result of calculating a two-dimensional histogram of the frequency of RGB values ​​for each pixel in the input image after leaf region masking. Figure 6(B) is the histogram after smoothing by applying a median filter from the state in Figure 6(A). In Figures 6(A) and (B), the brighter the color of the histogram (closer to white), the higher the frequency.

[0032] Figure 6(C) shows the 2D histogram of Figure 6(A) projected onto the B / G axis (horizontal axis). Figure 6(D) shows the 2D histogram of Figure 6(B) projected onto the B / G axis (horizontal axis). In this case, the vertical axis of the histograms shown in Figures 6(C) and (D) represents the frequency. The symbols written on the histogram in Figure 6(B) will be explained later.

[0033] Next, in step S305, the image processing unit 102 determines a representative value from the smoothed histogram. Here, since it is necessary to determine the leaf color correlation value in the input image, the representative value is preferably the most frequent value (mode) or the median on the histogram. This is preferable because it allows the representative value to be determined without being affected by outliers.

[0034] The white circles in Figure 6(B) indicate the location of the mode in the histogram. In this example, there are multiple modes with the same frequency. The black circles indicate the median of the frequency distribution in the histogram. When there are multiple modes in a histogram, as in this example, the coordinates obtained by averaging the coordinates (B / G, R / G) of each mode are shown as black triangles in Figure 6(B). The representative value of a histogram can be either the "median" shown by the black circles or the "mean of the modes" shown by the black triangles. Therefore, the representative value of a histogram is the coordinate value where the mode or median exists.

[0035] Next, in step S306, the image processing unit 102 receives ambient light information from the ambient light information acquisition unit 110. Various types of ambient light sensors 111 exist on the market, but for example, an illuminance sensor or a color sensor can be used. The structure of the sensor can be an existing one; it just needs to be able to output differences in brightness due to weather and time as a numerical value. In the process shown in Figure 3, the ambient light sensor 111 acquired information regarding the color balance of the ambient light at the time of image acquisition, but here the ambient light sensor 111 acquires information regarding the brightness of the ambient light from the ambient light information at the time of image acquisition. Information regarding the brightness of the ambient light is, for example, information regarding the brightness of the light illuminating the subject when photographing the subject in order to acquire color information.

[0036] Next, in step S307, the image processing unit 102 determines the state of the ambient light based on the information regarding the brightness of the ambient light acquired by the ambient light sensor 111 and a database of ambient light information (second database). Specifically, the image processing unit 102 determines whether the subject is in a state where direct sunlight is hitting it (first state) or a state where direct sunlight is not hitting the subject (second state).

[0037] In this embodiment, the ambient light state includes a first state (sunny) in which it is determined that the subject is exposed to direct sunlight, and a second state (cloudy) in which it is determined that the subject is not exposed to direct sunlight. In this embodiment, the ambient light state may also include a third state (partly cloudy) between the first and second states.

[0038] Here, we will explain how to determine the ambient light conditions. The ambient light conditions are determined by information regarding the brightness of the ambient light acquired by the ambient light sensor 111. Information regarding the brightness of the ambient light includes, for example, information in units of lux acquired by an illuminance sensor, the signal value of a specific channel from 10-bit or 12-bit RGB data acquired by a color sensor, or information obtained by converting RGB values ​​to luminance values. It is not limited to this, as it is information that changes according to the brightness.

[0039] One example of how to determine a threshold is to calculate the ambient light brightness on the brightest day of the year at the same time (for example, 12:00 PM) by assigning a value of 100 and the information on the darkest day to 0. A value of 60 or higher would be considered sunny, and a value below 60 would be considered cloudy. Here, this value from 0 to 100 is called the "sunshine level," and it corresponds to the state of the ambient light. Determining whether the weather is sunny or cloudy is equivalent to determining the state of the ambient light. If it is determined to be sunny, it can be inferred that there is a high probability that the subject is receiving direct sunlight. Conversely, if it is determined to be cloudy, it can be inferred that there is a low probability that the subject is receiving direct sunlight. In this way, the difference in brightness due to weather can be quantified, and if the sunshine level value is above a certain level, it can be inferred that there is a high probability that the subject is receiving direct sunlight.

[0040] While the same threshold can be used uniformly, it is preferable to define different thresholds for each region with different latitudes, as this allows for more accurate determination of the model database. For example, if the ambient light brightness information is 60 or higher, the model database for sunny weather is used; otherwise, the model database for cloudy weather is used. The model database for sunny weather is obtained by calculating an arbitrary correction coefficient based on the model database for cloudy weather. The correction coefficient itself and the method for calculating the correction coefficient are not limited. This database, which includes information on ambient light brightness and threshold information according to the weather, is called the ambient light brightness database and is included in the ambient light light information database (second database). The ambient light brightness database is used to determine brightness. It is preferable to include information on the names of each region and location information such as latitude and longitude in addition to this, as this can further improve accuracy.

[0041] Here, information regarding the brightness of ambient light is used, but the degree of sunshine could also be calculated based on information regarding the color balance of ambient light, for example, the balance of each RGB channel of ambient light. In this case, the R / G and B / G values ​​for various weather conditions are stored in a database of ambient light information, and the state of ambient light is determined by calculating the degree of sunshine by comparing these values ​​with the R / G and B / G values ​​at the time of shooting. The next step, S308, is a step in which, based on the result of determining the state of ambient light (determination result), one of several model databases is decided to use. Since the leaf color correlation value cannot be determined by the representative value of the histogram alone, a database of models related to color is also used. Here, "color" refers to "plant color" or "plant leaf color". The database of models related to color (first database) will henceforth be called the "model database". That is, the first database is a database of models related to plant color or plant leaf color.

[0042] An example of a model database (first database) will be explained with reference to Figures 7(A) and (B). Figures 7(A) and (B) show an example of a model database. The group of curves shown in Figures 7(A) and (B) are a group of curves determined based on the SPAD values ​​measured by a leaf colorimeter (SPAD meter) on actual rice leaves and the RGB values ​​when those leaves are photographed by the imaging unit 101. In this embodiment, curves corresponding to SPAD values ​​of 10, 20, 30, 40, 50, and 60 are shown for clarity. However, this embodiment is not limited to this, and the number of curves can be increased in increments of 1.0 or 0.1, etc., based on the desired final accuracy. This is preferable because it can improve accuracy.

[0043] In this embodiment, the model database does not necessarily have to be extracted from the same subject (leaves) as subject 140. Different varieties are acceptable as long as the leaf growth characteristics do not differ significantly within the same crop range. Furthermore, when the model database is represented on a two-dimensional plane with the horizontal axis as B / G (B value / G value) and the vertical axis as R / G (R value / G value), at least one function within the evaluation area that outputs the leaf color correlation value included in subject 140 has an "upward convex shape". In this case, the + direction on the vertical axis is upward and the + direction on the horizontal axis is to the right.

[0044] Here, the shape of the function is very important as it represents the state of the leaves, but the function itself is not limited. Based on measured values ​​such as the SPAD value of the leaves, polynomials, logarithmic functions, exponential functions, or combinations thereof can be used. The function does not need to be differentiable over the entire numerical range; it can also be a combination of straight lines with different slopes and intercepts. Due to the need to connect the functions, there may be regions that are partially convex downwards, but the function must be convex upwards when viewed globally. The evaluation region, for example, as explained using Figure 7(A), is the range where the horizontal axis B / G is between 0.4 and 1.0, and the vertical axis R / G is between 0.7 and 1.3. This varies depending on the inherent color intensity of the leaves of the plant. For common rice varieties, this range is the evaluation region. The evaluation region can be changed depending on the plant. Therefore, outside the arbitrarily defined evaluation region, the shape of the functions and curves in the model database can be anything, and they do not need to be defined.

[0045] In this embodiment, the color information of the subject was described as a histogram represented on a two-dimensional plane with B / G on the horizontal axis and R / G on the vertical axis. However, the definitions of the vertical and horizontal axes are not limited to these; if the imaging unit can acquire information up to the infrared (IR) region, the vertical axis may be set to IR / G, for example. Using the infrared region is preferable because it is useful not only for analyzing chlorophyll but also for analyzing water content.

[0046] Next, we will explain the two types of model databases shown in Figures 7(A) and 7(B) (the first model database and the second model database included in the first database). Figure 7(A) is a model database for use in cloudy weather conditions where direct sunlight does not hit the leaves (cloudy model database or second model database). Figure 7(B) is a model database for use in sunny weather conditions where direct sunlight hits the leaves (sunny model database or first model database). Which of these model databases to use is determined by the ambient light conditions determined in the previous step. The specific method of determination is as explained in step S307 of the previous step.

[0047] Next, in step S309, the image processing unit 102 determines a representative value of the leaf color correlation value for the input image based on the representative values ​​of the model database and the histogram. Here, the method for determining the representative value of the leaf color correlation value will be explained with reference to Figure 8. Figure 8 is a diagram showing an example in which the histogram and the first database are shown together, and it shows a diagram in which Figure 6(B) and Figure 7(A) are shown together.

[0048] The representative value of the final leaf color correlation is determined by which curve the mode or median of the histogram is closest to. For example, the representative value of the leaf color correlation is determined to be the curve corresponding to the curve where the distance between the coordinate point of the mode and each group of curves is minimized. In this example, for Figure 8, for example, assuming that the degree of sunshine on this day is 43 and it is judged as cloudy, we use the cloudy day model database in Figure 7(A). In this case, since it is closest to the curve of 35.5 (not shown), we can determine that the representative value of the leaf color correlation is 35.5.

[0049] Here, rice leaves were used as the subject 140, but this can also be applied to rice ears as a variation. For example, information about the color of rice ears (color information about the subject) and a model database related to the color of rice ears can be used as input. The information about the color of rice ears is then represented as a histogram. The model database is corrected (selected or changed) based on the result of determining the ambient light conditions. Based on the histogram and the model database, numerical information correlated with the concentration of substances contained in the rice ears, which are the subject, is output. The information about rice ears in this case is an image of rice ears reflected in a paddy field, and the ears can be extracted from that image to read the color information of the ears.

[0050] Furthermore, the model database for rice ear color is a database that collects data on how rice ear color changes from green to yellow and brown, and includes information on the correlation between color and moisture content, as well as when the rice ear is ready for harvest. By masking the rice ear region and representing only the rice ears in a histogram, it is possible to output numerical information correlated with the chlorophyll concentration and moisture content of the rice ear based on the model database. In this way, the same process can be applied not only to leaves but also to other parts of the plant, such as the ear.

[0051] Furthermore, the following is preferable for the model database. In the above, a model database for sunny weather is defined as the first model database, and a model database for cloudy weather is defined as the second model database, and it is decided which one to use depending on the threshold of sunshine. In addition to this, a third model database may be defined and interpolated by further correcting the two model databases with an arbitrary correction coefficient. That is, the first database includes the first model database for sunny weather and the second model database for cloudy weather, but it may also include a third model database for partly cloudy weather that interpolates between the first and second model databases. In this embodiment, the first model database is a database obtained from measured values ​​on sunny days, the second model database is a database obtained from measured values ​​on cloudy days, and the third model database is a database obtained from measured values ​​on partly cloudy days.

[0052] For example, to bridge the gap between the cloudy model database and the sunny model database, a three-dimensional model database that changes continuously according to the degree of sunshine may be used. That is, the first database may be a three-dimensional model database representing the B / G and R / G values ​​of the RGB values ​​of the image, and the ambient light information. For example, the horizontal axis (x axis) B / G and vertical axis (y axis) R / G are the same as shown in Figure 7, but the sunshine value in the depth direction (z axis) is set to a range from 0 to 100. Then, the cloudy model database is used for sunshine from 0 to 40. The sunny model database is used for sunshine from 60 to 100. From sunshine from 40 to 60, the curves from the cloudy model database to the sunny model database are interpolated with an arbitrary numerical range. The correction coefficient in this case is the sunshine value itself. This method is preferable because it allows the gap between model databases to be filled in a gradient manner, and further accuracy can be achieved.

[0053] In this embodiment, the modification means 102c selects one model database (one of the first to third model databases) from among multiple model databases in the first database as a method for changing the processing related to the first database based on the result of determining the ambient light state. However, this embodiment is not limited to this, and the modification means 102c may also correct (change) the first database based on the result of determining the ambient light state. For example, if the first database is a model database corresponding to a sunny state, and the modification means 102c determines that the ambient light state is cloudy, it may correct (change) the model database of the first database to a model database for cloudy conditions. Alternatively, if the first database is a model database corresponding to a cloudy state, and the modification means 102c determines that the ambient light state is sunny, it may correct (change) the model database of the first database to a model database for sunny conditions.

[0054] According to this embodiment, an image processing method, an image processing apparatus, an image processing system, and a program can be provided that accurately estimate the concentration of substances contained in an image from an image.

[0055] (Example 2) Next, with reference to Figure 9, the configuration and features of the image processing system 100 in Embodiment 2 of the present invention will be described. The basic configuration of the image processing system 100 in this embodiment is the same as the configuration described with reference to Figure 1, so the description of each part will be omitted. The characteristic part described in this embodiment is that the flow shown in Figure 4 is different.

[0056] In Example 1, the model database was corrected based on the ambient light conditions due to weather, ambient light information, and a database of ambient light information to determine a representative value for leaf color correlation. In this example, however, the color information of the subject is corrected based on the ambient light conditions. Specifically, correction (masking) is performed on areas in the input image that have specific values ​​for the G channel. This masks (excludes by applying a correction value of zero) areas that are bright due to increased reflection components caused by direct sunlight, and areas that are dark shadows created when direct sunlight hits the surrounding leaves, and these areas are not used in subsequent calculations.

[0057] Figure 9 is a flowchart showing the method for determining the leaf color correlation value in this embodiment. Steps S401 and S402 in Figure 9 are the same as steps S301 and S302 in Figure 4 described in Example 1, respectively, so their explanations are omitted.

[0058] Next, in step S403, the image processing unit 102 receives ambient light information from the ambient light information acquisition unit 110. Specifically, the ambient light information is information regarding the brightness of the ambient light. Here, the configuration of the ambient light sensor 111 is the same as in Example 1.

[0059] Next, in step S404, the image processing unit 102 determines the state of the ambient light based on the ambient light information acquired by the ambient light sensor 111. Based on the determination result in step S404, the image processing unit 102 decides whether to proceed to step S405 or step S407. Specifically, as shown in Example 1, the image processing unit 102 calculates the degree of sunshine based on the ambient light information acquired by the ambient light sensor 111. The image processing unit 102 then determines a threshold for the degree of sunshine based on a database of ambient light information and determines whether the current degree of sunshine exceeds the threshold.

[0060] The threshold value can be determined by either directly reading the threshold information from a database containing ambient light information, or by selecting the currently calculated region and reading the threshold information associated with that region from the database containing location information such as region and latitude. For example, if the threshold value in the ambient light information database is 60 and the current sunshine level is 75, proceed to the next step S405. On the other hand, if the current sunshine level is 20, proceed to the next step S407. Steps S407 to S410 are the same as steps S303 to S305 and S309 in Figure 4 described in Example 1, so their explanations will be omitted.

[0061] Next, in step S405, the image processing unit 102 normalizes the G channel values ​​in the leaf region by their maximum value (the maximum value information of ambient light brightness necessary for normalization). Then, in step S406, the image processing unit 102 masks pixels having arbitrary G channel values.

[0062] Figures 10(A) to 10(C) show examples of mask processing, histograms, and color model databases side by side. Figure 10(A) is an image with only the central part of the input image extracted. Figure 10(B) is a mask image with the leaf region extracted from the image in Figure 10(A). Figure 10(C) is a superimposed display of the 2D histogram and model database of the image in Figure 10(A) after mask processing using the mask image.

[0063] When capturing the input image, if the brightness exceeds a threshold, pixels with a normalized G-channel value of 0.25 or higher are masked. The threshold of 0.25 can be arbitrarily changed depending on the reflectance and reflected light intensity of the subject's leaves. Figures 11(A) and (B) show an example of masking, histogram, and a database of a color model. Figure 11(A) shows a masked image with pixels of 0.25 or higher in the normalized G-channel masked. Figure 11(B) shows a two-dimensional histogram after applying the mask from Figure 11(A) to the image in Figure 10(A). This correction makes it possible to exclude the values ​​of pixels strongly affected by direct sunlight, enabling highly accurate determination of the leaf color correlation value. At this time, since it is closest to the curve of 32.9 (not shown), the representative value of the leaf color correlation value can be determined to be 32.9.

[0064] More preferably, as shown in Figures 12(A) and (B), if direct sunlight causes strong shadows on other leaves, it is preferable to perform a correction that also masks the shadowed areas. Figures 12(A) and (B) show examples of masking, histograms, and color model databases together. Figure 12(A) is an image in which pixels with normalized G-channel values ​​of 0.1 or less and pixels with values ​​of 0.25 or more are masked. Figure 12(B) shows a two-dimensional histogram after applying the mask from Figure 12(A) to the image in Figure 10(A).

[0065] In cases where direct sunlight casts strong shadows on other leaves, masking the shaded areas is preferable as it improves accuracy. If the G-channel values ​​are masked uniformly based on a certain threshold regardless of the weather, components not affected by direct sunlight may also be cut out. On the other hand, as in this embodiment, it is preferable to have a step (step S404) that determines whether or not to mask the G-channel values ​​according to the degree of sunshine, as this improves accuracy. At this time, since it is closest to the curve of 29.6 (not shown), the representative value of the leaf color correlation can be determined to be 29.6. It is preferable that threshold information such as 0.25 and 0.1 is also included in the database of ambient light information.

[0066] According to this embodiment, an image processing method, an image processing apparatus, an image processing system, and a program can be provided that accurately estimate the concentration of substances contained in an image from an image.

[0067] (Example 3) Next, with reference to Figure 13, the configuration and features of the image processing system 100 in this embodiment will be described. The basic configuration of the image processing system 100 in this embodiment is the same as the configuration shown in Figure 1, so the explanation of each part will be omitted. The distinctive part described in this embodiment differs from the flow shown in Figure 4.

[0068] In Example 1, the model database was corrected based on the ambient light conditions due to weather to determine representative values ​​for leaf color correlation. In contrast, in this example, color information related to the subject is corrected based on the ambient light conditions. Specifically, correction processing (masking) is performed on areas in the input image that have specific color information luminance values. This masks (excludes by applying a correction value of zero) areas that are bright due to increased reflection components from direct sunlight, and areas that are dark shadows created by direct sunlight hitting leaves, and these areas are not used in subsequent calculations.

[0069] Figure 13 is a flowchart for calculating the leaf color correlation value in this embodiment. Steps S501 to S504 in Figure 13 are the same as steps S401 to S404 in Figure 9 described in Example 2, so their explanations are omitted.

[0070] Next, in step S505, the image processing unit 102 masks pixels that have an arbitrary brightness value. Figures 14(A) and (B) show an example of a model database that combines the masking process, histogram, and color. Figure 14(A) shows the values ​​expressed as follows, when the brightness value Y is input as 10-bit values ​​for each RGB channel of the input image.

[0071] Y = R value × 0.299 + G value × 0.587 + B value × 0.114 The number of bits for each RGB channel and the coefficients applied to the values ​​of each RGB channel are not limited to these, and each coefficient may be adjusted according to the characteristics of the leaf color. Figure 14(A) shows a mask image obtained by extracting the leaf region from the image in Figure 10(A) and then masking pixels with Y=380 or greater. Figure 14(B) shows a two-dimensional histogram obtained after applying the mask from Figure 14(A) to the image in Figure 10(A). By making this correction, it is possible to exclude the values ​​of pixels that are strongly affected by direct sunlight, making it possible to determine the leaf color correlation value with high accuracy. At this time, it is closest to the curve of 32.0 (not shown), so the representative value of the leaf color correlation value can be determined to be 32.0.

[0072] More preferably, as shown in Figure 15, if direct sunlight causes strong shadows on other leaves, it is preferable to also mask the shadowed areas. Figures 15(A) and (B) show examples of mask processing, histograms, and color model databases together. Figure 15(A) is an image obtained by extracting the leaf region from the image in Figure 10(A), and then further masking pixels with Y=100 or less and pixels with Y=380 or more. Figure 15(B) shows a two-dimensional histogram after applying the mask from Figure 15(A) to the image in Figure 10(A).

[0073] In cases where direct sunlight casts strong shadows on other leaves, masking the shaded areas is preferable as it improves accuracy. If the luminance value is uniformly masked based on a certain threshold regardless of the weather, components not affected by direct sunlight may also be cut out. However, it is preferable to include a step like step S504, which determines whether or not to mask the luminance value according to the degree of sunshine, as this improves accuracy. At this time, since it is closest to the curve of 29.6 (not shown), the representative value of the leaf color correlation value can be determined to be 29.6. It is preferable that threshold information such as 380 and 100 (threshold information for the luminance value of color information) is also included in the database of ambient light information.

[0074] According to this embodiment, an image processing method, an image processing apparatus, an image processing system, and a program can be provided that accurately estimate the concentration of substances contained in an image from an image.

[0075] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0076] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that can acquire information about a subject from an image with high accuracy.

[0077] Each embodiment's disclosure includes the following methods and configurations. (Method 1) An acquisition step to acquire first information regarding the subject's color, a first database of color models, and second information regarding ambient light, A determination step of determining the state of ambient light using the second information, An image processing method characterized by having a modification step of modifying at least one of the processing related to the first information or the processing related to the first database based on the result of determining the state of the ambient light. (Method 2) In the acquisition step described above, a second database relating to the second information is acquired, The image processing method according to Method 1, characterized in that in the determination step, the state of the ambient light is determined using the second information and the second database. (Method 3) The second piece of information is information relating to the brightness of the ambient light, The image processing method according to method 2, characterized in that the second database is a database relating to the brightness of the ambient light. (Method 4) The image processing method according to method 3, characterized in that the information relating to the brightness of ambient light is information relating to the brightness of light illuminating the subject when the subject is photographed in order to obtain the first information. (Method 5) The image processing method according to method 3 or 4, characterized in that the database relating to the brightness of the ambient light includes maximum value information of the brightness of the ambient light necessary for normalization. (Method 6) The information relating to the brightness of the ambient light changes according to the brightness. The image processing method according to any one of methods 3 to 5, characterized in that the database relating to the brightness of the ambient light is used to determine the brightness. (Method 7) The image processing method according to any one of methods 1 to 6, characterized in that the state of the ambient light includes a first state in which it is determined that the subject is exposed to direct sunlight, and a second state in which it is determined that the subject is not exposed to direct sunlight. (Method 8) The aforementioned first database includes a first model database and a second model database, The image processing method according to any one of methods 1 to 7, characterized in that, in the modification step, either the first model database or the second model database is selected based on the determination result of the state of the ambient light. (Method 9) The first database includes a first model database, a second model database, and a third model database that interpolates between the first and second model databases. The image processing method according to any one of methods 1 to 7, characterized in that, in the modification step, one of the first model database, the second model database, or the third model database is selected based on the determination result of the state of the ambient light. (Method 10) The image processing method according to any one of methods 1 to 7, characterized in that, in the modification step, the first database is corrected based on the determination result of the state of the ambient light. (Method 11) The image processing method according to any one of methods 1 to 10, characterized in that the first database is a model database represented by the B value / G value and R value / G value of the RGB values ​​of an image, and the second information. (Method 12) The image processing method according to any one of methods 1 to 7, characterized in that, in the modification step, the first information is subjected to masking based on the determination result of the state of the ambient light. (Method 13) In the acquisition step described above, input the second database relating to the second information, The image processing method according to method 12, characterized in that the second database has threshold information for the G channel value among the RGB values ​​of the first information. (Method 14) In the acquisition step described above, input the second database relating to the second information, The image processing method according to method 12, characterized in that the second database has threshold information for the brightness value of the first information. (Method 15) The image processing method according to any one of methods 8 to 10, further comprising an output step of outputting numerical information correlated with the concentration of substances contained in the subject, based on a histogram obtained from the first information and the first database. (Method 16) The image processing method according to method 15, characterized in that the histogram is a plane histogram consisting of bins based on the first information. (Method 17) The image processing method according to method 15 or 16, characterized in that the histogram is a histogram of the plane of R-value / G-value and B-value / G-value obtained by inputting the first information. (Method 18) The image processing method according to any one of methods 15 to 17, characterized in that, in the output step, the histogram is smoothed, a representative value of the histogram is calculated, and the numerical information is output based on the representative value and the first database. (Method 19) The image processing method according to any one of methods 15 to 18, characterized in that the representative value of the histogram is determined based on the mode or median of the histogram. (Method 20) The first database is characterized in that, when the graph is represented on a plane where the horizontal axis is B value / G value and the vertical axis is R value / G value, with the positive direction of the vertical axis being upward and the positive direction of the horizontal axis being to the right, at least one of the functions has an upward-convex shape. This is the image processing method according to any one of methods 1 to 19. (Method 21) The image processing method according to any one of methods 1 to 20, characterized in that the first database is a database of models relating to plant colors. (Method 22) The image processing method according to any one of methods 1 to 21, characterized in that the first information is image information of a plant acquired using a camera having at least three RGB channels. (Method 23) The image processing method according to any one of methods 1 to 22, characterized in that the first information is numerical information that correlates with at least the concentration of a substance contained in the subject. (Composition 1) An acquisition means for acquiring first information about the subject, a first database of a color model, and second information about ambient light, A determination means for determining the state of ambient light using the second information, An image processing apparatus characterized by having a modification means for modifying at least one of the processing relating to the first information or the processing relating to the first database based on the determination result of the state of the ambient light. (Configuration 2) The imaging unit captures images of the subject, An image processing system characterized by having the image processing device described in Configuration 1. (Composition 3) A program characterized by causing a computer to execute an image processing method described in any of methods 1 to 23.

[0078] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. At least some of the embodiments may be combined. [Explanation of Symbols]

[0079] 102 Image Processing Unit (Image Processing Device) 102a Input means (acquisition means) 102b Judgment means 102c Modification method

Claims

1. An acquisition step to acquire first information regarding the subject's color, a first database of color models, and second information regarding ambient light, A determination step of determining the state of ambient light using the second information, An image processing method characterized by having a modification step of modifying at least one of the processing related to the first information or the processing related to the first database based on the result of determining the state of the ambient light.

2. In the acquisition step described above, a second database relating to the second information is acquired, The image processing method according to claim 1, characterized in that in the determination step, the state of the ambient light is determined using the second information and the second database.

3. The second piece of information is information relating to the brightness of the ambient light, The image processing method according to claim 2, characterized in that the second database is a database relating to the brightness of the ambient light.

4. The image processing method according to claim 3, characterized in that the information relating to the brightness of ambient light is information relating to the brightness of light illuminating the subject when the subject is photographed in order to acquire the first information.

5. The image processing method according to claim 3, characterized in that the database relating to the brightness of the ambient light includes maximum value information of the brightness of the ambient light necessary for normalization.

6. The information relating to the brightness of the ambient light changes according to the brightness. The image processing method according to claim 3, characterized in that the database relating to the brightness of the ambient light is used to determine the brightness.

7. The image processing method according to claim 1, characterized in that the state of the ambient light includes a first state in which it is determined that the subject is exposed to direct sunlight, and a second state in which it is determined that the subject is not exposed to direct sunlight.

8. The first database includes a first model database and a second model database, The image processing method according to claim 1, characterized in that, in the modification step, either the first model database or the second model database is selected based on the determination result of the state of the ambient light.

9. The first database includes a first model database, a second model database, and a third model database that interpolates between the first model database and the second model database. The image processing method according to claim 1, characterized in that, in the modification step, one of the first model database, the second model database, or the third model database is selected based on the determination result of the state of the ambient light.

10. The image processing method according to claim 1, characterized in that, in the modification step, the first database is corrected based on the determination result of the state of the ambient light.

11. The image processing method according to claim 1, characterized in that the first database is a model database represented by the B / G value and R / G value of the RGB values ​​of an image, and the second information.

12. The image processing method according to claim 1, characterized in that, in the modification step, the first information is subjected to masking based on the determination result of the state of the ambient light.

13. In the acquisition step described above, input the second database relating to the second information, The image processing method according to claim 12, characterized in that the second database has threshold information for the G channel value among the RGB values ​​of the first information.

14. In the acquisition step described above, input the second database relating to the second information, The image processing method according to claim 12, characterized in that the second database has threshold information for the brightness value of the first information.

15. The image processing method according to claim 8, further comprising an output step of outputting numerical information correlated with the concentration of substances contained in the subject, based on a histogram obtained from the first information and the first database.

16. The image processing method according to claim 15, characterized in that the histogram is a plane histogram consisting of bins based on the first information.

17. The image processing method according to claim 15, characterized in that the histogram is a histogram of the plane of R-value / G-value and B-value / G-value obtained by inputting the first information.

18. The image processing method according to claim 15, characterized in that, in the output step, the histogram is smoothed, a representative value of the histogram is calculated, and the numerical information is output based on the representative value and the first database.

19. The image processing method according to claim 15, characterized in that the representative value of the histogram is determined based on the mode or median of the histogram.

20. The image processing method according to any one of claims 1 to 19, characterized in that when the first database is represented on a plane in which the horizontal axis of the graph is B value / G value and the vertical axis is R value / G value, with the positive direction of the vertical axis being upward and the positive direction of the horizontal axis being to the right, at least one of the functions has an upward convex shape.

21. The image processing method according to any one of claims 1 to 19, characterized in that the first database is a database of models relating to plant colors.

22. The image processing method according to any one of claims 1 to 19, characterized in that the first information is image information of a plant acquired using a camera having at least three RGB channels.

23. The image processing method according to any one of claims 1 to 19, characterized in that the first information is numerical information that correlates with at least the concentration of a substance contained in the subject.

24. An acquisition means for acquiring first information about the subject, a first database of a color model, and second information about ambient light, A determination means for determining the state of ambient light using the second information, An image processing apparatus characterized by having a modification means for modifying at least one of the processing relating to the first information or the processing relating to the first database based on the determination result of the state of the ambient light.

25. The imaging unit captures images of the subject, An image processing system characterized by having the image processing apparatus described in claim 24.

26. A program characterized by causing a computer to execute the image processing method described in any one of claims 1 to 19.

Images