Image processing method, image processing system, and program

Abstract

Provided is an image processing method for displaying a dental plaque region on the basis of an RGB image obtained by imaging teeth and dental plaque undergoing fluorescence reaction by irradiating the teeth with light including a wavelength region of blue light, the image processing method comprising: acquiring an input level of external light entering an imaging region for the RGB image (S12); performing dental plaque detection processing on the basis of the RGB image (S13); and displaying the result of dental plaque detection in different manners depending on the incident level of the external light entering the imaging region for the RGB image (S15, S16).

Description

Image processing method, image processing system, and program

[0001] The present disclosure relates to an image processing method, an image processing system, and a program.

[0002] Japanese Patent Application Laid-Open No. 2003-124222 discloses an image processing device that performs color correction on an image captured in an unshaded state.

[0003] Japanese Patent Application Laid-Open No. 2008-244794

[0004] However, there is a possibility that plaque regions in the oral cavity may be detected by taking an image of the oral cavity with an imaging device. In this case, when plaque is detected in the oral cavity using the imaging device while external light is incident on the oral cavity, there is a risk that the plaque region may be erroneously detected.

[0005] Therefore, the present disclosure provides an image processing method, an image processing system, and a program that can notify the user that a plaque region may have been erroneously detected due to external light entering the oral cavity.

[0006] An image processing method according to one aspect of the present disclosure is an image processing method for displaying a plaque area based on an RGB image obtained by photographing teeth and plaque that are undergoing a fluorescent reaction when light including a wavelength range of blue light is irradiated onto the teeth, the image processing method obtaining the incident level of external light entering the photographed area of ​​the RGB image, performing plaque detection processing based on the RGB image, and varying the display mode of the plaque detection result depending on the incident level of external light entering the photographed area of ​​the RGB image.

[0007] An image processing system according to one aspect of the present disclosure is an image processing system for displaying a plaque area based on an RGB image obtained by photographing teeth and plaque that are undergoing a fluorescent reaction when light including a wavelength range of blue light is irradiated onto the teeth, and the system includes an acquisition unit that acquires the incident level of external light entering the photographed area of ​​the RGB image, a plaque detection unit that performs plaque detection processing based on the RGB image, and a display control unit that changes the display mode of the plaque detection result depending on the incident level of external light entering the photographed area of ​​the RGB image.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the image processing method described above.

[0009] According to one aspect of the present disclosure, it is possible to realize an image processing method or the like that can notify that a plaque region may have been erroneously detected due to external light entering the oral cavity.

[0010] FIG. 1 is a perspective view of an intraoral camera in an intraoral camera system according to an embodiment. FIG. 2 is a schematic diagram of the intraoral camera system according to an embodiment. FIG. 3 is a block diagram showing the functional configuration of a mobile terminal according to an embodiment. FIG. 4 is a first diagram plotting the determination results of the presence or absence of external light according to an embodiment. FIG. 5 is a second diagram plotting the determination results of the presence or absence of external light according to an embodiment. FIG. 6 is a diagram showing a setting screen for determining external light according to an embodiment. FIG. 7 is a diagram showing an example display of a captured image and a plaque detection result in the presence or absence of external light according to an embodiment. FIG. 8 is a flowchart showing the operation of a mobile terminal according to an embodiment. FIG. 9 is a diagram showing an example display of the presence or absence of external light according to an embodiment. FIG. 10 is a diagram showing an example display prompting a user to block light according to an embodiment. FIG. 11 is a diagram schematically showing a captured image divided into multiple regions according to a first modification of the embodiment. FIG. 12 is a diagram showing the relationship between a value based on each pixel value and the number of pixels in a scene with external light. FIG. 13 is a diagram showing the relationship between a value based on each pixel value and the number of pixels in a scene without external light. FIG. 14 is a flowchart showing the operation of a mobile terminal according to a second modification of the embodiment. FIG. 15A is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from the upper left. FIG. 15B is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from the upper right. FIG. 15C is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from the lower right. FIG. 15D is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from the lower left. FIG. 15E is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from above. FIG. 15F is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from the right. FIG. 15G is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from below. FIG. 15H is a diagram showing an example of a determination result of the incidence level of each of the four divided regions when external light is incident from the left.FIG. 16A is a diagram showing another example of the determination result of the incidence level of each of the four divided regions when external light is incident from the upper left. FIG. 16B is a diagram showing another example of the determination result of the incidence level of each of the four divided regions when external light is incident from the upper right. FIG. 16C is a diagram showing another example of the determination result of the incidence level of each of the four divided regions when external light is incident from the lower right. FIG. 16D is a diagram showing another example of the determination result of the incidence level of each of the four divided regions when external light is incident from the lower left. FIG. 17A is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper left. FIG. 17B is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper right. FIG. 17C is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower left. FIG. 17D is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower right. FIG. 17E is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the left. FIG. 17F is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from above. FIG. 17G is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from below. FIG. 17H is a diagram showing a first example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the right. FIG. 18A is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper left. FIG. 18B is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper right. FIG. 18C is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower left. FIG. 18D is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower right. 18E and 18F are diagrams showing third examples of the determination results of the incidence level of each of the nine divided regions when external light is incident from the upper left and upper right, respectively.FIG. 18G is a diagram showing a third example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the bottom left. FIG. 18H is a diagram showing a third example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the bottom right. FIG. 19A is a diagram showing a fourth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the top left. FIG. 19B is a diagram showing a fourth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the top right. FIG. 19C is a diagram showing a fourth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the bottom left. FIG. 19D is a diagram showing a fourth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the bottom right. FIG. 19E is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the left. FIG. 19F is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from above. FIG. 19G is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from below. FIG. 19H is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the right. FIG. 20A is a diagram showing a fifth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper left. FIG. 20B is a diagram showing a fifth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper right. FIG. 20C is a diagram showing a fifth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower left. FIG. 20D is a diagram showing a fifth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower right. FIG. 20E is a diagram showing a sixth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper left. FIG. 20F is a diagram showing a sixth example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper right. 20G and 20H are diagrams showing sixth and sixth examples of the determination results of the incidence level of each of the nine divided regions when external light is incident from the bottom left and bottom right, respectively.FIG. 21A is a diagram showing a seventh example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper left. FIG. 21B is a diagram showing a seventh example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the upper right. FIG. 21C is a diagram showing a seventh example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower left. FIG. 21D is a diagram showing a seventh example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the lower right. FIG. 21E is a diagram showing a third example of the determination result of the incidence level of each of the nine divided regions when external light is incident from above. FIG. 21F is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from below. FIG. 21G is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the left. FIG. 21H is a diagram showing a second example of the determination result of the incidence level of each of the nine divided regions when external light is incident from the right. Fig. 22 is a diagram showing an example of first countermeasure information according to Modification 2 of the embodiment. Fig. 23 is a diagram showing an example of second countermeasure information according to Modification 2 of the embodiment. Fig. 24 is a diagram showing an example of third countermeasure information according to Modification 2 of the embodiment.

[0011] (Background to the Invention of the Present Disclosure) Prior to describing the present disclosure, the background to the invention of the present disclosure will be described with reference to FIGS. 12 and 13. FIG. 12 is a diagram showing the relationship between the number of pixels and a value based on each pixel value in a scene with external light. FIG. 13 is a diagram showing the relationship between the number of pixels and a value based on each pixel value in a scene without external light. (a) of FIG. 12 and (a) of FIG. 13 show the relationship between the number of pixels and each pixel value of an image (RGB image) formed by light received by an imaging element with and without external light. The vertical axis indicates the number of pixels, and the horizontal axis indicates the pixel value. (b) of FIG. 12 and (b) of FIG. 13 show the relationship between the number of pixels and the value of each parameter when the image is HSV converted. The vertical axis indicates the number of pixels, and the horizontal axis indicates the value of each parameter. Note that each parameter includes h (hue), s (saturation), v (lightness), and y (luminance). The horizontal axis indicates values ​​for 8 bits.

[0012] 12 and 13 are results obtained when the intraoral camera capturing the image of the oral cavity is equipped with a blue light blocking filter, as described below. Furthermore, "external light present" means that external light is present in the oral cavity, and will hereinafter also be referred to as "external light is incident," while "no external light present" means that external light is absent in the oral cavity, and will hereinafter also be referred to as "external light is not incident." Furthermore, "no external light present" can include not only the complete absence of external light, but also the substantial absence of external light.

[0013] As shown in Figure 12(a), when there is external light, the number of pixels with high pixel values ​​in red is significantly higher (see the dashed frame). For example, among red, blue, and green, only red has a large number of pixels with high pixel values ​​(high Rch). In this case, the image appears reddish overall.

[0014] 12B, when there is external light, there are many pixels with high values ​​(high V) for only lightness among the hue, saturation, lightness, and luminance (see dashed frame). In other words, the brightness of the image is high.

[0015] As is known from quantitative visible light induced fluorescence (QLF), bacteria in dental plaque fluoresce reddish-pink when irradiated with blue light (excitation fluorescence). Therefore, if external light causes the image to appear reddish overall, it becomes difficult to detect dental plaque regions from the image. For example, if external light is irradiated onto an area where there is no dental plaque, the area may be detected as containing dental plaque due to the influence of the external light.

[0016] As shown in FIG. 13A, when there is no external light, the number of pixels with high pixel values ​​in red is small (the number of pixels with high Rch is small) (see the dashed frame).

[0017] As shown in FIG. 13(b), when there is no external light, the number of pixels with high brightness is small (the number of pixels with high V is small) (see the dashed frame).

[0018] Thus, the number of pixels with high red pixel values ​​varies greatly depending on whether external light is present or not. The present inventors focused on the number of pixels with high red pixel values ​​and found that this number can be used as an example of a method for determining the presence or absence of external light. The present inventors then conducted extensive research into image processing methods and the like that can notify the user of the possibility that a plaque region has been erroneously detected due to external light entering the oral cavity, for example, using the number of pixels with high red pixel values, and devised the image processing methods and the like described below.

[0019] An image processing method according to a first aspect of the present disclosure is an image processing method for displaying a plaque area based on an RGB image obtained by photographing teeth and plaque that are undergoing a fluorescent reaction when light including a wavelength range of blue light is irradiated onto the teeth, the image processing method obtaining the incident level of external light entering the photographed area of ​​a first RGB image, performing plaque detection processing based on the first RGB image, and varying the display mode of the plaque detection result depending on the incident level of external light entering the photographed area of ​​the first RGB image.

[0020] This makes it possible to change the display mode of the plaque detection result depending on whether external light is incident into the oral cavity. When external light is incident into the oral cavity, there is a risk that the plaque region included in the plaque detection result may have been erroneously detected. For example, when external light is incident into the oral cavity, the display mode is changed so as to be able to notify the user that external light is incident, thereby realizing an image processing method that can notify the user that the plaque region may have been erroneously detected due to external light being incident into the oral cavity.

[0021] Furthermore, for example, the image processing method according to the second aspect may be the image processing method according to the first aspect, and when the incident level is not below a predetermined value, an image displaying the plaque detection result may be displayed with a display indicating that external light is incident superimposed on it.

[0022] This makes it possible to notify the user that there is a possibility that external light is entering, that is, that there is a possibility that the plaque region has been erroneously detected.

[0023] Furthermore, for example, the image processing method according to the third aspect may be an image processing method according to the first or second aspect, and when the incident level is below the predetermined value, an image displaying the plaque detection result may be displayed with a display indicating that no external light is incident superimposed on it.

[0024] This makes it possible to notify the user that no external light is being incident, that is, that there is a high possibility that the plaque region has been correctly determined.

[0025] Furthermore, for example, the image processing method according to the fourth aspect may be the image processing method according to the first aspect, wherein the display of the plaque detection result is prohibited when the incident level is not equal to or lower than a predetermined value.

[0026] This makes it possible to prevent plaque detection results that may include erroneous detection results from being displayed to the user when there is a possibility that external light is incident.

[0027] Also, for example, an image processing method according to a fifth aspect may be an image processing method according to any one of the first to fourth aspects, and the incident level may be acquired based on the detection result of a detector that detects external light incident on the shooting area of ​​the first RGB image.

[0028] This allows the detector to directly detect the presence or absence of external light, making it possible to more reliably detect the presence or absence of external light and also reduce the amount of processing required to determine the presence or absence of external light in the processing device that executes the image processing method.

[0029] Also, for example, an image processing method according to a sixth aspect may be an image processing method according to any one of the first to fifth aspects, in which the incidence level is calculated based on a first number of red pixels having a red pixel value equal to or greater than a first pixel value in the first RGB image.

[0030] This makes it possible to determine the presence or absence of external light based on the number of first red pixels, which is likely to vary depending on the presence or absence of external light.

[0031] Further, for example, an image processing method according to a seventh aspect may be the image processing method according to the sixth aspect, wherein the incidence level includes a first incidence level, the first red pixel count includes a first number of red pixels having the pixel value at each pixel value equal to or greater than the first pixel value, and the number of green pixels having the green pixel value equal to or greater than the first pixel value includes a second number of green pixels having the pixel value at each pixel value equal to or greater than the first pixel value, and the image processing method may calculate the first incidence level based on a first integrated value obtained by subtracting the second pixel count from the first pixel count for each pixel value equal to or greater than the first pixel value, and accumulating the first subtraction value obtained by subtracting the second pixel number from the first pixel number, where the first subtraction value is positive, and may vary the display manner of the plaque detection result depending on whether the first incidence level is equal to or less than a first predetermined value.

[0032] This allows the presence or absence of external light to be determined using the first incident level based on the first integrated value, and the display mode of the plaque detection result can be changed depending on the determination result of the presence or absence of external light.

[0033] Furthermore, for example, an image processing method according to an eighth aspect may be the image processing method according to the sixth aspect, wherein the incidence level includes a first incidence level, the first red pixel count includes a first number of red pixels having a pixel value equal to or greater than the first pixel value, and the number of green pixels having a green pixel value equal to or less than a fourth pixel value lower than the first pixel value includes a second number of green pixels having a pixel value equal to or less than the fourth pixel value, and the image processing method may subtract the second pixel count from the first pixel count, and if a first subtraction value obtained by subtracting the second pixel number from the first pixel number is positive, calculate the first incidence level based on the first subtraction value, and vary the display manner of the plaque detection result depending on whether the first incidence level is equal to or less than a first predetermined value.

[0034] This allows the presence or absence of external light to be determined using the first incident level based on the first integrated value, and the display mode of the plaque detection result can be changed depending on the determination result of the presence or absence of external light.

[0035] Further, for example, an image processing method according to a ninth aspect may be the image processing method according to the seventh aspect, wherein the incidence level includes a second incidence level, and the second number of red pixels having a red pixel value equal to or greater than a second pixel value higher than the first pixel value includes a third number of red pixels having that pixel value at each pixel value equal to or greater than the second pixel value, and the image processing method may calculate the second incidence level based on a second integrated value obtained by subtracting the second pixel number from the third pixel number at each pixel value equal to or greater than the second pixel value, and accumulating the second subtraction value, which is positive, and may vary the display manner of the plaque detection result depending on whether the second incidence level is equal to or less than a second predetermined value.

[0036] By using the second integrated value, it is possible to determine that external light is present even when external light is incident on a localized area (for example, when a localized area of ​​a tooth is shiny), thereby preventing erroneous detection in which it is determined that no external light is present even when external light is incident on the localized area.

[0037] Furthermore, for example, an image processing method according to a tenth aspect is an image processing method according to any one of the seventh to ninth aspects, in which the incident level includes a third incident level, a V image is generated consisting of V values ​​obtained by HSV conversion of each pixel value of the first RGB image, the third incident level is calculated based on the accumulated value of the number of pixels whose V value is equal to or greater than a third pixel value, and the display mode of the plaque detection result may be changed depending on whether the third incident level is equal to or less than a third predetermined value.

[0038] This allows the display mode of the plaque detection result to be changed based on the number of red pixels and brightness.

[0039] Furthermore, for example, the image processing method according to the eleventh aspect may be the image processing method according to the tenth aspect, wherein when the first incident level is less than the first predetermined value and greater than a fourth predetermined value that is lower than the first predetermined value, and the third incident level is less than the third predetermined value and greater than a fifth predetermined value that is lower than the third predetermined value, an image is displayed on which a message urging the user to block light is superimposed on the image displaying the plaque detection result.

[0040] This encourages shading when it is unclear whether external light is entering. By shading when external light is entering, the effects of external light can be suppressed. Also, by shading when external light is not entering, it is possible to confirm that external light is not entering.

[0041] Furthermore, for example, the image processing method according to the twelfth aspect may be the image processing method according to the sixth aspect, in which a V image is generated consisting of V values ​​obtained by HSV conversion of each pixel value of the first RGB image, and the incident level is calculated based on the accumulated value of the number of pixels whose V values ​​are equal to or greater than the first pixel value.

[0042] This makes it possible to determine the presence or absence of external light based on the V value.

[0043] Also, for example, the image processing method according to the thirteenth aspect is an image processing method according to any one of the second to fourth aspects, and the predetermined value may be set according to the lighting environment of the space in which the first RGB image is captured.

[0044] This allows a predetermined value appropriate for each lighting environment to be set, making it possible to accurately determine the presence or absence of external light even when the lighting environment changes.

[0045] Furthermore, for example, an image processing method according to a fourteenth aspect may be an image processing method according to any one of the second to fourth aspects, in which the first RGB image is divided into a plurality of regions, the incident level of external light incident on each of the divided regions is obtained, and the display mode of the plaque detection result is changed depending on the determination result of whether the incident level of each of the plurality of regions is below the predetermined value.

[0046] This allows the determination of the presence or absence of external light to be performed over a smaller area, so that even if external light is incident on a localized area, it is possible to accurately determine that external light is present, thereby suppressing erroneous detections in which it is determined that there is no external light even when external light is incident on a localized area.

[0047] Furthermore, for example, the image processing method according to the fifteenth aspect may be an image processing method according to any one of the second to fourth aspects, and may display countermeasure information for suppressing the effects of the external light according to the incident level.

[0048] As a result, when the countermeasure information is implemented by the user, it is possible to reduce the level of incident external light, thereby preventing external light from entering the oral cavity.

[0049] Furthermore, for example, an image processing method according to a 16th aspect may be the image processing method according to the 15th aspect, in which the first RGB image is divided into a plurality of regions, the incident level of external light incident on each of the divided regions is obtained, and countermeasure information is displayed according to at least one of the number of regions among the plurality of regions in which the incident level exceeds the predetermined value and the position of the region among the plurality of regions.

[0050] This makes it possible to display the number of regions into which external light is incident and information on measures to be taken depending on the direction in which the external light is incident, thereby effectively preventing external light from entering the oral cavity.

[0051] Furthermore, for example, the image processing method according to the seventeenth aspect may be the image processing method according to the sixteenth aspect, further comprising a determination unit that determines whether or not the user has responded to the countermeasure information, and after it is determined that the user has responded to the countermeasure information, acquires the post-countermeasure incident levels for each of the plurality of regions in a second RGB image obtained by photographing the teeth and plaque that are fluorescently reacting by irradiating the teeth with light including the wavelength range of blue light, and displays countermeasure information other than the countermeasure information based on the post-countermeasure incident levels for each of the plurality of regions.

[0052] This allows other countermeasure information to be displayed according to the external light reduction effect of the displayed countermeasure information. For example, if the external light reduction effect of the countermeasure information is insufficient, other countermeasure information that can further suppress the effects of external light can be displayed. Therefore, it is possible to more reliably suppress external light from entering the oral cavity.

[0053] Also, for example, the image processing method according to the 18th aspect may be the image processing method according to the 17th aspect, in which the other countermeasure information may be information corresponding to at least one of the number of areas among the plurality of areas in which the incident level after the countermeasure exceeds the predetermined value and the position of the area among the plurality of areas.

[0054] This makes it possible to display information on different measures depending on the number of regions into which external light is incident and the direction of incidence of the external light, thereby more effectively suppressing external light from entering the oral cavity.

[0055] Furthermore, for example, an image processing method according to a 19th aspect may be an image processing method according to any one of the 16th to 18th aspects, in which the direction in which the external light is incident is determined based on the positional relationship of areas among the plurality of areas in which the incident level exceeds the predetermined value, and the countermeasure information may include information regarding the direction in which the external light is incident.

[0056] This makes it possible to display countermeasure information that places particular emphasis on the incident direction of external light.

[0057] An image processing system according to a twentieth aspect of the present disclosure is an image processing system for displaying a plaque region based on an RGB image obtained by photographing teeth and plaque that are undergoing a fluorescent reaction when teeth are irradiated with light including a wavelength range of blue light, the image processing system including: an acquisition unit that acquires an incident level of external light incident on the photographed region of the RGB image; a plaque detection unit that performs plaque detection processing based on the RGB image; and a display control unit that changes the display mode of the plaque detection result depending on the incident level of external light incident on the photographed region of the RGB image. Also, a program according to a twentieth aspect of the present disclosure is a program for causing a computer to execute the image processing method according to any one of the first to nineteenth aspects.

[0058] This provides the same effect as the image processing method described above.

[0059] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0060] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.

[0061] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0062] Furthermore, in this specification, terms indicating the relationship between elements, such as "same," terms indicating the shape of elements, such as "L-shaped," "U-shaped," and "rectangular," as well as numerical values ​​and numerical ranges, are not expressions that only express a strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about several percent (or about 10%).

[0063] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0064] (Embodiment) Hereinafter, an intraoral camera system and an image processing method according to the present embodiment will be described with reference to Figs.

[0065] [1. Configuration of Intraoral Camera System] First, the configuration of an intraoral camera system including an intraoral camera according to the present embodiment will be described with reference to Figures 1 to 3. Figure 1 is a perspective view of an intraoral camera 10 in the intraoral camera system according to the present embodiment.

[0066] As shown in FIG. 1, the intraoral camera 10 has a toothbrush-shaped housing that can be handled with one hand, and the housing has a head portion 10a that is placed in the user's oral cavity when photographing the dentition, a handle portion 10b that the user holds, and a neck portion 10c that connects the head portion 10a and the handle portion 10b.

[0067] The photographing unit 21 is incorporated into the head unit 10a and the neck unit 10c. The photographing unit 21 has an image pickup element (not shown) and a lens (not shown) arranged on its optical axis LA.

[0068] The imaging element is a photographing device such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) element, and an image of the teeth is formed by a lens. The imaging element outputs a signal (image data) corresponding to the formed image to the outside. The row of teeth image captured by the imaging element is an example of an RGB image. The row of teeth image may be an image of the side of the teeth or an image of the occlusal surface of the teeth. The side of the teeth may be the inner (lingual) side or the outer (cheek) side.

[0069] The imaging unit 21 may further include a light-blocking filter that blocks light of a color irradiated from the illumination unit and transmits fluorescence emitted by plaque in response to the light. In this embodiment, the imaging unit 21 may include a blue-light blocking filter that blocks blue wavelength components contained in light incident on the image sensor 14. When irradiating teeth with light including a blue wavelength range to detect plaque, increasing the light including the blue wavelength range to enhance the excitation fluorescence of plaque causes the entire RGB image to appear blue. In this state, blue pixel values ​​become dominant over red and green pixel values, which may reduce the accuracy of determining the presence or absence of external light using red and green pixel values. To address this issue, a blue-light blocking filter blocks light including the blue wavelength range from the light before it enters the image sensor. The imaging unit 21 does not necessarily have to include a blue-light blocking filter.

[0070] The intraoral camera 10 is also equipped with a plurality of first to fourth light-emitting diodes (LEDs) 23A-23D as an illumination unit (illumination device) that irradiates the teeth to be photographed during photography. The first to fourth LEDs 23A-23D irradiate dental plaque with light of a color (e.g., single-color light) that causes the plaque to fluoresce when irradiated thereon. The first to fourth LEDs 23A-23D are, for example, blue LEDs that irradiate blue light including a wavelength having a peak at 405 nm. Note that the first to fourth LEDs 23A-23D are not limited to blue LEDs and may be any light source that irradiates light including the blue light wavelength range.

[0071] FIG. 2 is a schematic configuration diagram of the intraoral camera system according to the present embodiment.

[0072] As shown in FIG. 2 , the intraoral camera system according to this embodiment is generally configured such that the imaging unit 21 captures fluorescence emitted by plaque in response to light from the illumination unit 23 .

[0073] 2, the intraoral camera system includes an intraoral camera 10 and a mobile terminal 50. The intraoral camera system is an example of an image processing system.

[0074] The intraoral camera system is configured to notify the user that, when plaque regions are detected based on an image (RGB image) captured by the intraoral camera 10 while external light is entering the oral cavity, the plaque regions may have been erroneously detected due to the external light entering the oral cavity. The notification may be achieved by displaying information indicating the possibility of erroneous detection on the image (e.g., see FIG. 9 described below), or by outputting the possibility of erroneous detection by sound, light, or the like. Furthermore, external light refers to light other than the light emitted by the illumination unit 23, and examples include illumination light and sunlight. External light may include, for example, light of the same color as the fluorescence emitted by plaque.

[0075] The intraoral camera 10 includes a hardware unit 20, a signal processing unit 30, and a communication unit 40.

[0076] The hardware unit 20 is a physical element of the intraoral camera 10 and includes an imaging unit 21 , a sensor unit 22 , an illumination unit 23 , and an operation unit 24 .

[0077] The photographing unit 21 generates image data by photographing the teeth in the user's oral cavity. The photographing unit 21 receives a control signal from the camera control unit 31, performs operations such as photographing in accordance with the received control signal, and outputs image data of a moving or still image obtained by photographing to the image processing unit 32. The photographing unit 21 has the above-mentioned image sensor, a light-blocking filter, and a lens. The photographing unit 21 generates image data based on light that has passed through the light-blocking filter. Furthermore, the image data is a row of teeth image that shows multiple teeth, but it is sufficient that the row of teeth image shows at least one tooth.

[0078] The sensor unit 22 detects external light incident on the imaging area of ​​the RGB image. For example, the sensor unit 22 detects whether external light is incident into the oral cavity. The sensor unit 22 is disposed, for example, near the imaging unit 21. The sensor unit 22 may be disposed, for example, in the head unit 10a of the intraoral camera 10, similar to the imaging unit 21. In other words, the sensor unit 22 is located inside the user's oral cavity when the imaging unit 21 captures an image. The sensor unit 22 is an example of a detector.

[0079] The illumination unit 23 irradiates light (blue light in this embodiment) onto the region of the multiple regions in the oral cavity that is to be photographed by the photographing unit 21. The illumination unit 23 has the above-mentioned first to fourth LEDs 23A to 23D. The first to fourth LEDs 23A to 23D irradiate the photographing region with light from, for example, different directions from each other. This makes it possible to prevent shadows from appearing in the photographing region.

[0080] Each of the first to fourth LEDs 23A to 23D is configured to be at least controllable in terms of dimming. Each of the first to fourth LEDs 23A to 23D may be configured to be controllable in terms of dimming and color adjustment. The first to fourth LEDs 23A to 23D are arranged to surround the imaging unit 21.

[0081] The illumination unit 23 controls the illumination intensity (light emission intensity) according to the photographing area. The illumination intensity of each of the first to fourth LEDs 23A to 23D may be controlled uniformly, or may be controlled to be different from one another. The number of LEDs included in the illumination unit 23 is not particularly limited, and may be one, or five or more. Furthermore, the illumination unit 23 is not limited to having an LED as a light source, and may include other light sources.

[0082] The operation unit 24 receives operations from the user. The operation unit 24 is configured with, for example, push buttons, but may also be configured to receive operations by voice.

[0083] The hardware unit 20 may further include a battery (e.g., a secondary battery) that supplies power to each component of the intraoral camera 10, a coil for wireless charging by an external charger connected to a commercial power source, and an actuator necessary for at least one of composition adjustment and focus adjustment.

[0084] The signal processing unit 30 has functional components implemented by a CPU (Central Processing Unit) or an MPU (Micro Processor Unit) that execute various processes described below, and a memory unit 35 such as a ROM (Read Only Memory) or RAM (Random Access Memory) that stores programs for causing the functional components to execute various processes. The signal processing unit 30 has a camera control unit 31, an image processing unit 32, a control unit 33, a lighting control unit 34, and the memory unit 35.

[0085] The camera control unit 31 is mounted on, for example, the handle unit 10b of the intraoral camera 10 and controls the imaging unit 21. The camera control unit 31 controls at least one of the aperture and the shutter speed of the imaging unit 21 in response to a control signal from the image processing unit 32, for example.

[0086] The image processing unit 32 is mounted on, for example, the handle unit 10b of the intraoral camera 10, acquires the row of teeth image (image data) captured by the imaging unit 21, performs image processing on the acquired row of teeth image, and outputs the row of teeth image after the image processing to the camera control unit 31 and the control unit 33. The image processing unit 32 may also output the row of teeth image after the image processing to the memory unit 35, and store the row of teeth image after the image processing in the memory unit 35.

[0087] The image processing unit 32 is configured, for example, by a circuit, and performs image processing on the row-of-teeth image, such as noise removal, AWB (Automatic White Balance) processing, and contour enhancement processing. For example, the AWB processing may be performed mainly on the tooth region in the row-of-teeth image.

[0088] The row-of-teeth image output from the image processing unit 32 (the row-of-teeth image after image processing) may be transmitted to the mobile terminal 50 via the communication unit 40, and the transmitted row-of-teeth image may be displayed on the touch screen 54 of the mobile terminal 50. This allows the row-of-teeth image to be presented to the user.

[0089] The control unit 33 detects the area of ​​plaque based on the image data that has been image-processed by the image processing unit 32. The control unit 33 may input the image data and identify the area of ​​plaque that appears in the image data using a machine learning model that has been trained to output the area of ​​plaque that appears in the image data.

[0090] The illumination control unit 34 is mounted, for example, on the handle portion 10b of the intraoral camera 10 and controls the turning on and off of the first to fourth LEDs 23A to 23D. The illumination control unit 34 is configured, for example, by a circuit. For example, when a user performs an operation on the touch screen 54 of the mobile terminal 50 to start the intraoral camera 10, a corresponding signal is transmitted from the mobile terminal 50 to the signal processing unit 30 via the communication unit 40. The illumination control unit 34 of the signal processing unit 30 turns on the first to fourth LEDs 23A to 23D based on the received signal.

[0091] In addition to the above programs, the memory unit 35 stores images of the rows of teeth (image data) captured by the imaging unit 21. The memory unit 35 is realized by, for example, a semiconductor memory such as a ROM or a RAM, but is not limited to this.

[0092] The communication unit 40 is a wireless communication module that performs wireless communication with the mobile terminal 50. The communication unit 40 is mounted, for example, on the handle portion 10b of the intraoral camera 10, and performs wireless communication with the mobile terminal 50 based on a control signal from the signal processing unit 30. The communication unit 40 performs wireless communication with the mobile terminal 50 that complies with existing communication standards such as Wi-Fi (registered trademark) and Bluetooth (registered trademark). Via the communication unit 40, a dentition image showing teeth is transmitted from the intraoral camera 10 to the mobile terminal 50, and an operation signal is transmitted from the mobile terminal 50 to the intraoral camera 10.

[0093] The mobile device 50 displays the plaque area based on an RGB image of the teeth and plaque that are fluorescently reacting when light including a wavelength range of blue light is irradiated onto the teeth. The mobile device 50 also functions as a user interface for the intraoral camera system.

[0094] FIG. 3 is a block diagram showing the functional configuration of the mobile terminal 50 according to this embodiment.

[0095] 3, the mobile terminal 50 includes an acquisition unit 51, a control unit 52, a display control unit 53, and a touch screen 54. The mobile terminal 50 includes a processor, a memory, and the like. The memory is a ROM, a RAM, or the like, and can store programs executed by the processor. The acquisition unit 51, the control unit 52, and the display control unit 53 are realized by the processor, or the like, which executes programs stored in the memory. The mobile terminal 50 may be realized, for example, by a smartphone or tablet terminal capable of wireless communication.

[0096] The acquisition unit 51 acquires an RGB image from the intraoral camera 10. The acquisition unit 51 is, for example, a wireless communication module that performs wireless communication.

[0097] The control unit 52 executes a process for detecting a plaque region based on the RGB image. The control unit 52 executes a level determination process for determining the incident level of external light incident on the image capture region, and a plaque detection process for detecting a plaque region based on the RGB image. The control unit 52 is an example of a plaque detection unit.

[0098] Here, the level determination process in the control unit 52 will be described with reference to FIGS.

[0099] The control unit 52 determines whether external light is incident based on at least one of the number of pixels of a predetermined color in the RGB image and the number of pixels having a brightness equal to or greater than a predetermined value. In the present embodiment, as a first method, the control unit 52 may utilize the tendency for an image to become reddish overall when external light is incident, and determine whether external light is incident based on a first red pixel count that includes a first number of red pixels having the pixel value at each pixel value equal to or greater than a first pixel value. In the present embodiment, the control unit 52 may further determine whether external light is incident based on a first green pixel count that includes a second number of green pixels having the pixel value at each pixel value equal to or greater than the first pixel value.

[0100] For each pixel value equal to or greater than a first pixel value, the control unit 52 subtracts a second number of pixels having the pixel value from a first number of pixels having the pixel value, accumulates the positive first subtraction value obtained by subtracting the second number of pixels from the first number of pixels, and calculates a first incident level based on a first accumulated value obtained by accumulating the first subtraction values. The control unit 52 may, for example, calculate a value obtained by dividing the first accumulated value by the total number of pixels as the first incident level. Note that calculating the incident level is an example of obtaining the incident level.

[0101] As a second method, the control unit 52 may determine whether external light is incident by evaluating the number of pixels whose green pixel values ​​are smaller than the red pixel values. In this case, the control unit 52 may determine whether external light is incident based on a first red pixel count, which includes a first number of red pixels having pixel values ​​equal to or greater than the first pixel value, and a green pixel count, which includes a second number of green pixels having pixel values ​​(green pixel values) equal to or smaller than a fourth pixel value that is lower than the first pixel value. The control unit 52 may also subtract the second pixel count from the first pixel count, and if the first subtraction value obtained by subtracting the second pixel count from the first pixel count is positive, calculate the first incident level based on the first subtraction value. For example, the control unit 52 may calculate the first incident level by dividing the first subtraction value by the total number of pixels. Note that the first red pixel count is not limited to the above, and may be the number of pixels whose red pixel values ​​are relatively larger than the green pixel values, and the green pixel count may be the number of pixels whose green pixel values ​​are relatively larger than the red pixel values.

[0102] The control unit 52 then changes the display mode of the plaque detection result depending on the incident level of external light. Specifically, the control unit 52 determines whether the first incident level is equal to or less than a first predetermined value and outputs the determination result to the display control unit 53, thereby changing the display mode of the plaque detection result depending on the determination result. The first pixel value and the first predetermined value are set appropriately depending on the lighting environment of the space in which the intraoral camera 10 is used. For example, the first pixel value is set depending on the lighting environment under which the RGB image is captured. For example, if the illuminance is 800 lux and the pixel value is expressed in 8 bits, the first pixel value may be 130. Furthermore, from the viewpoint of preventing a determination that external light is present when there is no external light, i.e., preventing plaque detection from being stopped when there is no external light, the first pixel value may be set to a value higher than 130.

[0103] The control unit 52 also generates a V image consisting of V values ​​(brightness) obtained by HSV conversion of each pixel value of the RGB image, and calculates a third incident level based on a third integrated value of the number of pixels whose V values ​​are equal to or greater than a third pixel value. For example, the control unit 52 may determine the third incident level by dividing the third integrated value by the total number of pixels. The control unit 52 then determines whether the third incident level is equal to or less than a third predetermined value and outputs the determination result to the display control unit 53, thereby changing the display mode of the plaque detection result depending on the determination result. The third pixel value and the third predetermined value are appropriately set depending on the lighting environment of the space in which the intraoral camera 10 is used. For example, if the illuminance is 800 lux and the pixel value is expressed in 8 bits, the third pixel value may be 250.

[0104] FIG. 4 is a first diagram plotting the results of determination of the presence or absence of external light according to this embodiment. The horizontal axis of FIG. 4 indicates the first incident level ("R-G Range 1" in FIG. 4), and the vertical axis indicates the third incident level ("V Range" in FIG. 4). Furthermore, hatched circles in FIG. 4 indicate the first incident level and the third incident level calculated from RGB images captured when external light is present, and unhatched circles indicate the first incident level and the third incident level calculated from RGB images captured when external light is absent.

[0105] Area A in Figure 4 is an area where almost all circles indicating no external light are plotted. From this, it is possible to determine with high accuracy that there is no external light by determining that there is no external light when the first incident level is equal to or less than a threshold value r2 (e.g., approximately 0.18). Similarly, it is possible to determine with high accuracy that there is no external light by determining that there is no external light when the third incident level is equal to or less than a threshold value v2 (e.g., approximately 0.24). In this case, since there is no external light, the plaque detection process is executed.

[0106] Area C in FIG. 4 is an area where all circles indicating the presence of external light are plotted. Therefore, by determining that external light is present when the first incident level is equal to or greater than a threshold value r1 (e.g., 0.45), it is possible to determine with high accuracy that external light is present. Similarly, by determining that external light is present when the third incident level is equal to or greater than a threshold value v1 (e.g., 0.45), it is possible to determine with high accuracy that external light is present. In this case, since there is a high possibility that a plaque region will be falsely detected due to external light, the plaque detection process is not performed. The threshold value r1 is an example of a first predetermined value, and the threshold value r2 is an example of a fourth predetermined value. Furthermore, the threshold value v1 is an example of a third predetermined value, and the threshold value v2 is an example of a fifth predetermined value.

[0107] Area B in Fig. 4 is an area where circles with and without external light are plotted together. In this case, the plaque detection process is executed, but there is a possibility that a plaque region may be erroneously detected due to external light. Therefore, the display mode of the plaque detection result is made different when the area where at least one of the first incident level and the third incident level is plotted is area B from when it is area A. Specifically, when it is area B, the plaque detection result is displayed in a display mode that notifies the user that a plaque region may have been erroneously detected.

[0108] From the results of FIG. 4, for example, if the following determination formula is satisfied, the control unit 52 may determine that there is external light.

[0109] When using the first method, for example, if the value (number of pixels) obtained by subtracting the number of green pixels having a pixel value of 130 from the number of red pixels having a pixel value of 130 is positive, the control unit 52 uses the value itself in the calculation, and if the value is negative, the control unit 52 sets the value to zero. Next, the control unit 52 performs similar processing for each pixel value between 131 and 255. For example, if there are 100 red pixels having a pixel value of 130 and 50 green pixels having a pixel value of 130, the value in the pixel value 130 will be 50. Furthermore, if there are 100 red pixels having a pixel value of 131 and 150 green pixels having a pixel value of 131, the value in the pixel value 130 will be zero.

[0110] Furthermore, when the control unit 52 uses the second method, for example, the value (number of pixels) used in the calculation is the number of pixels having a red pixel value of 130 or more (i.e., the first pixel value is 130) minus the number of pixels having a green pixel value of 129 or less (i.e., the fourth pixel value is 129). For example, if there are 100 pixels having a red pixel value of 130 or more and 50 pixels having a green pixel value of 129 or less, the value is 50. Also, if there are 50 pixels having a red pixel value of 130 or more and 100 pixels having a green pixel value of 129 or less, the value is zero.

[0111] Then, the control unit 52 determines that there is external light when the following formula 1 is satisfied using a first integrated value obtained by integrating the positive value.

[0112] First integrated value / total number of pixels>0.45 (Equation 1)

[0113] In the first method described above, the first integrated value of two pixel values ​​130 and 131 is 50 (50 + 0). In the second method described above, if there are 100 pixels with red pixel values ​​of 130 or more and 50 pixels with green pixel values ​​of 129 or less, the first integrated value is 50 (100 - 50).

[0114] The control unit 52 may determine that external light is present when the first integrated value is higher than a predetermined value. In other words, the presence or absence of external light may be determined without dividing the first integrated value by the total number of pixels. 0.45 is an example of the first predetermined value.

[0115] Furthermore, the control unit 52 determines that there is external light when the following formula 2 is satisfied using a third integrated value obtained by integrating the number of pixels having brightness levels of 250 or more and 255 or less.

[0116] Third integrated value / total number of pixels>0.45 (Equation 2)

[0117] The control unit 52 may determine that external light is present when the third integrated value is higher than a predetermined value. In other words, the presence or absence of external light may be determined without dividing the third integrated value by the total number of pixels. 0.45 is an example of the third predetermined value.

[0118] Furthermore, control unit 52 may determine whether external light is incident based on a second red pixel number including a third number of red pixels having the pixel value at each pixel value equal to or greater than a second pixel value that is higher than the first pixel value. In the present embodiment, control unit 52 further determines whether external light is incident based on a first green pixel number including a second number of green pixels having the pixel value at each pixel value equal to or greater than the first pixel value.

[0119] The control unit 52 subtracts the second pixel number from the third pixel number for each pixel value equal to or greater than the second pixel value, accumulates the resulting positive second subtraction value, and calculates a second incident level based on the second accumulated value. For example, the control unit 52 may determine the second incident level by dividing the second accumulated value by the total number of pixels. The control unit 52 then determines whether the second incident level is equal to or less than a second predetermined value and outputs the determination result to the display control unit 53, thereby varying the display mode of the plaque detection result depending on the determination result. The second pixel value is appropriately set depending on the lighting environment of the space in which the intraoral camera 10 is used. For example, if the illuminance is 800 lux and the pixel value is expressed in 8 bits, the second pixel value may be 250.

[0120] Fig. 5 is a second diagram plotting the determination results of the presence or absence of external light according to this embodiment. The horizontal axis of Fig. 5 represents the second incident level ("R-G Range2" in Fig. 5), and the vertical axis represents the third incident level ("V Range" in Fig. 5). Furthermore, hatched circles in Fig. 5 represent the second incident level and the third incident level calculated from images captured when external light is present, and unhatched circles represent the second incident level and the third incident level calculated from images captured when external light is absent.

[0121] Region E in Figure 5 is a region where all circles indicating the presence of external light are plotted. Therefore, by determining that external light is present when the second incident level is equal to or greater than a threshold value r3 (e.g., 0.2), it is possible to determine with high accuracy that external light is present. In this case, the plaque detection process is not performed because there is a high possibility that the plaque region will be erroneously detected due to external light. The threshold value r3 is an example of a first predetermined value.

[0122] 5 is a region where circles with and without external light are plotted together. In this case, the plaque detection process is executed, but there is a possibility that a plaque region may be erroneously detected due to external light. Therefore, when the region where the second incident level and the third incident level are plotted is region D, the plaque detection result is displayed in a display manner that notifies the user that a plaque region may have been erroneously detected.

[0123] From the results of FIG. 5, for example, if the following determination formula is satisfied, the control unit 52 may determine that there is external light.

[0124] For example, if the value (number of pixels) obtained by subtracting the number of green pixels having a pixel value of 250 from the number of red pixels having a pixel value of 250 is positive, the control unit 52 uses the value itself in the calculation, and if the value is negative, the control unit 52 sets the value to zero. Next, the control unit 52 performs the same process for each pixel value between 251 and 255. Then, the control unit 52 determines that external light is present if the following formula 3 is satisfied using a second integrated value obtained by integrating the positive values.

[0125] Second integrated value / total number of pixels>0.2 (Equation 3)

[0126] The control unit 52 may determine that external light is present when the second integrated value is higher than a predetermined value. In other words, the presence or absence of external light may be determined without dividing the second integrated value by the total number of pixels. 0.2 is an example of the second predetermined value.

[0127] The control unit 52 may perform a determination using, for example, (Equation 1) to (Equation 3), and determine that external light is present when at least one of the equations is satisfied. For example, if external light is locally incident, a determination using Equation 1 may result in a determination that external light is absent. However, by performing a determination using Equation 2, it is possible to increase the likelihood of determining that external light is present even when external light is locally incident. In other words, it is possible to more accurately determine the presence or absence of external light.

[0128] As described above, the pixel values ​​used to count the number of pixels may be changed depending on the lighting environment of the space. The pixel values ​​(the first to third pixel values) may be input by the user, for example.

[0129] 6 is a diagram showing a setting screen S1 for determining external light according to the present embodiment. The setting screen S1 is displayed on a touch screen.

[0130] 6, the range of the number of pixels to be counted to determine the presence or absence of external light may be set by the user. Furthermore, the threshold value for determining the presence or absence of external light ("0.45" in Equations 1 and 2, and "0.2" in Equation 3) may also be set by the user.

[0131] The mobile terminal 50 stores a table in which the lighting environment to be used is associated with the range and threshold of the number of pixels that can be set in Figure 6, and the range and threshold of the number of pixels may be automatically changed based on the table and the illuminance measured by the sensor unit 22.

[0132] Furthermore, in the plaque detection process, the control unit 52 distinguishes between teeth or plaque and lips or gums based on the color in the RGB image. It is known that when light including a blue wavelength range is irradiated onto teeth, excited fluorescence is emitted from dentin, which then transmits through the enamel layer and emits green light. Lips and gums, on the other hand, have a bluish tint, allowing the control unit 52 to easily distinguish between teeth or plaque and lips or gums. Furthermore, using an RGB image that has undergone image processing such as white balance adjustment makes it easier to distinguish plaque regions. Image processing essentially decolorizes tooth regions, making it easier to distinguish plaque regions, which are areas of the teeth where plaque has adhered. Therefore, the control unit 52 can easily identify plaque regions in the tooth image. For example, by capturing an image of teeth after brushing and identifying plaque regions, it is possible to present unbrushed areas to the user.

[0133] 2 again, the display control unit 53 performs control to display the detected plaque region on the touch screen 54. The display control unit 53 changes the display mode of the plaque detection result depending on whether the incident level of external light incident on the shooting region of the RGB image is equal to or less than a predetermined value.

[0134] The touch screen 54 functions as an input device and an output device and is configured to be able to display, for example, a dentition image. The touch screen 54 may have, for example, a display unit such as a liquid crystal display panel and a touch panel disposed on top of the display unit.

[0135] An example of an image displayed on the touch screen 54 will now be described with reference to FIG. 7 . FIG. 7 is a diagram showing a captured image and a display example of a plaque detection result with and without external light according to the present embodiment. FIG. 7 shows a captured image and a display example of a plaque detection result with and without external light. The captured image is an RGB image obtained by capturing an image of the teeth and gums in the oral cavity with the intraoral camera 10. The display of the plaque detection result is an image in which the plaque region detected after the plaque detection process is performed by the control unit 52 is highlighted, and is an image displayed on the touch screen 54.

[0136] When there is no external light, the plaque region in the captured image and the region determined to be plaque in the display of the plaque detection result are substantially the same region.

[0137] On the other hand, when external light is present, although there is no plaque area in the captured image, an area determined to be plaque is displayed in the display of the plaque detection result. This area is a false positive area, an area that has been erroneously detected due to the influence of external light. In this case, it is advisable to notify the user that it is a false positive (see FIG. 9 etc., which will be described later).

[0138] In this way, by using the intraoral camera system, the user can take an image of the inside of their own oral cavity with the intraoral camera 10 and check the condition of the inside of their oral cavity displayed on the mobile terminal 50. This allows the user to easily check the health condition of their own teeth.

[0139] 2. Operation of the Intraoral Camera System Next, the operation of the intraoral camera system configured as described above will be described with reference to Figs. 8 to 10. Fig. 8 is a flowchart showing the operation (image processing method) of the mobile terminal 50 according to this embodiment. Note that the processing shown in Fig. 8 is, for example, processing performed in real time, and is performed each time one frame or multiple frames of image data are obtained.

[0140] The intraoral camera 10 transmits image data (RGB images) generated by a user photographing the teeth and gums in their oral cavity using the intraoral camera 10 to the mobile terminal 50. Here, the image data may be a video or one or more still images. Furthermore, when the image data is a video or a plurality of still images, sensor data (external light detection results) from the sensor unit 22 may be transmitted for each frame of the video or for each still image. Furthermore, when the image data is a video, the sensor data may be transmitted for each plurality of frames.

[0141] Furthermore, the image data may be transmitted in real time, or may be transmitted all at once after a series of photographs (for example, photographs of all teeth in the oral cavity) have been taken.

[0142] 8, the acquisition unit 51 acquires (S11) an RGB image transmitted from the intraoral camera 10. The RGB image is an image obtained by the intraoral camera 10 photographing the teeth and dental plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light.

[0143] Next, the control unit 52 acquires the incident level of external light based on the acquired RGB image (S12). The control unit 52 may calculate the incident level by, for example, dividing at least one of the first to third integrated values ​​based on the RGB image by the total number of pixels. Note that the incident level includes at least one of the first to third incident levels.

[0144] Next, the control unit 52 performs plaque detection processing based on the acquired RGB image (S13).

[0145] Next, the control unit 52 determines whether the incident level acquired in step S12 is equal to or lower than a predetermined value (S14). Step S14 corresponds to determining whether external light is incident into the oral cavity.

[0146] Next, when the control unit 52 determines that the incident level is not equal to or less than the predetermined value (No in S14), it outputs the determination result of step S14 to the display control unit 53, thereby causing an image including information indicating that external light is incident and the plaque detection result, or an image including no plaque detection result (for example, an RGB image acquired from the intraoral camera 10) to be displayed on the touch screen 54 (S15). When the control unit 52 determines that the incident level is equal to or less than the predetermined value (Yes in S14), it outputs the determination result of step S14 to the display control unit 53, thereby causing an image including the plaque detection result to be displayed (normal display) (S16).

[0147] 9A and 9B are diagrams showing examples of displaying whether or not external light is present according to the present embodiment. Fig. 9 shows a row-of-teeth image. Note that, for convenience, the display of plaque detection results (e.g., highlight display) is omitted in Fig. 9A and 9B.

[0148] FIG. 9(a) shows an image displayed in step S15, in which first information (a circle with diagonal hatching in FIG. 9(a)) indicating that external light is incident (or that external light may be incident) is superimposed on the image including the plaque detection result. The first information is displayed, for example, superimposed on an area different from the tooth area and the plaque area (e.g., a highlighted area). Note that if the incident level is not equal to or lower than a predetermined value, the control unit 52 may prohibit the display of the plaque detection result or the execution of the plaque detection process.

[0149] (b) of Figure 9 is the image displayed in step S16, in which second information indicating that no external light is incident (a circle without diagonal hatching in (b) of Figure 9) is superimposed on the image including the plaque detection results. The second information is different from the first information. Note that the second information does not have to be displayed. In other words, in step S16, information indicating that no light is incident and an image including only the plaque detection results from the plaque detection results may be displayed.

[0150] In this way, the control unit 52 displays the first information when there is external light, and displays the second information or only the plaque detection result when there is no external light. In this way, the control unit 52 changes the display mode of the plaque detection result depending on whether there is external light or not.

[0151] The control unit 52 may further determine whether at least one of the following conditions is satisfied: the first incident level is equal to or less than a first predetermined value (e.g., threshold value r1 shown in FIG. 4 ) and equal to or greater than a fourth predetermined value (e.g., threshold value r2 shown in FIG. 4 ) that is lower than the first predetermined value; and the third incident level is equal to or less than a third predetermined value (e.g., threshold value v1 shown in FIG. 4 ) and equal to or greater than a fifth predetermined value (e.g., threshold value v2 shown in FIG. 4 ) that is lower than the third predetermined value. If it is determined that the conditions are satisfied, the control unit 52 may output the determination result to the display control unit 53, thereby superimposing information urging the user to block light on the image displaying the plaque detection result. Note that each predetermined value may be set according to the lighting environment under which the RGB image is captured.

[0152] FIG. 10 is a diagram showing an example of a display prompting the user to block light according to the present embodiment.

[0153] As shown in Fig. 10, the control unit 52 may issue a notification urging the user to block light to prevent external light from entering the oral cavity. Furthermore, instead of displaying a message urging the user to block light, the control unit 52 may also display a message urging the user to dim or turn off the lights. The message urging the user to block light may include, for example, a message urging the user to close curtains. The message urging the user to block light is displayed, for example, superimposed on an area different from the tooth area and the plaque area (for example, a highlighted area).

[0154] (First Modification of the Embodiment) An intraoral camera system according to this modification will be described below with reference to Fig. 11. The following description will focus on differences from the embodiment, and descriptions of the same or similar content as the embodiment will be omitted or simplified.

[0155] FIG. 11 is a diagram showing a schematic diagram of a captured image divided into a plurality of regions according to this modification.

[0156] As shown in Fig. 11, the control unit 52 may divide the RGB image into multiple regions (four regions R1 to R4 in the example of Fig. 11), acquire the incident level of external light incident on each of the multiple divided regions, and vary the display mode of the plaque detection result depending on the determination result of whether the incident level of each of the multiple regions is equal to or less than a predetermined value. For example, the control unit 52 may determine that external light is present when the incident level of at least one of the multiple regions is not equal to or less than a predetermined value. The control unit 52 may perform determination using Equation 1 and Equation 3 shown in the embodiment for each of the multiple regions, and determine that external light is present when at least one region satisfies at least one of Equation 1 and Equation 3.

[0157] The first pixel value is set to, for example, a value larger than the first pixel value in the embodiment. For example, the smaller the area on the image for which the incidence level is determined, the larger the first pixel value may be set. Similarly to the first pixel value, the second pixel value is set to, for example, a value larger than the second pixel value in the embodiment. In this modification, the first pixel value is, for example, 0.8, and the second pixel value is, for example, 0.5, but is not limited thereto. The intraoral camera 10 may store a table that associates the sizes of multiple areas with the first pixel values ​​and second pixel values, and automatically set the first pixel values ​​and second pixel values ​​based on the table. The sizes of the multiple areas may also be set by, for example, a user.

[0158] The number of divisions is not particularly limited as long as it is at least 2. The sizes, shapes, etc. of the regions may be the same or different.

[0159] (Second Modification of the Embodiment) An intraoral camera system according to this modification will be described below with reference to Figures 14 to 24. Note that the following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified.

[0160] 14 is a flowchart showing the operation (image processing method) of the mobile terminal 50 according to this modification. The intraoral camera system according to this modification differs from the intraoral camera system according to the embodiment in that it divides an RGB image into multiple regions and presents information encouraging the user to take measures against external light based on the incident level of external light for each of the multiple regions (e.g., the presence or absence of external light). The following description will be made using the reference numerals of the intraoral camera system according to the embodiment.

[0161] 14, the acquisition unit 51 acquires (S21) an RGB image transmitted from the intraoral camera 10. The RGB image is an image obtained by the intraoral camera 10 photographing the teeth and dental plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light.

[0162] Next, the control unit 52 divides the RGB image into a plurality of regions (S22). The control unit 52 may divide the RGB image into two or more regions, and may divide the RGB image into, for example, four regions as shown in Fig. 11, nine regions, or any other number of regions. In this modified example, the cases of division into four regions and nine regions will be mainly described.

[0163] In the following, an example will be described in which the multiple regions are quadrangular regions; however, the shape is not limited to a quadrangular shape and may be any other shape, such as a circle, a triangle, or a polygon with pentagons or more sides. Furthermore, a single RGB image may contain multiple shaped regions. Furthermore, the areas of the multiple regions may be equal, or the area of ​​at least one region may be different from the areas of the other regions. Furthermore, the quadrangular shapes may be square or rectangular.

[0164] Next, the control unit 52 acquires, for each of the plurality of regions, the incident level of external light incident on that region (S23). For example, the control unit 52 calculates at least one integrated value of the first to third integrated values ​​for each of the plurality of regions, and divides the calculated at least one integrated value by the total number of pixels in that region to calculate the incident level. One incident level is calculated for each region. Note that the incident levels for each of the plurality of regions do not necessarily have to be acquired by calculation, and may, for example, be acquired from an external device via the acquisition unit 51.

[0165] Next, the control unit 52 determines the presence or absence of external light for each of the multiple regions (S24). The control unit 52 determines the presence or absence of external light for each of the multiple regions based on the incident level of the region. The control unit 52 may determine the presence or absence of external light for each of the multiple regions using, for example, the above (Equation 1) or (Equation 2). In this case, the first integrated value shown in (Equation 1) and the second integrated value shown in (Equation 2) are values ​​in the region, and the total number of pixels is the total number of pixels in the region.

[0166] Next, the control unit 52 determines the incident direction of external light relative to the RGB image (S25). That is, the control unit 52 determines the incident direction of external light incident on the intraoral imaging region. The control unit 52 may determine the incident direction of external light based on, for example, the positional relationship (e.g., position in the RGB image) of regions among multiple regions where the incident level exceeds a predetermined value (i.e., regions where it is determined that external light is present).

[0167] The control unit 52 may determine whether external light is incident on the RGB image based on the determination result of step S24, and if it determines that external light is incident on the RGB image, may execute the processes of step S25 and thereafter, and if it determines that external light is not incident on the RGB image, may end the processes. The control unit 52 may determine that external light is incident on the RGB image if it determines in step S24 that external light is present in at least one region among the multiple regions.

[0168] Here, an example of the relationship between the position of an area in an RGB image determined to have external light and the incident direction of the external light will be described with reference to FIGS. 15A to 21H. First, a case where there are four areas (an RGB image divided into four) will be described with reference to FIGS. 15A to 16D. By dividing the RGB image into four, it is possible to determine the incident direction of external light while suppressing an increase in the processing load of the mobile terminal 50. FIGS. 15A to 15H are diagrams showing an example of the determination results of the incident level of each of the four divided areas when external light is incident from each direction. FIGS. 16A to 16D are diagrams showing other example of the determination results of the incident level of each of the four divided areas when external light is incident from each direction. Furthermore, areas determined to have external light are indicated by diagonal hatching, and areas determined to have no external light are indicated by a rectangular frame (without hatching).

[0169] 15A to 15D show the incident direction of external light when it is determined that external light is present in one of the four divided regions, FIGS. 15E to 15H show the incident direction of external light when it is determined that external light is present in two of the four divided regions, and FIGS. 16A to 16D show the incident direction of external light when it is determined that external light is present in three of the four divided regions. The four divided regions are referred to as regions R11 to R14. Region R11 is the upper left region of the RGB image, region R12 is the upper right region of the RGB image, region R13 is the lower left region of the RGB image, and region R14 is the lower right region of the RGB image.

[0170] 15A and 15B, when it is determined that external light is present only in region R11, the control unit 52 determines that external light is incident from the upper left (upper left side) of the RGB image. When it is determined that external light is present only in region R12, the control unit 52 determines that external light is incident from the upper right (upper right side) of the RGB image. Also, as shown in FIGS. 15C and 15D, when it is determined that external light is present only in region R14, the control unit 52 determines that external light is incident from the lower right (lower right side) of the RGB image. When it is determined that external light is present only in region R13, the control unit 52 determines that external light is incident from the lower left (lower left side) of the RGB image. In this way, when it is determined that external light is present in only one of the four divided regions, the control unit 52 may determine that external light is incident from, for example, a diagonal direction.

[0171] 15E and 15F, when it is determined that external light is present only in regions R11 and R12, the control unit 52 determines that external light is incident from above (the upper side) in the RGB image. When it is determined that external light is present only in regions R12 and R14, the control unit 52 determines that external light is incident from the right (the right side) in the RGB image. When it is determined that external light is present only in regions R13 and R14, the control unit 52 determines that external light is incident from below (the lower side) in the RGB image. When it is determined that external light is present only in regions R11 and R13, the control unit 52 determines that external light is incident from the left (the left side) in the RGB image. In this way, when it is determined that external light is present in only two of the four regions, the control unit 52 may determine that external light is incident from either the up, down, left, or right direction. The two regions determined to have external light may be, for example, diagonally adjacent regions in the RGB image. For example, it may be determined that only the regions R11 and R14 or only the regions R12 and R13 have external light. In this case, for example, the control unit 52 may determine that external light is incident from two directions.

[0172] 16A and 16B, when it is determined that external light is present only in regions R11 to R13, the control unit 52 determines that external light is incident from the upper left of the RGB image. When it is determined that external light is present only in regions R11, R12, and R14, the control unit 52 determines that external light is incident from the upper right of the RGB image. When it is determined that external light is present only in regions R12 to R14, the control unit 52 determines that external light is incident from the lower right of the RGB image. When it is determined that external light is present only in regions R11, R13, and R14, the control unit 52 determines that external light is incident from the lower left of the RGB image. In this way, when it is determined that external light is present only in three of the four divided regions, the control unit 52 may determine that external light is incident from, for example, a diagonal direction.

[0173] Next, a case where there are nine regions (an RGB image divided into nine) will be described with reference to FIGS. 17A to 21H. Dividing an RGB image into nine regions allows, for example, more precise determination of the incident direction of external light. FIGS. 17A to 21H are diagrams showing examples of the determination results of the incident level for each of the nine divided regions when external light is incident from various directions. FIGS. 17A to 17H show the incident direction of external light when it is determined that external light is present in one of the nine divided regions. FIGS. 18A to 18H show the incident direction of external light when it is determined that external light is present in two of the nine divided regions. FIGS. 19A to 19H show the incident direction of external light when it is determined that external light is present in three of the nine divided regions. Furthermore, FIGS. 20A to 20H show the incident direction of external light when it is determined that external light is present in four of the nine divided regions. FIGS. 21A to 21H show the incident direction of external light when it is determined that external light is present in five of the nine divided regions.

[0174] The nine divided regions are referred to as regions R21 to R29. Region R21 is the upper left region of the RGB image, region R22 is the upper center region of the RGB image, region R23 is the upper right region of the RGB image, region R24 is the center left region of the RGB image, and region R25 is the center region of the RGB image. Region R26 is the center right region of the RGB image, region R27 is the lower left region of the RGB image, region R28 is the lower center region of the RGB image, and region R29 is the lower right region of the RGB image.

[0175] 17A and 17B, when it is determined that external light is present only in region R21, the control unit 52 determines that external light is incident from the upper left of the RGB image, and when it is determined that external light is present only in region R23, the control unit 52 determines that external light is incident from the upper right of the RGB image. Also, as shown in Figures 17C and 17D, when it is determined that external light is present only in region R27, the control unit 52 determines that external light is incident from the lower left of the RGB image, and when it is determined that external light is present only in region R29, the control unit 52 determines that external light is incident from the lower right of the RGB image.

[0176] 17E and 17F, when it is determined that external light is present only in region R24, the control unit 52 determines that external light is incident from the left in the RGB image, and when it is determined that external light is present only in region R22, the control unit 52 determines that external light is incident from above in the RGB image. Also, as shown in Figures 17G and 17H, when it is determined that external light is present only in region R28, the control unit 52 determines that external light is incident from below in the RGB image, and when it is determined that external light is present only in region R26, the control unit 52 determines that external light is incident from the right in the RGB image.

[0177] Figures 18A to 18D show cases where it is determined that there is external light in two vertically consecutive areas, and Figures 18E to 18H show cases where it is determined that there is external light in two horizontally consecutive areas.

[0178] 18A and 18B, when it is determined that external light is present only in regions R21 and R24, the control unit 52 determines that external light is incident from the upper left of the RGB image, and when it is determined that external light is present only in regions R23 and R26, the control unit 52 determines that external light is incident from the upper right of the RGB image. Also, as shown in Figures 18C and 18D, when it is determined that external light is present only in regions R24 and R27, the control unit 52 determines that external light is incident from the lower left of the RGB image, and when it is determined that external light is present only in regions R26 and R29, the control unit 52 determines that external light is incident from the lower right of the RGB image.

[0179] 18E and 18F, when it is determined that external light is present only in regions R21 and R22, the control unit 52 determines that external light is incident from the upper left of the RGB image, and when it is determined that external light is present only in regions R22 and R23, the control unit 52 determines that external light is incident from the upper right of the RGB image. Also, as shown in Figures 18G and 18H, when it is determined that external light is present only in regions R27 and R28, the control unit 52 determines that external light is incident from the lower left of the RGB image, and when it is determined that external light is present only in regions R28 and R29, the control unit 52 determines that external light is incident from the lower right of the RGB image.

[0180] The two consecutive regions may be a region including the region R25, or may be regions that are consecutive in a diagonal direction (for example, regions R23 and R25).

[0181] Figures 19A to 19D show cases where it is determined that there is external light in three areas arranged in an L shape, and Figures 19E to 19H show cases where it is determined that there is external light in three areas arranged in a straight line.

[0182] 19A and 19B , when it is determined that external light is present only in regions R21, R22, and R24, the control unit 52 determines that external light is incident from the upper left of the RGB image. When it is determined that external light is present only in regions R22, R23, and R26, the control unit 52 determines that external light is incident from the upper right of the RGB image. Also, as shown in FIGS. 19C and 19D , when it is determined that external light is present only in regions R24, R27, and R28, the control unit 52 determines that external light is incident from the lower left of the RGB image. When it is determined that external light is present only in regions R26, R28, and R29, the control unit 52 determines that external light is incident from the lower right of the RGB image. In this way, when it is determined that external light is present only in three of the nine divided regions arranged in an L-shape, the control unit 52 may determine that external light is incident from one of the diagonal directions.

[0183] As shown in Figures 19E and 19F, if the control unit 52 determines that external light is present only in regions R21, R24, and R27, it determines that external light is entering from the left of the RGB image. If the control unit 52 determines that external light is present only in regions R21 to R23, it determines that external light is entering from above the RGB image. Also, as shown in Figures 19G and 19H, if the control unit 52 determines that external light is present only in regions R27 to R29, it determines that external light is entering from below the RGB image. If the control unit 52 determines that external light is present only in regions R23, R26, and R29, it determines that external light is entering from the right of the RGB image. In this way, if the control unit 52 determines that external light is present only in three linear regions out of the nine divided regions, it determines that external light is entering from either the top, bottom, left, or right direction.

[0184] Figures 20A to 20D show cases where it is determined that there is external light in four areas arranged in a square shape, and Figures 20E to 20H show cases where it is determined that there is external light in four areas arranged in an L shape.

[0185] 20A and 20B , when it is determined that external light is present only in regions R21, R22, R24, and R25, the control unit 52 determines that external light is incident from the upper left of the RGB image. When it is determined that external light is present only in regions R22, R23, R25, and R26, the control unit 52 determines that external light is incident from the upper right of the RGB image. Also, as shown in FIGS. 20C and 20D , when it is determined that external light is present only in regions R24, R25, R27, and R28, the control unit 52 determines that external light is incident from the lower left of the RGB image. When it is determined that external light is present only in regions R25, R26, R28, and R29, the control unit 52 determines that external light is incident from the lower right of the RGB image. In this way, when it is determined that external light is present only in four of the nine divided regions arranged in a rectangular shape, the control unit 52 may determine that external light is incident from, for example, a diagonal direction.

[0186] 20E and 20F, when it is determined that external light is present only in regions R21 to R24, the control unit 52 determines that external light is incident from the upper left of the RGB image. When it is determined that external light is present only in regions R21 to R23 and R26, the control unit 52 determines that external light is incident from the upper right of the RGB image. Also, as shown in FIGS. 20G and 20H, when it is determined that external light is present only in regions R24 and R27 to R29, the control unit 52 determines that external light is incident from the lower left of the RGB image. When it is determined that external light is present only in regions R26 to R29, the control unit 52 determines that external light is incident from the lower right of the RGB image. In this way, when it is determined that external light is present only in four L-shaped regions out of the nine divided regions, the control unit 52 may determine that external light is incident from, for example, a diagonal direction.

[0187] Figures 21A to 21D show cases where it is determined that there is external light in five areas lined up including the central area R25, and Figures 21E to 21H show cases where it is determined that there is external light in five areas lined up but not including the central area R25 (for example, a U-shaped area).

[0188] As shown in Figures 21A and 21B, if the control unit 52 determines that external light is present only in regions R21 to R25, it determines that external light is incident from the upper left of the RGB image. If the control unit 52 determines that external light is present only in regions R21 to R23, R25, and R26, it determines that external light is incident from the upper right of the RGB image. Also, as shown in Figures 21C and 21D, if the control unit 52 determines that external light is present only in regions R21, R24, R25, R27, and R28, it determines that external light is incident from the lower left of the RGB image. If the control unit 52 determines that external light is present only in regions R23, R25, R26, R28, and R29, it determines that external light is incident from the lower right of the RGB image. In this way, if the control unit 52 determines that external light is present only in five of the nine divided regions, including the central region R25, it may determine that external light is incident from, for example, a diagonal direction.

[0189] 21E and 21F, when it is determined that external light is present only in regions R21 to R24 and R26, the control unit 52 determines that external light is incident from above in the RGB image, and when it is determined that external light is present only in regions R24 and R26 to R29, the control unit 52 determines that external light is incident from below in the RGB image. Also, as shown in Figures 21G and 21H, when it is determined that external light is present only in regions R21, R22, R24, R27 and R28, the control unit 52 determines that external light is incident from the left in the RGB image, and when it is determined that external light is present only in regions R22, R23, R26, R28 and R29, the control unit 52 determines that external light is incident from the right in the RGB image. In this way, if the control unit 52 determines that external light is present only in five U-shaped areas out of the nine divided areas, or in five areas obtained by rotating the U-shape by 90 degrees, 180 degrees, or 270 degrees, it may determine that external light is entering from any direction, for example, up, down, left, or right.

[0190] 15A to 21H , the positions of the regions determined to have external light present are merely examples and are not limited to those described above. Furthermore, the correspondence between the regions determined to have external light present and the incident direction of external light in those regions is also merely examples and is not limited to those described above. For example, two or more incident directions of external light may be determined based on at least one of the number and positions of the regions determined to have external light present.

[0191] 15A to 21H, it is possible to identify directions in which external light is not incident. Such directions in which external light is not incident may be used, for example, to determine countermeasure information, which will be described later.

[0192] 14 , next, the portable device 50 outputs countermeasure information according to the incident direction (S26). For example, the control unit 52 may determine countermeasure information including countermeasures according to the incident direction and transmit the determined countermeasure information to an external device. Furthermore, the display control unit 53 may display the countermeasure information determined by the control unit 52 on the touch screen 54.

[0193] The determined countermeasure information will now be described with reference to FIGS. 22 to 24. FIGS. 22 to 24 are diagrams showing examples of each countermeasure information according to this modification. The second countermeasure information shown in FIG. 23 includes countermeasures that impose a greater burden on the user than the first countermeasure information shown in FIG. 22, and the third countermeasure information shown in FIG. 24 includes countermeasures that impose a greater burden on the user than the second countermeasure information shown in FIG. 23, but is not limited to these. Note that a greater burden on the user is exemplified by an increase in the number of body parts that the user moves, or an increase in the range of movement of the user (for example, a change from a countermeasure that does not require the user to move to a countermeasure that requires the user to move).

[0194] Note that Figure 22 shows measures to be taken when external light is incident from above (upper direction), Figure 23 shows measures to be taken when external light is incident from the upper right direction, and Figure 24 shows measures to be taken when there are many areas where external light is determined to be present.

[0195] FIG. 22 shows an example of the display of first countermeasure information, which includes measures such as wearing a light-shielding cover or tilting the head downward. Tilt-down corresponds to tilting the face in the opposite direction to the incident direction of external light. In this way, a countermeasure of tilting the face in an appropriate direction according to the incident direction of external light may be suggested. Information indicating that the care state cannot be correctly evaluated is also displayed. Displaying this information is an example of informing the user that a plaque area may have been erroneously detected due to external light entering the oral cavity. This can prompt the user to implement the first countermeasure information. Tilt-down is an example of information regarding the direction of incident external light.

[0196] 23 shows an example of a display of second countermeasure information, which includes covering the mouth from the upper right with the hand that is not holding the toothbrush while the head is facing downward. The countermeasure shown in FIG. 23 includes covering the direction of incoming external light with the hand in addition to the countermeasure of facing the head downward shown in FIG. 22. In this way, a countermeasure of covering the direction of incoming external light with the hand may be suggested.

[0197] The control unit 52 may determine to output the first countermeasure information or the second countermeasure information, for example, when it is determined that external light is present in one or two of the four regions into which the RGB image is divided (see FIGS. 15A to 15H), or when it is determined that external light is present in one to four of the nine regions into which the RGB image is divided (see FIGS. 17A to 20H). The control unit 52 may determine to output the first countermeasure information, for example, when it is determined that external light is present in one of the four regions into which the RGB image is divided (see FIGS. 15A to 15D), or when it is determined that external light is present in one or two of the nine regions into which the RGB image is divided (see FIGS. 17A to 18H). In addition, the control unit 52 may decide to output the second countermeasure information, for example, when it is determined that there is external light in two of the four regions into which the RGB image is divided (see Figures 15E to 15H), or when it is determined that there is external light in three or four of the nine regions into which the RGB image is divided (see Figures 19A to 20H).

[0198] Furthermore, for example, when it is determined that external light is present in one or two of the four regions into which an RGB image is divided, or when it is determined that external light is present in one to four of the nine regions into which an RGB image is divided, if the determination that external light is present is the first time, the control unit 52 may determine the first countermeasure information, and when it is the second or subsequent time (i.e., when countermeasure information was output the first time and the countermeasure was implemented, but it was still determined that external light was present), the control unit 52 may determine the second countermeasure information. In this way, the control unit 52 may determine the countermeasure information to be output depending on whether it is determined that external light is present after outputting the countermeasure information.

[0199] 24 shows a display example of third countermeasure information including turning off the lights in the room or moving the location where the intraoral camera is used as a countermeasure. The first and second countermeasure information are countermeasures that can be performed without the user moving, but the third countermeasure information includes a countermeasure that requires the user to move or control a device external to the intraoral camera system, and therefore imposes a greater burden on the user than the first and second countermeasure information.

[0200] The control unit 52 may decide on the third countermeasure information, for example, when it is determined that there is external light in three or more of the four regions into which the RGB image is divided (see Figures 16A to 16D), or when it is determined that there is external light in five or more of the nine regions into which the RGB image is divided (see Figures 21A to 21H).

[0201] Furthermore, the control unit 52 may determine the third countermeasure information, for example, when it is determined that external light is present in one or two of the four regions into which the RGB image is divided, or when it is determined that external light is present in one to four of the nine regions into which the RGB image is divided, and the determination that external light is present is the second or subsequent time. The control unit 52 may determine the third countermeasure information, for example, when it has output the second countermeasure information and it is still determined that external light is present despite the implementation of that countermeasure. In this way, the control unit 52 may determine the countermeasure information depending on whether it is determined that external light is present after outputting the countermeasure information.

[0202] Furthermore, the control unit 52 may determine the countermeasure information based on the position of an area where the incident level exceeds a predetermined value (i.e., an area determined to have external light). For example, the control unit 52 may determine the first countermeasure information or the second countermeasure information when external light is incident in only one direction, and may determine the third countermeasure information when external light is incident in two or more directions.

[0203] The mobile terminal 50 may determine and display countermeasure information according to at least one of the number of areas among the plurality of areas where the incident level exceeds a predetermined value and the positions (positions on the RGB image) of the areas among the plurality of areas where the incident level exceeds a predetermined value.

[0204] At least one of the first to third countermeasure information may include information regarding the direction in which external light is incident. The information regarding the direction in which external light is incident may be, for example, information indicating the direction from which external light is incident, or information indicating a countermeasure based on the direction in which external light is incident.

[0205] 14 again, the control unit 52 determines whether the user has implemented the countermeasure indicated in the output countermeasure information (S27). In this modification, the control unit 52 functions as a determination unit that determines whether the user has responded to the output countermeasure information. The control unit 52 may determine that the user has implemented the countermeasure information, for example, when the touch screen 54 receives a user operation indicating that the countermeasure included in the countermeasure information has been implemented, when the control unit 52 acquires, via the acquisition unit 51, information based on an image of the user captured by an external imaging device and indicating that the countermeasure indicated in the countermeasure information has been implemented, or when a predetermined time has elapsed since the countermeasure information was output.

[0206] Next, if the portable device 50 determines that the user has performed the action (Yes in S27), it determines whether or not external light has been blocked (S28). Specifically, the portable device 50 executes the processes of steps S21 to S24 again. For example, after it is determined that the user has responded to the action information, the portable device 50 may acquire the incident light levels after the action has been taken for each of a plurality of regions in the second RGB image obtained by irradiating the teeth with light including a wavelength range of blue light and photographing the teeth and dental plaque that are reacting to fluorescence.

[0207] If the control unit 52 determines in the re-executed step S24 that there is no external light in each of the multiple areas, it determines Yes in step S28, and if it determines that there is external light in at least one of the multiple areas, it determines No in step S28.

[0208] If the external light is not blocked (No in S28), the control unit 52 outputs (S29) another countermeasure information different from the countermeasure information output in step S26. The other countermeasure information is, for example, information according to at least one of the number of regions among the plurality of regions whose incident level after the countermeasure exceeds a predetermined value and the position of the region among the plurality of regions.

[0209] If the control unit 52 has output the first countermeasure information in step S26, it may output the second countermeasure information in step S29, and if the control unit 52 has output the second countermeasure information in step S26, it may output the third countermeasure information in step S29. The control unit 52 may determine and display the second countermeasure information, which is different from the first countermeasure information, based on the incidence level after the countermeasure is implemented for each of the multiple regions, for example.

[0210] Furthermore, the control unit 52 may determine another countermeasure information based on a difference between at least one of the number of regions determined to have external light in step S24 and the positions of the regions (i.e., the incident direction of external light) and at least one of the number of regions determined to have external light in step S28 and the positions of the regions (i.e., the incident direction of external light). For example, if the first countermeasure information is output in step S26 and the number of regions determined to have external light in step S28 is equal to or greater than the number of regions determined to have external light in step S24, or if the incident direction of external light is the same in step S28 and step S24, the control unit 52 may output third countermeasure information as another countermeasure information, and if at least one of the cases is satisfied, the control unit 52 may output second countermeasure information as another countermeasure information.

[0211] The method for determining the other countermeasure information is not limited to the above, and other methods may be used. For example, the other countermeasure information may be determined based on a table in which the output countermeasure information and at least one of the number and positions of areas determined to have external light before and after the output of the countermeasure information are associated with the other countermeasure information.

[0212] After step S29 is executed, or if the determination in step S27 is No, the process returns to step S27 and continues.

[0213] Furthermore, if the external light is blocked (Yes in S28), the control unit 52 executes the processes from step S30 onwards.

[0214] The process of step S30 is similar to the process shown in step S13 of FIG. 8, and the process shown in step S31 is similar to the process shown in step S16 of FIG. 8, so a description thereof will be omitted.

[0215] (Other Embodiments) As described above, the intraoral camera system etc. according to one or more aspects have been described based on the embodiments etc., but the present disclosure is not limited to these embodiments etc. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art can conceive of to the present embodiment and embodiments constructed by combining components of different embodiments may also be included in the present disclosure.

[0216] For example, in the above embodiment, the mobile terminal 50 is exemplified as the user's information terminal, but the information terminal may be a stationary information terminal.

[0217] Furthermore, the intraoral camera 10 in the above-described embodiments may be mounted on, for example, an oral irrigator.

[0218] In addition, in the above-described embodiment, an example in which a display is used to notify of a false detection has been described, but the notification method is not limited to a display, and may be, for example, by emitting a sound or emitting light (e.g., a warning light). In this case, the mobile terminal 50 may be equipped with a sound output device such as a speaker or a light emitting device such as an LED.

[0219] In the above embodiment, an example has been described in which image processing such as AWB is performed by the image processing unit 32 of the intraoral camera 10, but it may also be performed by, for example, the control unit 52 of the mobile terminal 50. Furthermore, part or all of the processing performed by the control unit 52 of the mobile terminal 50 may also be performed by the control unit 33 of the intraoral camera 10.

[0220] In addition, in the second modification of the above embodiment, an example in which countermeasure information is determined and displayed based on the determination result of the presence or absence of external light in each of a plurality of regions has been described. However, this is not limited to this. Countermeasure information for suppressing the effects of external light may be displayed without dividing the RGB image into a plurality of regions. For example, the control unit 52 may display countermeasure information according to the incident level calculated from the entire RGB image as described in the embodiment. For example, the control unit 52 may determine the first countermeasure information when the incident level is less than a first level, determine the second countermeasure information when the incident level is equal to or greater than the first level and less than a second level higher than the first level, and determine the third countermeasure information when the incident level is equal to or greater than the second level.

[0221] Furthermore, the mobile terminal 50 according to the above embodiment has been described as an example in which the display mode of the plaque detection result is changed according to the incident level of external light incident on the captured area of ​​the RGB image, and information on measures to suppress the effects of external light according to the incident level is displayed. However, the present disclosure is not limited to this, and may be realized as an image processing method that does not change the display mode of the plaque detection result, but displays information on measures to suppress the effects of external light according to the incident level. For example, the present disclosure may be realized as an image processing method for displaying a plaque area based on an RGB image obtained by photographing teeth and plaque that are fluorescently reacting by irradiating the teeth with light including a wavelength range of blue light, the image processing method acquiring the incident level of external light incident on the captured area of ​​the RGB image, and displaying information on measures to suppress the effects of external light according to the incident level. This makes it possible to effectively suppress external light entering the oral cavity.

[0222] Furthermore, in the second modification of the above embodiment, an example of displaying countermeasure information has been described. However, for example, an RGB image reflecting the determination result of the presence or absence of external light (for example, an RGB image in which the area where external light is present is clearly indicated) may also be displayed. In this case, the RGB image may be displayed as a horizontally inverted image. This allows the incident direction of external light in the image as seen by the user to match the actual incident direction, making it easier for the user to intuitively grasp the incident direction.

[0223] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0224] Furthermore, the order in which each step in the flowchart is executed is merely an example for specifically explaining the present disclosure, and other orders may also be used. For example, the process of step S14 shown in FIG. 8 may be executed between steps S12 and S13. If step S14 is No, external light is incident, so the plaque detection process of step S13 may be omitted. Furthermore, some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0225] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0226] Furthermore, the mobile terminal 50 according to the above-described embodiments may be realized as a single device or may be realized by multiple devices. When the mobile terminal 50 is realized by multiple devices, the components of the mobile terminal 50 may be distributed among the multiple devices in any manner. For example, at least some of the functions of the mobile terminal 50 may be realized by the intraoral camera 10 (e.g., the signal processing unit 30). When the mobile terminal 50 is realized by multiple devices, the communication method between the multiple devices is not particularly limited and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0227] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, the term "LSI" is used, but depending on the level of integration, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After LSI fabrication, a field programmable gate array (FPGA) that can be programmed or a reconfigurable processor that can reconfigure the connections or settings of circuit cells within the LSI may also be used. Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0228] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system comprising a microprocessor, ROM, RAM, etc. The ROM stores computer programs. The system LSI achieves its functions when the microprocessor operates in accordance with the computer programs.

[0229] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the image processing method shown in FIG. 8 or FIG.

[0230] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.

[0231] The present disclosure is applicable to intraoral camera systems.

[0232] DESCRIPTION OF SYMBOLS 10 Intraoral camera 10a Head unit 10b Handle unit 10c Neck unit 20 Hardware unit 21 Photography unit 22 Sensor unit 23 Illumination unit 23A First LED 23B Second LED 23C Third LED 23D Fourth LED 24 Operation unit 30 Signal processing unit 31 Camera control unit 32 Image processing unit 33, 52 Control unit 34 Illumination control unit 35 Memory unit 40 Communication unit 50 Portable terminal 51 Acquisition unit 53 Display control unit 54 Touch screen R1, R2, R3, R4, R11, R12, R13, R14, R21, R22, R23, R24, R25, R26, R27, R28, R29 Area

Claims

1. An image processing method for displaying plaque areas based on RGB images obtained by photographing the teeth and plaque that are undergoing a fluorescent reaction when light including a blue light wavelength range is irradiated onto the teeth, the image processing method comprising the steps of: obtaining an incident level of external light entering the photographing area of ​​a first RGB image; performing plaque detection processing based on the first RGB image; and varying the display mode of the plaque detection result depending on the incident level of external light entering the photographing area of ​​the first RGB image.

2. The image processing method according to claim 1, further comprising the steps of: displaying an image showing the plaque detection result with a display indicating that external light is entering the image when the incidence level is not below a predetermined value.

3. The image processing method according to claim 2, further comprising the steps of: displaying an image showing the plaque detection result with a display indicating that no external light is being incident on the image when the incidence level is equal to or lower than the predetermined value.

4. The image processing method according to claim 1, wherein display of the plaque detection result is prohibited if the incidence level is not equal to or lower than a predetermined value.

5. An image processing method according to any one of claims 1 to 4, wherein the incidence level is obtained based on the detection result of a detector that detects external light incident on the shooting area of ​​the first RGB image.

6. An image processing method according to any one of claims 1 to 4, wherein the incidence level is calculated based on a first number of red pixels having a red pixel value equal to or greater than a first pixel value in the first RGB image.

7. The image processing method according to claim 6, wherein the incidence level includes a first incidence level, the first red pixel number includes a first number of red pixels having the pixel value at each pixel value equal to or greater than the first pixel value, and the number of green pixels having the green pixel value equal to or greater than the first pixel value includes a second number of green pixels having the pixel value at each pixel value equal to or greater than the first pixel value, the image processing method subtracts the second pixel number from the first pixel number at each pixel value equal to or greater than the first pixel value, calculates the first incidence level based on a first accumulated value obtained by accumulating a first subtraction value obtained by subtracting the second pixel number from the first pixel number, the first subtraction value being positive, and varies a display manner of the plaque detection result depending on whether the first incidence level is equal to or less than a first predetermined value.

8. The image processing method according to claim 6, wherein the incidence level includes a first incidence level, the first red pixel number includes a first number of red pixels having a pixel value equal to or greater than the first pixel value, and the number of green pixels having a green pixel value equal to or less than a fourth pixel value lower than the first pixel value includes a second number of green pixels having a pixel value equal to or less than the fourth pixel value, the image processing method comprising: subtracting the second pixel number from the first pixel number; and, if a first subtraction value obtained by subtracting the second pixel number from the first pixel number is positive, calculating the first incidence level based on the first subtraction value; and varying a display manner of the plaque detection result depending on whether the first incidence level is equal to or less than a first predetermined value.

9. The image processing method according to claim 7, wherein the incidence level includes a second incidence level, and a second number of red pixels having a red pixel value equal to or greater than a second pixel value higher than the first pixel value includes a third number of red pixels having the pixel value at each of the pixel values ​​equal to or greater than the second pixel value, and the image processing method subtracts the second pixel number from the third pixel number at each of the pixel values ​​equal to or greater than the second pixel value, calculates the second incidence level based on a second accumulated value obtained by accumulating the second subtraction value obtained by subtracting the second pixel number from the third pixel number, the second subtraction value being positive, and varies the display manner of the plaque detection result depending on whether the second incidence level is equal to or less than a second predetermined value.

10. The image processing method according to claim 7, wherein the incidence level includes a third incidence level, a V image is generated consisting of V values ​​obtained by HSV conversion of each pixel value of the first RGB image, the third incidence level is calculated based on an integrated value of the number of pixels whose V value is equal to or greater than a third pixel value, and further, the display manner of the plaque detection result is varied depending on whether the third incidence level is equal to or less than a third predetermined value.

11. The image processing method of claim 10, further comprising the steps of: displaying an image in which a display encouraging the user to block light is superimposed on an image displaying the plaque detection result when the first incidence level is equal to or less than the first predetermined value and equal to or greater than a fourth predetermined value lower than the first predetermined value, and the third incidence level is equal to or less than the third predetermined value and equal to or greater than a fifth predetermined value lower than the third predetermined value.

12. The image processing method according to claim 6, further comprising the steps of: generating a V image consisting of V values ​​obtained by HSV conversion of each pixel value of the first RGB image; and calculating the incidence level based on an integrated value of the number of pixels whose V values ​​are equal to or greater than the first pixel value.

13. The image processing method according to any one of claims 2 to 4, wherein the predetermined value is set according to a lighting environment of a space in which the first RGB image is captured.

14. An image processing method according to any one of claims 2 to 4, comprising: dividing the first RGB image into a plurality of regions; obtaining the incident level of external light incident on each of the plurality of divided regions; and varying the display manner of the plaque detection result depending on a determination result of whether the incident level of each of the plurality of regions is below the predetermined value.

15. The image processing method according to any one of claims 2 to 4, further comprising displaying information on measures to suppress the influence of the external light according to the incident level.

16. The image processing method according to claim 15, further comprising: dividing the first RGB image into a plurality of regions; acquiring the incident level of external light incident on each of the plurality of divided regions; and displaying countermeasure information according to at least one of the number of the plurality of regions in which the incident level exceeds the predetermined value and the position of the region within the plurality of regions.

17. An image processing method as described in claim 16, further comprising a judgment unit that judges whether or not a user has responded to the countermeasure information, and after it is judged that the user has responded to the countermeasure information, acquiring post-countermeasure incidence levels for each of the multiple regions in a second RGB image obtained by photographing the tooth and plaque that are undergoing a fluorescent reaction by irradiating the tooth with light including the wavelength range of blue light, and displaying countermeasure information other than the countermeasure information based on the post-countermeasure incidence levels for each of the multiple regions.

18. The image processing method according to claim 17, wherein the other countermeasure information is information corresponding to at least one of the number of areas among the plurality of areas in which the incidence level after the countermeasure exceeds the specified value, and the position of the area among the plurality of areas.

19. The image processing method according to claim 16, further comprising determining a direction in which the external light is incident based on a positional relationship of an area among the plurality of areas in which the incident level exceeds the predetermined value, and wherein the countermeasure information includes information regarding the direction in which the external light is incident.

20. An image processing system for displaying a plaque area based on an RGB image obtained by photographing the teeth and plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light, the image processing system comprising: an acquisition unit that acquires an incident level of external light entering the shooting area of ​​the RGB image; a plaque detection unit that performs plaque detection processing based on the RGB image; and a display control unit that changes the display mode of the plaque detection result depending on the incident level of external light entering the shooting area of ​​the RGB image.

21. A program for causing a computer to execute the image processing method according to any one of claims 1 to 4.

Images