Image processing method, image processing system, and program

Abstract

Provided is an image processing method for displaying a tartar region on the basis of an RGB image obtained by capturing an image of teeth and tartar, which have been subjected to a fluorescence reaction when the teeth are irradiated with light in a wavelength range including blue light, an incidence level of external light incident on an imaging region of an RGB image is obtained (S12), tartar detection processing is performed on the basis of the RGB image (S13), and display forms of tartar detection results are varied in accordance with the incidence level of external light incident on the imaging region of the RGB image (S15, S16).

Description

Technical Field

[0001] The present disclosure relates to an image processing method, an image processing system, and a program. Background Art

[0002] Patent Document 1 discloses an image processing apparatus that performs color correction on an image captured in a state where it is not shielded from light.

[0003] (Prior Art Document)

[0004] (Patent Document)

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2008-244794 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] Here, a technique for photographing the inside of the oral cavity using a photographing device to detect a tartar area inside the oral cavity is being studied. In this case, when using a photographing device to detect tartar or the like inside the oral cavity in a state where external light is incident on the inside of the oral cavity, there is a possibility of misdetecting the tartar area.

[0008] Accordingly, the present disclosure provides an image processing method, an image processing system, and a program capable of notifying the possibility of misdetecting a tartar area due to external light incident on the inside of the oral cavity.

[0009] Means for Solving the Problems

[0010] An image processing method according to one aspect of the present disclosure is an image processing method for displaying a tartar area based on an RGB image, the RGB image being an image obtained by photographing teeth and tartar that have undergone a fluorescence reaction when light in a wavelength range including blue light is irradiated on the teeth, in the image processing method, an incident level of external light incident on a photographing area of a first RGB image is obtained, a tartar detection process is performed based on the first RGB image, and a display form of a tartar detection result is made different according to the incident level of the external light incident on the photographing area of the first RGB image.

[0011] An image processing system according to an aspect of the present disclosure is an image processing system for displaying a calculus region based on an RGB image obtained by photographing a tooth and calculus that have undergone a fluorescence reaction when light in a wavelength range including blue light is irradiated onto the tooth, the image processing system including: an acquisition unit that acquires an incident level of external light incident on a photographing region of the RGB image; a calculus detection unit that performs calculus detection processing based on the RGB image; and a display control unit that makes the display form of the calculus detection result different according to the incident level of the external light incident on the photographing region of the RGB image.

[0012] A program according to an aspect of the present disclosure causes a computer to execute the above-described image processing method.

[0013] Advantageous Effects of the Invention

[0014] According to an aspect of the present disclosure, it is possible to implement an image processing method or the like that can notify the possibility of misdetecting a calculus region due to external light incident on the oral cavity. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a perspective view of an intraoral camera in an intraoral camera system according to an embodiment.

[0016] Figure 2 is a schematic configuration diagram of an intraoral camera system according to an embodiment.

[0017] Figure 3 is a block diagram showing a functional configuration of a portable terminal according to an embodiment.

[0018] Figure 4 is a first diagram showing a determination result of the presence or absence of external light according to an embodiment.

[0019] Figure 5 is a second diagram showing a determination result of the presence or absence of external light according to an embodiment.

[0020] Figure 6 is a diagram showing a setting screen for external light determination according to an embodiment.

[0021] Figure 7 is a diagram showing a display example of a photographed image and a calculus detection result in the case of the presence or absence of external light according to an embodiment.

[0022] Figure 8 is a flowchart showing the operation of a portable terminal according to an embodiment.

[0023] Figure 9 is a diagram showing a display example of the presence or absence of external light according to an embodiment.

[0024] Figure 10 This is a diagram showing an example of a display for prompting shading related to an embodiment.

[0025] Figure 11 This is a diagram schematically showing a state in which a photographed image is divided into a plurality of regions according to Modification 1 of the embodiment.

[0026] Figure 12 This is a diagram showing the relationship between the value of each pixel value and the number of pixels in a scene with external light.

[0027] Figure 13 This is a diagram showing the relationship between the value of each pixel value and the number of pixels in a scene without external light.

[0028] Figure 14 This is a flowchart showing the operation of a portable terminal according to Modification 2 of the embodiment.

[0029] Figure 15A This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the upper left side.

[0030] Figure 15B This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the upper right side.

[0031] Figure 15C This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the lower right side.

[0032] Figure 15D This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the lower left side.

[0033] Figure 15E This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the upper side.

[0034] Figure 15F This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the right side.

[0035] Figure 15G This is a diagram showing an example of the determination result of the incident level of each region divided into four regions when external light is incident from the lower side.

[0036] Figure 15HIt is a diagram showing an example of the determination results of the incident levels of the respective regions divided into 4 regions when external light is incident from the left side.

[0037] Figure 16A It is a diagram showing another example of the determination results of the incident levels of the respective regions divided into 4 regions when external light is incident from the upper left side.

[0038] Figure 16B It is a diagram showing another example of the determination results of the incident levels of the respective regions divided into 4 regions when external light is incident from the upper right side.

[0039] Figure 16C It is a diagram showing another example of the determination results of the incident levels of the respective regions divided into 4 regions when external light is incident from the lower right side.

[0040] Figure 16D It is a diagram showing another example of the determination results of the incident levels of the respective regions divided into 4 regions when external light is incident from the lower left side.

[0041] Figure 17A It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the upper left side.

[0042] Figure 17B It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the upper right side.

[0043] Figure 17C It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the lower left side.

[0044] Figure 17D It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the lower right side.

[0045] Figure 17E It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the left side.

[0046] Figure 17F It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the upper side.

[0047] Figure 17G It is a diagram showing the first example of the determination results of the incident levels of the respective regions divided into 9 regions when external light is incident from the lower side.

[0048] Figure 17H This is a first example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the right side.

[0049] Figure 18A This is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper left side.

[0050] Figure 18B This is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper right side.

[0051] Figure 18C This is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower left side.

[0052] Figure 18D This is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower right side.

[0053] Figure 18E This is a third example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper left side.

[0054] Figure 18F This is a third example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper right side.

[0055] Figure 18G This is a third example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower left side.

[0056] Figure 18H This is a third example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower right side.

[0057] Figure 19A This is a fourth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper left side.

[0058] Figure 19B This is a fourth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper right side.

[0059] Figure 19CIt is a fourth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower left side.

[0060] Figure 19D It is a fourth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower right side.

[0061] Figure 19E It is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the left side.

[0062] Figure 19F It is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper side.

[0063] Figure 19G It is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower side.

[0064] Figure 19H It is a second example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the right side.

[0065] Figure 20A It is a fifth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper left side.

[0066] Figure 20B It is a fifth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper right side.

[0067] Figure 20C It is a fifth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower left side.

[0068] Figure 20D It is a fifth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower right side.

[0069] Figure 20E It is a sixth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper left side.

[0070] Figure 20F It is a sixth example of a diagram showing the determination results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper right side.

[0071] Figure 20G It is a diagram of the sixth example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower left side.

[0072] Figure 20H It is a diagram of the sixth example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower right side.

[0073] Figure 21A It is a diagram of the seventh example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper left side.

[0074] Figure 21B It is a diagram of the seventh example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper right side.

[0075] Figure 21C It is a diagram of the seventh example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower left side.

[0076] Figure 21D It is a diagram of the seventh example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower right side.

[0077] Figure 21E It is a diagram of the third example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the upper side.

[0078] Figure 21F It is a diagram of the second example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the lower side.

[0079] Figure 21G It is a diagram of the second example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the left side.

[0080] Figure 21H It is a diagram of the second example showing the judgment results of the incident levels of the respective regions divided into nine regions when external light is incident from the right side.

[0081] Figure 22 It is a diagram showing an example of the first countermeasure information related to Modification Example 2 of the embodiment.

[0082] Figure 23 It is a diagram showing an example of the second countermeasure information related to Modification Example 2 of the embodiment.

[0083] Figure 24This is a diagram showing an example of the third countermeasure information related to Modification 2 of the embodiment. Detailed implementation mode

[0084] (Process of obtaining the present disclosure)

[0085] Before explaining the present disclosure, first refer to Figure 12 and Figure 13 to explain the formation process of the present disclosure. Figure 12 This is a diagram showing the relationship between the values of each pixel value and the number of pixels in a scene with external light. Figure 13 This is a diagram showing the relationship between the values of each pixel value and the number of pixels in a scene without external light. Figure 12 (a) of Figure 13 (a) of Figure 12 (b) of Figure 13 (b) of

[0086] In addition, Figure 12 and Figure 13 show the results in the case where the intraoral camera for photographing the oral cavity has a blue light cut filter described later. And having external light means that there is external light in the oral cavity, and hereinafter it is also described as having external light incident, etc. Having no external light means that there is no external light in the oral cavity, and hereinafter it is also described as having no external light incident, etc. In addition, the case of having no external light may include not only the case of completely having no external light but also the case of substantially having no external light.

[0087] As Figure 12 (a) of

[0088] Figure 12 (b) of

[0089] ​Here, according to the quantitative light-induced fluorescence method (QLF method), it can be known that when blue light is irradiated, bacteria in dental plaque emit pink fluorescence with a red color (excitation fluorescence). Therefore, when the overall image has a red color due to external light, it becomes difficult to detect the dental plaque area based on the image. For example, when an area without dental plaque is irradiated with external light, it may be detected as having dental plaque in that area due to the influence of the external light.

[0090] As Figure 13 shown in (a) of [], in the absence of external light, the number of pixels with a high pixel value in red is small (the number of pixels with a high Rch is small) (see the dotted box).

[0091] As Figure 13 shown in (b) of [], in the absence of external light, the number of pixels with a high brightness is small (the number of pixels with a high V is small) (see the dotted box).

[0092] In this way, the presence or absence of external light causes a significant difference in the number of pixels with a high pixel value in red. The inventors of the present invention focused on the number of pixels with a high pixel value in this red and found that an example of a method for determining the presence or absence of external light can utilize this number of pixels. Moreover, the inventors of the present invention, for example, intensively studied an image processing method and the like that can notify the possibility of misdetecting a dental plaque area due to external light incident on the oral cavity by using the number of pixels with a high pixel value in red, and proposed the image processing method and the like to be described below.

[0093] The image processing method according to the first aspect of the present disclosure is an image processing method for displaying a dental plaque area based on an RGB image, where the RGB image is an image obtained by photographing the teeth and dental plaque that have undergone a fluorescence reaction when light in a wavelength range including blue light is irradiated on the teeth. In the image processing method, the incident level of external light incident on the photographing area of the first RGB image is obtained, a dental plaque detection process is performed based on the first RGB image, and the display form of the dental plaque detection result is made different according to the incident level of the external light incident on the photographing area of the first RGB image.

[0094] Accordingly, it is possible to make the display form of the dental plaque detection result different according to whether external light is incident on the oral cavity. When external light is incident on the oral cavity, there is a possibility that the dental plaque area included in the dental plaque detection result is misdetected. For example, when external light is incident on the oral cavity, the display form is changed to a display form that can notify the incidence of external light, so that an image processing method that can notify the possibility of misdetecting a dental plaque area due to external light incident on the oral cavity can be realized.

[0095] In addition, for example, in the image processing method according to the second mode, based on the image processing method according to the first mode, when the incident level is not below the specified value, an image can be displayed which is obtained by overlapping a display showing the detection result of dental calculus on an image showing the incidence of external light.

[0096] Accordingly, it is possible to notify the user of the possibility of external light incidence, in other words, to notify the user of the possibility of misdetecting the dental calculus area.

[0097] In addition, for example, in the image processing method according to the third mode, based on the image processing method according to the first mode or the second mode, when the incident level is below the specified value, an image can be displayed which is obtained by overlapping a display showing no incidence of external light on an image showing the detection result of dental calculus.

[0098] Accordingly, it is possible to notify the user that there is no external light incidence, in other words, to notify the user that the possibility of correctly judging the dental calculus area is high.

[0099] In addition, for example, in the image processing method according to the fourth mode, based on the image processing method according to the first mode, when the incident level is not below the specified value, the display of the dental calculus detection result can be prohibited.

[0100] Accordingly, it is possible to suppress the display of the dental calculus detection result that may include an incorrect detection result to the user when there is a possibility of external light incidence.

[0101] In addition, for example, in the image processing method according to the fifth mode, based on any one of the image processing methods according to the first mode to the fourth mode, the incident level can be obtained based on the detection result of a detector that detects external light incident on the imaging area of the first RGB image.

[0102] Accordingly, the presence or absence of external light can be directly detected by the detector, so the presence or absence of external light can be detected more reliably. And it is possible to reduce the processing amount required for the processing device to process the presence or absence of external light when executing the image processing method.

[0103] In addition, for example, in the image processing method according to the sixth mode, based on any one of the image processing methods according to the first mode to the fifth mode, the incident level can be calculated based on the number of first red pixels having a red pixel value equal to or higher than a first pixel value in the first RGB image.

[0104] Accordingly, the presence or absence of external light can be judged based on the number of first red pixels that are likely to have a difference in pixel number due to the presence or absence of external light.

[0105] In addition, for example, in the image processing method according to the seventh mode, based on the image processing method according to the sixth mode, the incident level may include a first incident level, the first red pixel count may include the first pixel count of red pixels each having a pixel value equal to or greater than the first pixel value, the green pixel count of green pixels having a green pixel value equal to or greater than the first pixel value, and may include a second pixel count of green pixels each having a pixel value equal to or greater than the first pixel value. In the image processing method, for each pixel value equal to or greater than the first pixel value, the second pixel count is subtracted from the first pixel count, and based on the first cumulative value obtained by accumulating the first subtraction value that is positive when the second pixel count is subtracted from the first pixel count, the first incident level is calculated. Depending on whether the first incident level is below a first specified value, the display form of the dental calculus detection result is made different.

[0106] Accordingly, it is possible to determine the presence or absence of external light using the first incident level based on the first cumulative value, and to make the display form of the dental calculus detection result different according to the determination result of the presence or absence of external light.

[0107] In addition, for example, in the image processing method according to the eighth mode, based on the image processing method according to the sixth mode, the incident level may include a first incident level, the first red pixel count may include the first pixel count of red pixels having a pixel value equal to or greater than the first pixel value, the green pixel count of green pixels having a green pixel value lower than the first pixel value and equal to or lower than a fourth pixel value, and may include a second pixel count of green pixels having a pixel value equal to or lower than the fourth pixel value. In the image processing method, the second pixel count is subtracted from the first pixel count, and when the first subtraction value obtained by subtracting the second pixel count from the first pixel count is positive, the first incident level is calculated based on the first subtraction value. Depending on whether the first incident level is below a first specified value, the display form of the dental calculus detection result is made different.

[0108] Accordingly, it is possible to determine the presence or absence of external light using the first incident level based on the first cumulative value, and to make the display form of the dental calculus detection result different according to the determination result of the presence or absence of external light.

[0109] In addition, for example, in the image processing method according to the ninth mode, based on the image processing method according to the seventh mode, the incident level may include a second incident level. The second number of red pixels having a red pixel value higher than the first pixel value and higher than the second pixel value is included. The third number of pixels includes red pixels each having a pixel value equal to or higher than the second pixel value. In the image processing method, for each pixel value equal to or higher than the second pixel value, the second number of pixels is subtracted from the third number of pixels. The second incident level is calculated based on a second cumulative value obtained by accumulating a second subtraction value that is positive when the second subtraction value obtained by subtracting the second number of pixels from the third number of pixels is positive. Depending on whether the second incident level is equal to or lower than a second specified value, the display form of the calculus detection result is made different.

[0110] Accordingly, by using the second cumulative value, even when external light is incident on a local area (for example, there is specular reflection in a local area of a tooth), it is possible to determine that there is external light. Therefore, it is possible to suppress a false detection in which it is determined that there is no external light although external light is incident on a local area.

[0111] In addition, for example, in the image processing method according to the tenth mode, based on any one of the image processing methods according to the seventh to ninth modes, the incident level may include a third incident level. A V image composed of V values obtained by performing HSV conversion on the pixel values of the first RGB image is generated. The third incident level is calculated based on a cumulative value of the number of pixels having a V value equal to or higher than a third pixel value. Further, depending on whether the third incident level is equal to or lower than a third specified value, the display form of the calculus detection result is made different.

[0112] Accordingly, it is possible to make the display form of the calculus detection result different according to the number of red pixels and the brightness.

[0113] In addition, for example, in the image processing method according to the eleventh mode, based on the image processing method according to the tenth mode, when the first incident level is equal to or lower than the first specified value and higher than a fourth specified value lower than the first specified value, and when the third incident level is equal to or lower than the third specified value and higher than a fifth specified value lower than the third specified value, an image is displayed which is an image obtained by overlapping a display urging shading on an image displaying the calculus detection result.

[0114] Accordingly, shading is urged when it is not clear whether external light is incident. By shading when external light is incident, the influence of external light can be suppressed. And by shading when no external light is incident, it can be confirmed that no external light is incident.

[0115] In addition, for example, in the image processing method according to the twelfth aspect, based on the image processing method according to the sixth aspect, it may be to generate a V image composed of V values obtained by performing HSV conversion on each pixel value of the first RGB image, and calculate the incident level based on the cumulative value of the number of pixels with a V value equal to or greater than the first pixel value.

[0116] Accordingly, it is possible to determine the presence or absence of external light based on the V value.

[0117] In addition, for example, in the image processing method according to the thirteenth aspect, based on any one of the image processing methods according to the second to fourth aspects, the specified value may be set according to the lighting environment of the space in which the first RGB image is captured.

[0118] Accordingly, a suitable specified value is set for each lighting environment, so that even when the lighting environment changes, it is possible to correctly determine the presence or absence of external light.

[0119] In addition, for example, in the image processing method according to the fourteenth aspect, based on any one of the image processing methods according to the second to fourth aspects, the first RGB image may be divided into a plurality of regions, and for each of the divided plurality of regions, the incident level of external light incident on that region is obtained, and the display form of the dental calculus detection result is made different according to the determination result of whether the incident level of each region of the plurality of regions is below the specified value.

[0120] Accordingly, it is possible to determine the presence or absence of external light in a narrower area. Therefore, even when external light is incident on a local area, it is possible to correctly determine the presence of external light. Thus, it is possible to suppress false detection where it is determined that there is no external light although external light is incident on a local area.

[0121] In addition, for example, in the image processing method according to the fifteenth aspect, based on any one of the image processing methods according to the second to fourth aspects, it may be possible to display countermeasure information corresponding to the incident level for suppressing the influence of the external light.

[0122] Accordingly, by the user executing the countermeasure information, it is possible to suppress the incident level of external light. Therefore, it is possible to suppress external light from entering the oral cavity.

[0123] In addition, for example, the image processing method according to the sixteenth mode, based on the image processing method according to the fifteenth mode, may be to divide the first RGB image into a plurality of regions, and for each of the divided plurality of regions, obtain the incident level of external light incident on that region, and display countermeasure information corresponding to at least one of the number of regions among the plurality of regions where the incident level exceeds the specified value and the position of that region in the plurality of regions.

[0124] Accordingly, it is possible to display countermeasure information corresponding to the number of regions where external light is incident and the incident direction of the external light. Therefore, it is possible to efficiently suppress external light from entering the oral cavity.

[0125] In addition, for example, the image processing method according to the seventeenth mode, based on the image processing method according to the sixteenth mode, may further include a determination unit that determines whether the user has responded to the countermeasure information, and obtains the incident level after countermeasures for each region of the plurality of regions in the second RGB image. The second RGB image is an image obtained by photographing the teeth and dental calculus that have fluoresced when the teeth are irradiated with light in a wavelength range including blue light after it is determined that the user has responded to the countermeasure information. Based on the incident level after countermeasures for each region of the plurality of regions, display other countermeasure information different from the countermeasure information.

[0126] Accordingly, it is possible to display other countermeasure information, which is countermeasure information corresponding to the reduction effect of external light brought about by the displayed countermeasure information. For example, it is possible to display other countermeasure information that can further suppress the influence of external light when the reduction effect of external light obtained by the countermeasure information is insufficient. Therefore, it is possible to more surely suppress external light from entering the oral cavity.

[0127] In addition, for example, the image processing method according to the eighteenth mode, based on the image processing method according to the seventeenth mode, the other countermeasure information may be information corresponding to at least one of the number of regions among the plurality of regions where the incident level after countermeasures exceeds the specified value and the position of that region in the plurality of regions.

[0128] Accordingly, it is possible to display other countermeasure information corresponding to the number of regions where external light is incident and the incident direction of the external light. Therefore, it is possible to more efficiently suppress external light from entering the oral cavity.

[0129] In addition, for example, in the image processing method according to the nineteenth aspect, based on the image processing method according to any one of the sixteenth to eighteenth aspects, it may be possible to determine the direction of external light incidence based on the positional relationship of the regions in the plurality of regions where the incidence level exceeds the specified value, and the countermeasure information includes information regarding the direction of external light incidence.

[0130] Accordingly, it is possible to display countermeasure information that particularly emphasizes the direction of external light incidence.

[0131] The image processing system according to the twentieth aspect of the present disclosure is an image processing system for displaying a dental calculus region based on an RGB image, where the RGB image is an image obtained by photographing the tooth and dental calculus that have fluoresced when irradiated with light in a wavelength range including blue light, and the image processing system includes: an acquisition unit that acquires the incidence level of external light incident on the imaging region of the RGB image; a dental calculus detection unit that performs dental calculus detection processing based on the RGB image; and a display control unit that makes the display form of the dental calculus detection result different according to the incidence level of the external light incident on the imaging region of the RGB image. In addition, a program according to the twenty-first aspect of the present disclosure is for causing a computer to execute the image processing method according to any one of the first to nineteenth aspects.

[0132] Accordingly, the same effects as the above-described image processing method can be achieved.

[0133] In addition, these general or specific aspects can be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or can be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0134] In addition, the embodiments to be described below all show general or specific examples. The numerical values, shapes, constituent elements, arrangement positions and connection forms of the constituent elements, steps, and the order of steps shown in the following embodiments are all examples, and their main purpose is not to limit the present disclosure. And, among the constituent elements of the following embodiments, those not described in the constituent elements showing independent technical solutions are described as optional constituent elements.

[0135] In addition, each figure is a schematic diagram and not a precise illustration. Therefore, for example, the scales in each figure are not necessarily the same. And in each figure, the same reference signs are given to substantially the same constituents, and repeated descriptions are omitted or simplified.

[0136] In addition, in this specification, terms indicating the relationship between equivalent elements, terms indicating the shape of elements such as L-shaped, U-shaped, and rectangular shapes, as well as numerical values and numerical ranges not only represent expressions with strict meanings, but also mean ranges that are substantially equivalent. For example, it may mean including a difference of approximately a few percent (or approximately 10%).

[0137] In addition, in this specification, ordinal numbers such as "first" and "second" do not mean the number or order of components without special limitations, but are used for the purpose of distinguishing to avoid confusion of the same type of components.

[0138] (Embodiment)

[0139] The following describes the intraoral camera system and the image processing method according to this embodiment with reference to Figures 1 to 10 for explanation.

[0140] [1. Configuration of Intraoral Camera System]

[0141] First, regarding the configuration of the intraoral camera system including an intraoral camera according to this embodiment, reference is made to Figures 1 to 3 for explanation. Figure 1 is a perspective view of the intraoral camera 10 in the intraoral camera system according to this embodiment.

[0142] As Figure 1 [[ID=]26]shown, the intraoral camera 10 has a toothbrush-shaped housing that can be used with one hand. The housing includes a head 10a that is inside the user's oral cavity when photographing the tooth row, a handle portion 10b that the user holds, and a neck 10c that connects the head 10a and the handle portion 10b.

[0143] The imaging unit 21 is assembled in the head 10a and the neck 10c. The imaging unit 21 has an imaging element (not shown) and a lens (not shown) disposed on its optical axis LA.

[0144] The imaging element is, for example, a photographic device such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) element. The image of the teeth is imaged through the lens. A signal (image data) corresponding to the imaged image is output from the imaging element to the outside. The tooth row image captured by the imaging element is an example of an RGB image. And the tooth row image can be an image of the side of the teeth or an image of the occlusal surface of the teeth. And regarding the side of the teeth, it can be the inner (lingual) side or the outer (buccal) side.

[0145] The photographing unit 21 may also include a light-shielding filter that shields the colored light irradiated by the illumination unit and allows the fluorescence emitted by dental calculus due to the light to pass through. In the present embodiment, the photographing unit 21 may also include a blue light cut filter that cuts off the light component of the blue wavelength included in the light incident on the imaging element 14. When light in a wavelength range including blue light is irradiated onto teeth to detect dental calculus, and the light in the wavelength range including blue light is enhanced to enhance the excitation fluorescence of dental calculus, the entire RGB image will have a blue tint. In this state, since the blue pixel value dominates over the red pixel value and the green pixel value, the determination accuracy may decrease when using the red pixel value and the green pixel value to determine the presence or absence of external light. As a countermeasure, for the light before it enters the imaging element, the blue light cut filter cuts off the light in the wavelength range including blue light. Additionally, the photographing unit 21 may not include a blue light cut filter.

[0146] In addition, the intraoral camera 10 is equipped with first to fourth LEDs 23A to 23D as an illumination unit (lighting device) that irradiates light onto the teeth as the object to be photographed during shooting. The first to fourth LEDs 23A to 23D irradiate colored light (for example, single-color light), and the dental calculus emits fluorescence when irradiated with this light. The first to fourth LEDs 23A to 23D are, for example, blue LEDs that irradiate blue light having a wavelength with a peak at 405 nm. Additionally, the first to fourth LEDs 23A to 23D may be any light source that can irradiate light in a wavelength range including blue light, and is not limited to blue LEDs.

[0147] Figure 2 It is a schematic configuration diagram of the intraoral camera system according to the present embodiment.

[0148] As Figure 2 shown, the intraoral camera system according to the present embodiment is schematically configured such that the photographing unit 21 photographs the fluorescence emitted by dental calculus in response to the light from the illumination unit 23.

[0149] As Figure 2 shown, the intraoral camera system includes an intraoral camera 10 and a portable terminal 50. The intraoral camera system is an example of an image processing system.

[0150] The intraoral camera system is configured such that, when detecting a dental calculus area based on an image (RGB image) captured with the intraoral camera 10 in a state where external light enters the oral cavity, it is possible to notify the user of the possibility of misdetecting the dental calculus area due to the external light entering the oral cavity. The notification may be to display information indicating the possibility of misdetection on the image (for example, refer to Figure 9 ) described later, or the possibility of misdetection may be output by sound, light, etc. And the external light refers to light other than the light irradiated by the illumination unit 23, and examples thereof include illumination light, sunlight, etc. The external light may include, for example, light of the same color as the fluorescence emitted by dental calculus.

[0151] The intraoral camera 10 includes a hardware unit 20, a signal processing unit 30, and a communication unit 40.

[0152] The hardware unit 20 is a physical-level element in the intraoral camera 10 and has a photographing unit 21, a sensor unit 22, an illumination unit 23, and an operation unit 24.

[0153] The photographing unit 21 generates image data by photographing the teeth in the user's oral cavity. The photographing unit 21 receives a control signal from the camera control unit 31, performs operations such as photographing according to the received control signal, and outputs the image data of the moving image or still image obtained by photographing to the image processing unit 32. The photographing unit 21 has the above-described imaging element, light-shielding filter, and lens. In the photographing unit 21, the image data is generated based on the light passing through the light-shielding filter. And although the image data here is a dental arch image showing multiple teeth, as long as it is a dental arch image showing at least one tooth, it is acceptable.

[0154] The sensor unit 22 detects external light incident on the photographing area of the RGB image. For example, the sensor unit 22 detects whether there is external light incident in the oral cavity. The sensor unit 22 is, for example, arranged near the photographing unit 21. For example, like the photographing unit 21, the sensor unit 22 may also be arranged at the head 10a of the intraoral camera 10. In other words, when the photographing unit 21 performs photographing, the sensor unit 22 is located in the user's oral cavity. The sensor unit 22 is an example of a detector.

[0155] The illumination unit 23 irradiates light (blue light in this embodiment) to the area photographed by the photographing unit 21 in a plurality of areas in the oral cavity. The illumination unit 23 has a plurality of LEDs such as the above-described first LED 23A to fourth LED 23D. The plurality of LEDs of the first LED 23A to fourth LED 23D irradiate light to the photographing area from mutually different directions. Thereby, it is possible to suppress the generation of shadows in the photographing area.

[0156] Each of the first LED 23A to the fourth LED 23D is configured to be able to control dimming at least. Each of the first LED 23A to the fourth LED 23D may also be configured to be able to control dimming and color adjustment. A plurality of LEDs of the first LED 23A to the fourth LED 23D are arranged so as to surround the imaging unit 21.

[0157] The illumination unit 23 controls the irradiation intensity (luminous intensity) according to the imaging area. The irradiation intensity of each of the first LED 23A to the fourth LED 23D may be controlled to be the same or may be controlled to be different from each other. In addition, the number of LEDs included in the illumination unit 23 is not particularly limited and may be one or may be five or more. Further, the illumination unit 23 is not limited to having an LED as a light source and may have other light sources.

[0158] The operation unit 24 receives operations from the user. The operation unit 24 is constituted by, for example, buttons or the like, but may also be configured to receive operations by voice.

[0159] In addition, the hardware unit 20 may further include a battery (for example, a secondary battery) that supplies power to each component of the intraoral camera 10, a coil that performs wireless charging through an external charger connected to a commercial power supply, and an actuator required for at least one of composition adjustment and focus adjustment.

[0160] The signal processing unit 30 includes: each functional component implemented by a CPU (Central Processing Unit) or an MPU (Micro Processor Unit) that executes various processes described later, and a memory unit 35 such as a ROM (Read Only Memory) and a RAM (Random Access Memory) that stores programs for causing each functional component to execute various processes. The signal processing unit 30 includes a camera control unit 31, an image processing unit 32, a control unit 33, an illumination control unit 34, and a memory unit 35.

[0161] The camera control unit 31 is mounted, for example, on the handle unit 10b of the intraoral camera 10 and controls the imaging unit 21. The camera control unit 31 controls at least one of the aperture and the shutter speed of the imaging unit 21, for example, according to a control signal from the image processing unit 32.

[0162] The image processing unit 32 is mounted, for example, on the handle portion 10b of the intraoral camera 10, obtains a tooth row image (image data) captured by the imaging unit 21, performs image processing on the obtained tooth row image, and outputs the tooth row image after the image processing to the camera control unit 31 and the control unit 33. Further, the image processing unit 32 may output the tooth row image after the image processing to the memory unit 35 to store the tooth row image after the image processing in the memory unit 35.

[0163] The image processing unit 32 is constituted by a circuit, for example, and performs image processing such as noise reduction, AWB (Automatic White Balance) processing, and edge enhancement processing on the tooth row image. The AWB processing can be mainly performed on the tooth region in the tooth row image, for example.

[0164] In addition, the tooth row image (the tooth row image after the image processing) output from the image processing unit 32 can be transmitted to the portable terminal 50 via the communication unit 40, and the transmitted tooth row image can be displayed on the touch screen 54 of the portable terminal 50. Accordingly, the tooth row image can be presented to the user.

[0165] The control unit 33 detects a calculus region based on the image data after the image processing performed by the image processing unit 32. The control unit 33 uses a machine learning model that has been learned to output the calculus region reflected in the image data when the image data is input, to determine the calculus region reflected in the image data.

[0166] The illumination control unit 34 is mounted, for example, on the handle portion 10b of the intraoral camera 10, and controls the lighting and extinguishing of the first LED 23A to the fourth LED 23D. The illumination control unit 34 is constituted by a circuit, for example. For example, when the user performs an operation to start the intraoral camera 10 on the touch screen 54 of the portable terminal 50, a corresponding signal is transmitted from the portable terminal 50 to the signal processing unit 30 via the communication unit 40. The illumination control unit 34 of the signal processing unit 30 lights the first LED 23A to the fourth LED 23D according to the received signal.

[0167] In addition to the above programs, the memory unit 35 stores the tooth row image (image data) captured by the imaging unit 21 and the like. The memory unit 35 is implemented by a semiconductor memory such as a ROM and a RAM, for example, but is not limited thereto.

[0168] The communication unit 40 is a wireless communication module that performs wireless communication with the portable terminal 50. The communication unit 40 is mounted, for example, on the handle portion 10b of the intraoral camera 10 and performs wireless communication with the portable terminal 50 according to a control signal from the signal processing unit 30. Wireless communication is performed between the communication unit 40 and the portable terminal 50 according to existing communication standards such as WiFi (registered trademark) and Bluetooth (registered trademark). A dental arch image showing teeth is transmitted from the intraoral camera 10 to the portable terminal 50 via the communication unit 40, or an operation signal is transmitted from the portable terminal 50 to the intraoral camera 10.

[0169] The portable terminal 50 displays, for example, a calculus area based on an RGB image obtained by photographing teeth and calculus that have fluoresced when light in a wavelength range including blue light is irradiated onto the teeth. Further, the portable terminal 50 functions as a user interface of the intraoral camera system.

[0170] Figure 3 It is a block diagram showing the functional configuration of the portable terminal 50 according to the present embodiment.

[0171] As Figure 3 shown, the portable terminal 50 includes an acquisition unit 51, a control unit 52, a display control unit 53, and a touch screen 54. The portable terminal 50 includes a processor, a memory, and the like. The memory is a ROM, a RAM, or the like and can store programs executed by the processor. The acquisition unit 51, the control unit 52, and the display control unit 53 are implemented by a processor or the like that executes programs stored in the memory. The portable terminal 50 can be implemented by, for example, a smartphone or a tablet terminal capable of performing wireless communication.

[0172] The acquisition unit 51 acquires an RGB image from the intraoral camera 10. The acquisition unit 51 is, for example, a wireless communication module that performs wireless communication.

[0173] The control unit 52 performs processing for detecting a calculus area based on the RGB image. The control unit 52 performs a level determination process for determining the incident level of external light incident on the imaging area of the image and a calculus detection process for detecting a calculus area based on the RGB image. The control unit 52 is an example of a calculus detection unit.

[0174] Here, the level determination process performed by the control unit 52 will be described with reference to Figures 4 to 6 .

[0175] The control unit 52 determines whether there is an incident of external light based on at least one of the number of pixels of a specified color and the number of pixels having a brightness equal to or higher than a specified value in the RGB image. In the present embodiment, as a first method, the control unit 52 can determine whether there is an incident of external light based on the first red pixel number, which includes the first pixel number of red pixels each having a pixel value equal to or higher than a first pixel value, since the entire image tends to be red when there is an incident of external light. Moreover, in the present embodiment, the control unit 52 can also determine whether there is an incident of external light based on the first green pixel number, which includes the second pixel number of green pixels each having a pixel value equal to or higher than a first pixel value.

[0176] For each pixel value equal to or higher than the first pixel value, the control unit 52 subtracts the second pixel number having that pixel value from the first pixel number having that pixel value, accumulates the first subtraction value obtained by subtracting the second pixel number from the first pixel number and being positive, and calculates a first incident level based on the first cumulative value obtained by accumulating the first subtraction values. For example, the control unit 52 can calculate the value obtained by dividing the first cumulative value by the total number of pixels as the first incident level. In addition, calculating the incident level is an example of obtaining the incident level.

[0177] In addition, as a second method, the control unit 52 can determine whether there is an incident of external light by evaluating the number of pixels whose green pixel value is smaller than the red pixel value. In this case, the control unit 52 can determine whether there is an incident of external light based on the first red pixel number, which includes the first pixel number of red pixels each having a pixel value equal to or higher than a first pixel value, and the green pixel number, which includes the second pixel number of green pixels each having a pixel value equal to or lower than a fourth pixel value (green pixel value) lower than the first pixel value. Moreover, the control unit 52 can subtract the second pixel number from the first pixel number, and when the first subtraction value obtained by subtracting the second pixel number from the first pixel number is positive, calculate a first incident level based on the first subtraction value. For example, the control unit 52 can calculate the value obtained by dividing the first subtraction value by the total number of pixels as the first incident level. In addition, the first red pixel number is not limited to the above, and the first red pixel number can be the number of pixels having a larger red pixel value than the green pixel value, and the green pixel number can be the number of pixels having a larger green pixel value than the red pixel value.

[0178] Moreover, the control unit 52 makes the display form of the dental calculus detection result different according to the incident level of external light. Specifically, the control unit 52 determines whether the first incident level is below the first specified value, and outputs the determination result to the display control unit 53, thereby making the display form of the dental calculus detection result different according to the determination result. In addition, the first pixel value and the first specified value are appropriately set according to the illumination environment of the space where the intraoral camera 10 is used. For example, the first pixel value is set according to the illumination environment for capturing the RGB image. For example, when the illuminance is 800 lux and the pixel value is represented by 8 bits, the first pixel value may be 130. And, from the viewpoint of suppressing the following situation, the first pixel value may be set to a value higher than 130, which is the situation where external light is judged to exist without external light, in other words, the judgment of dental calculus is stopped without external light.

[0179] In addition, the control unit 52 generates a V image composed of V values (brightness) obtained by performing HSV conversion on each pixel value of the RGB image, and calculates the third incident level based on the third cumulative value of the number of pixels whose V values are above the third pixel value. The control unit 52 can, for example, use the value obtained by dividing the third cumulative value by the total number of pixels as the third incident level. Moreover, the control unit 52 determines whether the third incident level is below the third specified value, and outputs the determination result to the display control unit 53, thereby making the display form of the dental calculus detection result different according to the determination result. In addition, the third pixel value and the third specified value are appropriately set according to the illumination environment of the space where the intraoral camera 10 is used. For example, when the illuminance is 800 lux and the pixel value is represented by 8 bits, the third pixel value may be 250.

[0180] Figure 4 is the first diagram showing the determination result of the presence or absence of external light according to the present embodiment. Figure 4 The horizontal axis represents the first incident level ( Figure 4 "R - G Range1" in Figure 4 ), and the vertical axis represents the third incident level ( Figure 4 "V Range" in Figure 4 ). And,

[0181] Figure 4Region A is an area where almost all drawn circles are without external light. Therefore, when the first incident level is below the threshold r2 (for example, about 0.18), it is determined that there is no external light, enabling the determination of no external light with high precision. Similarly, when the third incident level is below the threshold v2 (for example, about 0.24), it is determined that there is no external light, enabling the determination of no external light with high precision. In this case, since there is no external light, tartar detection processing is performed.

[0182] Figure 4 Region C is an area where all drawn circles are with external light. Therefore, when the first incident level is above the threshold r1 (for example, 0.45), it is determined that there is external light, enabling the determination of external light with high precision. Similarly, when the third incident level is above the threshold v1 (for example, 0.45), it is determined that there is external light, enabling the determination of external light with high precision. In this case, due to the external light, the possibility of misdetecting the tartar area is high, so the tartar detection processing is not performed. The threshold r1 is an example of the first specified value, and the threshold r2 is an example of the fourth specified value. In addition, the threshold v1 is an example of the third specified value, and the threshold v2 is an example of the fifth specified value.

[0183] Figure 4 Region B is an area where circles with external light and circles without external light are drawn and mixed together. In this case, although the tartar detection processing is performed, there is a possibility of misdetecting the tartar area due to the external light. Then, when the area where at least one of the first incident level and the third incident level is drawn is Region B and Region A, the display form of the tartar detection result is made different. Specifically, in the case of Region B, the tartar detection result is displayed in a display form that can inform the user of the possibility of misdetection of the tartar area.

[0184] In addition, for example, when the following judgment formula obtained according to Figure 4 the result is satisfied, the control unit 52 can determine that there is external light.

[0185] When the control unit 52 uses the first method, for example, when the value (number of pixels) obtained by subtracting the number of green pixels with a pixel value of 130 from the number of red pixels with a pixel value of 130 is positive, the value itself is directly used for calculation. When the value is negative, the value is set to zero. Next, the control unit 52 performs the same processing for each pixel value from 131 to 255. For example, when there are 100 red pixels with a pixel value of 130 and 50 green pixels with a pixel value of 130, the value for a pixel value of 130 becomes 50. And, for example, when there are 100 red pixels with a pixel value of 131 and 150 green pixels with a pixel value of 131, the value for a pixel value of 130 becomes 0.

[0186] In addition, when the control unit 52 uses the second method, for example, the value (number of pixels) obtained by subtracting the number of pixels with a green pixel value of 129 or less (in other words, the fourth pixel value is 129) from the number of pixels with a red pixel value of 130 or more (in other words, the first pixel value is 130) is used in the calculation. For example, when there are 100 pixels with a red pixel value of 130 or more and 50 pixels with a green pixel value of 129 or less, the value becomes 50. In addition, for example, when there are 50 pixels with a red pixel value of 130 or more and 100 pixels with a green pixel value of 129 or less, the value becomes 0.

[0187] Moreover, when the control unit 52 uses the first cumulative value obtained by accumulating the value when it is positive and satisfies the following formula 1, it is determined that there is external light.

[0188] First cumulative value / Total number of pixels > 0.45 ··· (Formula 1)

[0189] In the above first method, the first cumulative value of the two pixel values of 130 and 131 becomes 50 (50 + 0). In addition, in the above second method, when there are 100 pixels with a red pixel value of 130 or more and 50 pixels with a green pixel value of 129 or less, the first cumulative value becomes 50 (100 - 50).

[0190] In addition, the control unit 52 can determine that there is external light when the first cumulative value is higher than a specified value. In other words, it is possible to determine the presence or absence of external light without dividing the first cumulative value by the total number of pixels. 0.45 is an example of the first specified value.

[0191] In addition, when the control unit 52 satisfies the following formula 2 using, for example, the third cumulative value, it is determined that there is external light. The third cumulative value is obtained by cumulating the number of pixels for each brightness value between 250 and 255 in brightness.

[0192] Third cumulative value / Total number of pixels > 0.45 ··· (Formula 2)

[0193] In addition, the control unit 52 may determine that there is external light when the third cumulative value is higher than a specified value. In other words, it is possible to determine the presence or absence of external light without dividing the third cumulative value by the total number of pixels. 0.45 is an example of the third specified value.

[0194] In addition, the control unit 52 may determine whether external light is incident based on the following second number of red pixels, where the second number of red pixels includes the third number of pixels of red pixels having each pixel value greater than or equal to a second pixel value that is greater than the first pixel value. Further, in the present embodiment, the control unit 52 further determines whether external light is incident based on the following first number of green pixels, where the first number of green pixels includes the second number of pixels of green pixels having each pixel value greater than or equal to the first pixel value.

[0195] For each pixel value greater than or equal to the second pixel value, the control unit 52 subtracts the second number of pixels from the third number of pixels, cumulates the second subtraction value that is positive when the second subtraction value obtained by subtracting the second number of pixels from the third number of pixels, and calculates a second incident level based on the second cumulative value obtained by cumulating the second subtraction value. The control unit 52 may, for example, use the value obtained by dividing the second cumulative value by the total number of pixels as the second incident level. Further, the control unit 52 outputs the determination result to the display control unit 53 by determining whether the second incident level is less than or equal to a second specified value, and thereby makes the display form of the dental calculus detection result different according to the determination result. In addition, the second pixel value is appropriately set according to the illumination environment of the space using the intraoral camera 10. For example, when the illuminance is 800 lux and the pixel value is represented by 8 bits, the second pixel value may be 250.

[0196] Figure 5 is a second graph showing the determination result of the presence or absence of external light according to the present embodiment. Figure 5 The horizontal axis of Figure 5 represents the second incident level ( Figure 5 “R-G Range2” in Figure 5 ), and the vertical axis represents the third incident level ( Figure 5The non-shaded circles represent the second incident level and the third incident level calculated based on the image captured without external light.

[0197] Figure 5 Region E is the region where circles with external light are all drawn. Therefore, when the second incident level is above the threshold r3 (for example, 0.2), it is determined that there is external light, and the presence of external light can be determined with high precision. In this case, since there is a high possibility of misdetecting the calculus area due to external light, the calculus detection process is not performed. The threshold r3 is an example of the first specified value.

[0198] Figure 5 Region D is the region where circles with external light and circles without external light are drawn together. In this case, although the calculus detection process is performed, there is a possibility of misdetecting the calculus area due to external light. Thus, when the region where the second incident level and the third incident level are drawn is region D, the calculus detection result is displayed in a display form that can inform the user of the possibility of misdetecting the calculus area.

[0199] In addition, for example, when the following judgment formula obtained according to Figure 5 the result is satisfied, the control unit 52 can determine that there is external light.

[0200] For example, when the value (number of pixels) obtained by subtracting the number of green pixels with a pixel value of 250 from the number of red pixels with a pixel value of 250 by the control unit 52 is positive, the value itself is directly used for calculation. When the value is negative, the value is set to zero. Next, the control unit 52 performs the same process for each pixel value from 251 to 255. Moreover, when the second cumulative value obtained by accumulating the value when it is positive by the control unit 52 satisfies the following formula 3, it is determined that there is external light.

[0201] Second cumulative value / total number of pixels > 0.2 ··· (Formula 3)

[0202] In addition, the control unit 52 can determine that there is external light when the second cumulative value is higher than the specified value. In other words, it is possible to determine the presence or absence of external light without dividing the second cumulative value by the total number of pixels. 0.2 is an example of the second specified value.

[0203] The control unit 52 makes judgments on (Equation 1) to (Equation 3) respectively. When at least one of the equations is satisfied, it can be determined that there is external light. For example, when a local area is irradiated with external light, it may be determined that there is no external light in the judgment using Equation 1. However, by using the judgment of Equation 2, even if the local area is irradiated with external light, the possibility of determining that there is external light can be increased. In other words, the presence or absence of external light can be determined with higher accuracy.

[0204] In addition, as described above, regarding the pixel value for counting the number of pixels, it can be changed according to the spatial illumination environment. This pixel value (the first to third pixel values described above) can be input by the user, for example.

[0205] Figure 6 FIG. is a diagram showing a setting screen S1 for external light determination according to the present embodiment. The setting screen S1 is a screen displayed on the touch screen.

[0206] As Figure 6 shown, the range of the number of pixels for counting the number of pixels to determine the presence or absence of external light can be set by the user. Furthermore, the threshold values for determining the presence or absence of external light (''0.45'' in Equation 1 and Equation 2 and ''0.2'' in Equation 3) can also be set by the user.

[0207] In addition, the portable terminal 50 may store a table corresponding to the illumination environment of using the intraoral camera and Figure 6 the range of the number of pixels and the threshold values that can be set therein. According to this table and the illuminance measured by the sensor unit 22, the range of the number of pixels and the threshold values are automatically changed.

[0208] In addition, in the dental calculus detection process, the control unit 52 identifies teeth or dental calculus and lips or gums based on the hue on the RGB image. It is known that when light in a wavelength range including blue light is irradiated onto teeth, excited fluorescence is emitted from dentin and green light is emitted through enamel. On the other hand, since lips and gums are blue, the control unit 52 can easily identify teeth or dental calculus and lips or gums. And by using the RGB image that has undergone image processing such as white balance adjustment processing, it is possible to more easily distinguish the dental calculus area. By performing image processing to substantially colorless the area of the teeth, it is possible to easily distinguish the area where dental calculus adheres to the teeth, that is, the dental calculus area. Therefore, the control unit 52 can easily determine the dental calculus area in the image of the teeth. Accordingly, for example, an image of the teeth after brushing is taken, the dental calculus area is determined, and the area that is not brushed clean is presented to the user.

[0209] Refer to again Figure 2, the display control unit 53 performs control for causing the detected calculus region to be displayed on the touch screen 54. The display control unit 53 makes the display form of the calculus detection result different according to whether the incident level of the external light incident on the photographing region of the RGB image is below a specified value.

[0210] The touch screen 54 functions as an input device and an output device, and is configured to be able to display, for example, an image of a dental arch. The touch screen 54 may have, for example, a display unit such as a liquid crystal display panel, and a touch screen disposed overlapping with the display unit.

[0211] Here, an example of the image displayed on the touch screen 54 will be described with reference to Figure 7 as follows. Figure 7 is a diagram showing a display example of a photographed image and a calculus detection result in the presence or absence of external light according to the present embodiment. In Figure 7 , display examples of the photographed image and the calculus detection result are shown for the case where there is external light and the case where there is no external light, respectively. The photographed image is an RGB image obtained by the intraoral camera 10 photographing the teeth and gums in the oral cavity. And, the display of the calculus detection result means an image in which the calculus region detected after the calculus detection process of the execution control unit 52 is prominently displayed, and is an image displayed on the touch screen 54.

[0212] In the case where there is no external light, the calculus region in the photographed image and the region determined to be calculus in the display of the calculus detection result are substantially the same region.

[0213] On the other hand, in the case where there is external light, although it is not a calculus region in the photographed image, a region determined to be calculus appears in the display of the calculus detection result. This region is a false positive region and is a region misdetected due to the influence of external light. In this case, it is possible to notify the user that it is a false positive (refer to Figure 9 etc. described later).

[0214] In this way, by using the intraoral camera system, the user can photograph an image of his or her own oral cavity through the intraoral camera 10 and confirm the state of the oral cavity displayed on the portable terminal 50. Accordingly, it is easy for the user to confirm the health state of his or her own teeth.

[0215] [2. Operation of the Intraoral Camera System]

[0216] Next, regarding the operation of the intraoral camera system configured as described above, reference will be made to Figures 8 to 10 for description. Figure 8 is a flowchart showing the operation (image processing method) of the portable terminal 50 according to the present embodiment. In addition, Figure 8The processing shown is, for example, real-time processing that is performed whenever image data of one or more frames is obtained.

[0217] The intraoral camera 10 sends image data (RGB image) to the portable terminal 50. This image data is the image data generated when the user uses the intraoral camera 10 to photograph the teeth and gums inside their own mouth. Here, the image data can be a moving image or one or more still images. Also, in the case where the image data is a moving image or multiple still images, sensor data (external light detection result) can be sent by the sensor unit 22 for each frame of the moving image or each still image. Additionally, in the case where the image data is a moving image, sensor data can be sent for every multiple frames.

[0218] Furthermore, the transmission of the image data can be performed in real-time or can be aggregated and sent after a series of shootings (for example, shooting of all the teeth inside the mouth).

[0219] As Figure 8 shown, the acquisition unit 51 acquires the RGB image sent from the intraoral camera 10 (S11). The RGB image is an image obtained by the intraoral camera 10 photographing the teeth and dental plaque that have fluoresced when irradiated with light in a wavelength range including blue light.

[0220] Next, the control unit 52 obtains the incident level of the external light based on the acquired RGB image (S12). The control unit 52 can, for example, calculate the incident level by dividing at least one of the first cumulative value to the third cumulative value based on the RGB image by the total number of pixels. Additionally, the incident level includes at least one of the first incident level to the third incident level.

[0221] Next, the control unit 52 performs a dental plaque detection process based on the acquired RGB image (S13).

[0222] Next, the control unit 52 determines whether the incident level obtained in step S12 is below a specified value (S14). Step S14 corresponds to determining whether external light has entered the oral cavity.

[0223] Next, when the control unit 52 determines that the incident level is not below the specified value (No in S14), it outputs the determination result of step S14 to the display control unit 53, so that an image including information indicating the incidence of external light and the tartar detection result, or an image not including the tartar detection result (for example, an RGB image obtained from the intraoral camera 10) is displayed on the touch screen 54 (S15). Further, when the control unit 52 determines that the incident level is below the specified value (Yes in S14), it outputs the determination result of step S14 to the display control unit 53, so that an image including the tartar detection result is displayed (normal display) (S16).

[0224] Figure 9 FIG. is a diagram showing a display example of the presence or absence of external light according to the present embodiment. ​ FIG. shows a dental arch image. In addition, for the sake of convenience of explanation, in ​ (a) and (b) of, the illustration of the display (for example, highlighting) of the tartar detection result is omitted.

[0225] ​ (a) of is the image displayed in step S15, and a first piece of information indicating the incidence of external light (or the possibility of external light incidence) ( ​ the shaded circle with diagonal lines in (a) of ) is superimposed on the image including the tartar detection result and the superimposed image is displayed. The first piece of information is, for example, superimposed on and displayed in a region different from the tooth region and the tartar region (for example, the highlighted region). Further, when the incident level is not below the specified value, the control unit 52 may prohibit the display of the tartar detection result or may prohibit the execution of the tartar detection process.

[0226] ​ (b) of is the image displayed in step S16, and a second piece of information indicating the absence of external light incidence ( ​ the non-shaded circle with diagonal lines in (b) of ) is superimposed on the image including the tartar detection result and the superimposed image is displayed. The second piece of information is different from the first piece of information. Further, the second piece of information may not be displayed. In other words, in the case of step S16, an image including only the tartar detection result among the information indicating the absence of light incidence and the tartar detection result may be displayed.

[0227] In this way, the control unit 52 displays the first piece of information when there is external light, and displays the second piece of information or only the tartar detection result when there is no external light. In this way, the control unit 52 makes the display form of the tartar detection result different according to the presence or absence of external light.

[0228] Further, the control unit 52 further determines whether the first incident level satisfies a first specified value (for example, ​below the shown threshold r1) and below the fourth specified value lower than the first specified value (e.g., ​ above the shown threshold r2), and when the third incident level is below the third specified value (e.g., ​ below the shown threshold v1) and above the fifth specified value lower than the third specified value (e.g., ​ above the shown threshold v2), in the case of determining that at least one of them is satisfied, the determination result is output to the display control unit 53, and information for urging light shielding is overlapped on the image showing the dental calculus detection result and displayed. In addition, each specified value can be set according to the illumination environment in which the RGB image is captured.

[0229] ​ is a diagram showing an example of the display for urging light shielding according to the present embodiment.

[0230] As ​ shown, the control unit 52 can give a notification to urge the user to prevent external light from entering the oral cavity by light shielding. And the control unit 52 can display a prompt to dim the illumination or turn off the illumination instead of the display for urging light shielding. In addition, the display for urging light shielding can include, for example, a display for urging closing the curtain. In addition, the display for urging light shielding is, for example, overlapped and displayed in an area different from the tooth area and the dental calculus area (e.g., the highlighted area).

[0231] (Modification Example 1 of the Embodiment)

[0232] Hereinafter, regarding the intraoral camera system according to this modification example, refer to ​ to explain. In addition, the differences from the embodiment will be mainly described below, and the same or similar contents as those of the embodiment will be omitted or briefly described.

[0233] ​ is a diagram schematically showing a state in which the captured image according to this modification example is divided into a plurality of regions.

[0234] As ​ shown, the control unit 52 may divide the RGB image into a plurality of regions ( ​ in the example of four regions of regions R1 to R4), obtain the incident level of external light incident on each of the divided plurality of regions, and make the display form of the dental calculus detection result different according to the determination result of whether the incident level of each of the plurality of regions is below the specified value. The control unit 52 can determine that there is external light, for example, when the incident level of at least one region among the plurality of regions is not below the specified value. The control unit 52 can perform the determinations of Expression 1 and Expression 3 shown in the embodiment in each of the plurality of regions, and determine that there is external light when at least one of Expression 1 and Expression 3 is satisfied in at least one region.

[0235] In addition, the first pixel value is set to a value larger than the first pixel value in the embodiment. For example, when it is determined that the area on the image of the incident level is narrower, a larger first pixel value can be set. And, similar to the first pixel value, the second pixel value is set to a value larger than the second pixel value in the embodiment. In this modification example, the first pixel value is 0.8 and the second pixel value is 0.5, for example, but is not limited thereto. The intraoral camera 10 may store a table in which the sizes of multiple regions are associated with the first pixel value and the second pixel value, and the first pixel value and the second pixel value may be automatically set according to this table. And, the sizes of the multiple regions, etc. can be set by the user, for example.

[0236] In addition, the number of divisions is not particularly limited as long as it is two or more. And, the sizes, shapes, etc. of the respective regions may be equal or different.

[0237] (Modification Example 2 of the Embodiment)

[0238] Next, for the intraoral camera system according to this modification example, refer to ​ to explain. In addition, the differences from the embodiment will be mainly described below, and the same or similar content as the embodiment will be omitted or simplified.

[0239] ​ is a flowchart showing the operation (image processing method) of the portable terminal 50 according to this modification example. The difference between the intraoral camera system according to this modification example and the intraoral camera system according to the embodiment is the part that divides the RGB image into multiple regions and prompts the user with information urging countermeasures against external light according to the incident level of external light (for example, the presence or absence of external light) for each of the multiple regions. The symbols of the intraoral camera system according to the embodiment will be used for the following description.

[0240] As ​ shown, the acquisition unit 51 acquires the RGB image transmitted from the intraoral camera 10 (S21). The RGB image is an image obtained by the intraoral camera 10 photographing teeth and dental plaque that have undergone a fluorescence reaction when light in a wavelength range including blue light is irradiated on the teeth.

[0241] Next, the control unit 52 divides the RGB image into multiple regions (S22). The control unit 52 only needs to divide the RGB image into two or more regions. For example, as ​ shown, the RGB image can be divided into 4 regions, or can be divided into 9 regions, or can be divided into other numbers of regions. In this modification example, the cases of dividing into 4 regions and dividing into 9 regions will be mainly described.

[0242] In addition, examples where the shapes of multiple regions are quadrilaterals are described below. However, the shape is not limited to a quadrilateral and can be any other shape such as a circle, a triangle, a polygon with five or more sides, etc. Also, regions of multiple shapes can coexist in an RGB image. Further, the areas of the multiple regions can be equal, or the area of at least one region can be different from the areas of the other regions. Additionally, the quadrilateral can be a square or a rectangle.

[0243] Next, for each of the multiple regions, the control unit 52 obtains the incident level of external light incident on the region (S23). For example, for each of the multiple regions, the control unit 52 calculates at least one cumulative value of the first to third cumulative values of the region, and divides the calculated at least one cumulative value by the total number of pixels in the region to calculate the incident level. One incident level is calculated for one region. Additionally, the incident level for each of the multiple regions is not limited to being obtained by calculation and can be obtained from an external device via the acquisition unit 51, for example.

[0244] Next, for each of the multiple regions, the control unit 52 determines the presence or absence of external light (S24). For each of the multiple regions, the control unit 52 determines the presence or absence of external light in the region based on the incident level of the region. For example, for each of the multiple regions, the control unit 52 uses the above (Equation 1) or (Equation 2) to determine the presence or absence of external light in the region. In this case, the first cumulative value shown in (Equation 1) and the second cumulative value shown in (Equation 2) are values of the region, and the total number of pixels is the total number of pixels in the region.

[0245] Next, the control unit 52 determines the incident direction of external light with respect to the RGB image (S25). In other words, the control unit 52 determines the incident direction of external light incident on the imaging region in the oral cavity. The control unit 52 can, for example, determine the incident direction of external light based on the positional relationship (e.g., position in the RGB image) of the regions in which the incident level exceeds a specified value among the multiple regions (i.e., the regions determined to have external light).

[0246] In addition, the control unit 52 can determine whether external light is incident on the RGB image based on the determination result of step S24. In the case where it is determined that external light is incident on the RGB image, the processing after step S25 is executed. In the case where it is determined that external light is not incident on the RGB image, the processing ends. The control unit 52 can determine that external light is incident on the RGB image when it is determined in step S24 that at least one of the multiple regions has external light.

[0247] Here, an example of the relationship between the position of a region determined to have external light in an RGB image and the incident direction of the external light will be described with reference to ​ First, the case where there are four regions (the case where the RGB image is divided into four regions) will be described with reference to ​ By dividing the RGB image into four regions, for example, it is possible to both suppress an increase in the processing of the portable terminal 50 and determine the incident direction of the external light. ​ FIG. is an example showing the determination result of the incident level of each of the four regions into which the RGB image is divided when external light is incident from each direction. ​ FIG. is another example showing the determination result of the incident level of each of the four regions into which the RGB image is divided when external light is incident from each direction. In addition, a region determined to have external light is represented by diagonal hatching, and a region determined to have no external light is represented by a quadrilateral frame (without shading).

[0248] ​ shows the incident direction of the external light when it is determined that one of the four regions into which the RGB image is divided has external light, ​ shows the incident direction of the external light when it is determined that two of the four regions into which the RGB image is divided have external light, ​ shows the incident direction of the external light when it is determined that three of the four regions into which the RGB image is divided have external light. In addition, the four divided regions are set as regions R11 to R14. Region R11 is the upper left region in the RGB image, region R12 is the upper right region in the RGB image, region R13 is the lower left region in the RGB image, and region R14 is the lower right region in the RGB image.

[0249] As shown in ​ and ​ when the control unit 52 determines that only region R11 has external light, it determines that the external light is incident from the upper left (upper left side) of the RGB image. When it determines that only region R12 has external light, it determines that the external light is incident from the upper right (upper right side) of the RGB image. And as shown in ​ and ​ when the control unit 52 determines that only region R14 has external light, it determines that the external light is incident from the lower right (lower right side) of the RGB image. When it determines that only region R13 has external light, it determines that the external light is incident from the lower left (lower left side) of the RGB image. In this way, when the control unit 52 determines that only one of the four divided regions has external light, for example, it can be determined that the external light is incident from any one of the diagonal directions.

[0250] In addition, as​ and ​ As shown in ​ , when the control unit 52 determines that there is external light only in regions R11 and R12, it determines that the external light is incident from above (the upper side) of the RGB image. When it determines that there is external light only in regions R12 and R14, it determines that the external light is incident from the right side (the right side) of the RGB image. And, as ​ and ​ shown in ​ , when the control unit 52 determines that there is external light only in regions R13 and R14, it determines that the external light is incident from below (the lower side) of the RGB image. When it determines that there is external light only in regions R11 and R13, it determines that the external light is incident from the left side (the left side) of the RGB image. In this way, when the control unit 52 determines that there is external light in only 2 of the 4 divided regions, for example, it can be determined that the external light is incident from any one of the up, down, left, and right directions. In addition, the 2 regions determined to have external light can be, for example, the regions on the diagonal line in the RGB image. For example, it can be determined that there is external light only in regions R11 and R14, or only in regions R12 and R13. In this case, for example, the control unit 52 can determine that the external light is incident from two directions.

[0251] In addition, as ​ and ​ shown in ​ , when the control unit 52 determines that there is external light only in regions R11 to R13, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in regions R11, R12, and R14, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​ shown in ​ , when the control unit 52 determines that there is external light only in regions R12 to R14, it determines that the external light is incident from the lower right side of the RGB image. When it determines that there is external light only in regions R11, R13, and R14, it determines that the external light is incident from the lower left side of the RGB image. In this way, when the control unit 52 determines that there is external light in only 3 of the 4 divided regions, for example, it can be determined that the external light is incident from any one of the diagonal directions.

[0252] Next, the case where there are 9 regions (the case where the RGB image is divided into 9 regions) will be described with reference to ​ . By dividing the RGB image into 9 regions, for example, the incident direction of the external light can be judged more finely. ​ is a diagram showing examples of the determination results of the incident levels of the respective regions divided into 9 regions when the external light is incident from each direction. ​Shows the incident direction of external light when it is determined that one of the nine divided regions has external light. ​ Shows the incident direction of external light when it is determined that two of the nine divided regions have external light. ​ Shows the incident direction of external light when it is determined that three of the nine divided regions have external light. And, ​ Shows the incident direction of external light when it is determined that four of the nine divided regions have external light. ​ Shows the incident direction of external light when it is determined that five of the nine divided regions have external light.

[0253] In addition, the nine divided regions are set as regions R21 to R29. Region R21 is the upper left region in the RGB image, region R22 is the upper central region in the RGB image, region R23 is the upper right region in the RGB image, region R24 is the central left region in the RGB image, and region R25 is the central region in the RGB image. Furthermore, region R26 is the central right region in the RGB image, region R27 is the lower left region in the RGB image, region R28 is the lower central region in the RGB image, and region R29 is the lower right region in the RGB image.

[0254] As ​ And ​ Shown, when the control unit 52 determines that only region R21 has external light, it determines that the external light is incident from the upper left side of the RGB image. When it determines that only region R23 has external light, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ And ​ Shown, when the control unit 52 determines that only region R27 has external light, it determines that the external light is incident from the lower left side of the RGB image. When it determines that only region R29 has external light, it determines that the external light is incident from the lower right side of the RGB image.

[0255] As ​ And ​ Shown, when the control unit 52 determines that only region R24 has external light, it determines that the external light is incident from the left side of the RGB image. When it determines that only region R22 has external light, it determines that the external light is incident from the upper side of the RGB image. And, as ​ And ​As shown, when the control unit 52 determines that there is external light only in the region R28, it determines that the external light is incident from the lower side of the RGB image. When it determines that there is external light only in the region R26, it determines that the external light is incident from the right side of the RGB image.

[0256] ​ The case where it is determined that there is external light in two consecutive regions in the vertical direction is shown. ​ The case where it is determined that there is external light in two consecutive regions in the horizontal direction is shown.

[0257] As ​ and ​ shown, when the control unit 52 determines that there is external light only in the regions R21 and R24, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in the regions R23 and R26, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​ shown, when the control unit 52 determines that there is external light only in the regions R24 and R27, it determines that the external light is incident from the lower left side of the RGB image. When it determines that there is external light only in the regions R26 and R29, it determines that the external light is incident from the lower right side of the RGB image.

[0258] As ​ and ​ shown, when the control unit 52 determines that there is external light only in the regions R21 and R22, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in the regions R22 and R23, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​ shown, when the control unit 52 determines that there is external light only in the regions R27 and R28, it determines that the external light is incident from the lower left side of the RGB image. When it determines that there is external light only in the regions R28 and R29, it determines that the external light is incident from the lower right side of the RGB image.

[0259] In addition, the two consecutive regions may be regions including the region R25, or may be regions that are continuous in the diagonal direction (for example, the regions R23 and R25, etc.).

[0260] ​ The case where it is determined that there is external light in three regions arranged in an L shape is shown. ​ The case where it is determined that there is external light in three regions that are continuously arranged in a straight line is shown.

[0261] As ​ and ​As shown, when the control unit 52 determines that there is external light only in regions R21, R22, and R24, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in regions R22, R23, and R26, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R24, R27, and R28, it determines that the external light is incident from the lower left side of the RGB image. When it determines that there is external light only in regions R26, R28, and R29, it determines that the external light is incident from the lower right side of the RGB image. In this way, when the control unit 52 determines that there is external light only in three regions arranged in an L shape among the nine divided regions, for example, it can be determined that the external light is incident from any one of the diagonal directions.

[0262] As ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R21, R24, and R27, it determines that the external light is incident from the left side of the RGB image. When it determines that there is external light only in regions R21 to R23, it determines that the external light is incident from the upper side of the RGB image. And, as ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R27 to R29, it determines that the external light is incident from the lower side of the RGB image. When it determines that there is external light only in regions R23, R26, and R29, it determines that the external light is incident from the right side of the RGB image. In this way, when the control unit 52 determines that there is external light only in three regions in a straight line shape among the nine divided regions, for example, it can be determined that the external light is incident from any one of the up, down, left, or right directions.

[0263] ​ shows the case where it is determined that there is external light in four regions arranged in a quadrilateral shape, ​ shows the case where it is determined that there is external light in four regions arranged in an L shape.

[0264] As ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R21, R22, R24, and R25, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in regions R22, R23, R25, and R26, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​As shown, when the control unit 52 determines that there is external light only in regions R24, R25, R27, and R28, it determines that the external light is incident from the lower left side of the RGB image. When it determines that there is external light only in regions R25, R26, R28, and R29, it determines that the external light is incident from the lower right side of the RGB image. In this way, when the control unit 52 determines that there is external light only in four regions arranged in a quadrilateral shape among the nine divided regions, for example, it can be determined that the external light is incident from any direction of the oblique direction.

[0265] As ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R21 to R24, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in regions R21 to R23 and R26, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R24, R27 to R29, it determines that the external light is incident from the lower left side of the RGB image. When it determines that there is external light only in regions R26 to R29, it determines that the external light is incident from the lower right side of the RGB image. In this way, when the control unit 52 determines that there is external light only in four regions arranged in an L-shaped among the nine divided regions, for example, it can be determined that the external light is incident from any direction of the oblique direction.

[0266] ​ Shows the case where it is determined that there is external light in five regions arranged including the central region R25, ​ shows the case where it is determined that there is external light in five regions (for example, a U-shaped region) arranged without including the central region R25.

[0267] As ​ and ​ shown, when the control unit 52 determines that there is external light only in regions R21 to R25, it determines that the external light is incident from the upper left side of the RGB image. When it determines that there is external light only in regions R21 to R23, R25, and R26, it determines that the external light is incident from the upper right side of the RGB image. And, as ​ and ​As shown, when the control unit 52 determines that there is external light only in regions R21, R24, R25, R27, and R28, it determines that the external light is incident from the lower left side of the RGB image. When it determines that there is external light only in regions R23, R25, R26, R28, and R29, it determines that the external light is incident from the lower right side of the RGB image. In this way, when the control unit 52 determines that there is external light only in five regions arranged in a manner including the central region R25 among the nine divided regions, for example, it can be determined that the external light is incident from any one of the diagonal directions.

[0268] As ​ well as ​ shown, when the control unit 52 determines that there is external light only in regions R21 to R24 and R26, it determines that the external light is incident from the upper side of the RGB image. When it determines that there is external light only in regions R24, R26 to R29, it determines that the external light is incident from the lower side of the RGB image. And, as ​ well as ​ shown, when the control unit 52 determines that there is external light only in regions R21, R22, R24, R27, and R28, it determines that the external light is incident from the left side of the RGB image. When it determines that there is external light only in regions R22, R23, R26, R28, and R29, it determines that the external light is incident from the right side of the RGB image. In this way, when the control unit 52 determines that there is external light only in five U-shaped regions or five regions obtained by rotating the U shape by 90 degrees, 180 degrees, or 270 degrees among the nine divided regions, for example, it can be determined that the external light is incident from any one of the up, down, left, or right directions.

[0269] In addition, the positions of the regions determined to have external light in the above examples ​ are an example and are not limited to the above examples. Moreover, the correspondence between the regions determined to have external light and the incident direction of the external light in the above examples ​ is also an example and is not limited to the above. For example, based on at least one of the number and position of the regions determined to have external light, the incident direction of the external light can be determined to be two or more.

[0270] In addition, in ​ it is possible to determine the direction in which the external light is not incident based on the determination results of the presence or absence of external light for each of multiple regions. Such a direction in which the external light is not incident can be used, for example, in the determination of countermeasure information described later.

[0271] Refer again to ​, next, the portable terminal 50 outputs countermeasure information corresponding to the incident direction (S26). For example, the control unit 52 may determine countermeasure information including countermeasures corresponding to the incident direction, and send the determined countermeasure information to an external device. Also, the display control unit 53 may cause the countermeasure information determined by the control unit 52 to be displayed on the touch screen 54.

[0272] Here, regarding the determined countermeasure information, refer to ​ for description. ​ is a diagram showing an example of each piece of countermeasure information related to this modified example. ​ The second countermeasure information shown includes countermeasures with a greater burden on the user to execute than ​ the first countermeasure information shown, ​ The third countermeasure information shown includes countermeasures with a greater burden on the user to execute than ​ the second countermeasure information shown, but is not limited thereto. In addition, examples of a large execution burden can include examples where the body parts of the user's actions increase, and examples where the movement range of the user himself / herself becomes wider (for example, including changing from a countermeasure where the user himself / herself does not move to a countermeasure where the user himself / herself moves).

[0273] In addition, ​ shows countermeasures in the case where external light is incident from above (the upward direction), ​ shows countermeasures in the case where external light is incident from the upper right direction, ​ shows countermeasures in the case where the area determined to have external light is large.

[0274] ​ shows a display example of the first countermeasure information including wearing a light shield or tilting the head downward as a countermeasure. Tilting the head downward is equivalent to turning the face in the direction opposite to the incident direction of the external light. In this way, countermeasures can be proposed to turn the face in an appropriate direction corresponding to the incident direction of the external light. Also, information indicating that the nursing state cannot be correctly evaluated can be displayed. Displaying this information means, for example, notifying the possibility of misdetecting the tartar area due to external light entering the oral cavity. Accordingly, the user can be urged to execute the first countermeasure information. In addition, tilting the head downward is an example of information regarding the incident direction of the external light.

[0275] ​ shows a display example of the second countermeasure information including covering the mouth from the upper right direction with the hand that is not holding the toothbrush in the state where the head is tilted downward. ​ The countermeasures shown in addition to ​ the countermeasure of tilting the head downward shown, also include countermeasures of blocking the incident direction of the external light with the hand. In this way, countermeasures can be proposed to block the appropriate direction corresponding to the incident direction of the external light with the hand.

[0276] For example, when the control unit 52 determines that external light exists in one or two of the four regions into which the RGB image is divided (refer to ​ ), or when it determines that external light exists in one to four of the nine regions into which the RGB image is divided (refer to ​ ), it can decide to output the first countermeasure information or the second countermeasure information. For example, when the control unit 52 determines that external light exists in one of the four regions into which the RGB image is divided (refer to ​ ), or when it determines that external light exists in one or two of the nine regions into which the RGB image is divided (refer to ​ ), it can decide to output the first countermeasure information. Also, for example, when the control unit 52 determines that external light exists in two of the four regions into which the RGB image is divided (refer to ​ ), or when it determines that external light exists in three or four of the nine regions into which the RGB image is divided (refer to ​ ), it can decide to output the second countermeasure information.

[0277] For example, when the control unit 52 determines that external light exists in one or two of the four regions into which the RGB image is divided, or when it determines that external light exists in one to four of the nine regions into which the RGB image is divided, if the determination of the existence of external light is the first determination, it can decide on the first countermeasure information. If it is the second or subsequent time (in other words, when countermeasure information was output at the first time and it is still determined that there is external light after the countermeasure has been executed), then it can decide on the second countermeasure information. In this way, the control unit 52 can decide on the output countermeasure information based on whether external light is determined to exist after the countermeasure information has been output.

[0278] ​ A display example of the third countermeasure information is shown, which includes turning off the room lighting or moving the usage location of the intraoral camera as countermeasures. The first countermeasure information and the second countermeasure information are countermeasures that can be executed even if the user does not move themselves. However, the third countermeasure information includes the user's own movement or the user's control of devices outside the intraoral camera system, and includes countermeasures that impose a greater burden on the user to execute than the first countermeasure information and the second countermeasure information.

[0279] For example, when the control unit 52 determines that external light exists in three or more of the four regions into which the RGB image is divided (refer to ​ ), or when it determines that external light exists in five or more of the nine regions into which the RGB image is divided (refer to Figures 21A to 21H ), it can decide on the third countermeasure information.

[0280] In addition, for example, when the control unit 52 determines that external light exists in one or two of the four regions into which the RGB image is divided, or when it determines that external light exists in one to four of the nine regions into which the RGB image is divided, if this determination of the existence of external light is made after the second time, it can be determined as the third countermeasure information. For example, when the control unit 52 outputs the second countermeasure information and still determines that external light exists after the countermeasure has been executed, it can be determined as the third countermeasure information. In this way, the control unit 52 can determine the countermeasure information based on whether external light is determined to exist after outputting the countermeasure information.

[0281] In addition, for example, the control unit 52 can determine the countermeasure information based on the position of the region where the incident level exceeds the specified value (in other words, the region where external light is determined to exist). For example, when there is only one direction of the incident direction of external light, the control unit 52 can determine it as the first countermeasure information or the second countermeasure information, and when the incident direction of external light is two or more directions, it can be determined as the third countermeasure information.

[0282] In addition, the portable terminal 50 can determine the countermeasure information corresponding to at least one of the number of regions where the incident level exceeds the specified value among the multiple regions and the position (position on the RGB image) of the regions where the incident level exceeds the specified value among the multiple regions, and display the countermeasure information.

[0283] In addition, at least one of the first countermeasure information to the third countermeasure information may include information about the direction of external light incidence. The information about the direction of external light incidence can be, for example, information indicating from which direction the external light is incident, or information indicating the countermeasure based on the incident direction of the external light.

[0284] Refer again to Figure 14 , the control unit 52 determines whether the user has executed the countermeasure shown in the output countermeasure information (S27). In this modified example, the control unit 52 functions as a determination unit that determines whether the user has responded to the output countermeasure information. The control unit 52 can determine that the user has executed the countermeasure information in the following cases: for example, when an operation by the user is received by the touch screen 54, and this operation shows that the countermeasure included in the countermeasure information has been executed, or when information about an image of the user taken by an external imaging device is obtained via the acquisition unit 51 and this information shows that the countermeasure shown in the countermeasure information has been executed, or when a specified time has elapsed after the countermeasure information is output.

[0285] Next, when the portable terminal 50 determines that the user has executed a countermeasure (Yes in S27), it determines whether the external light is blocked (S28). Specifically, the portable terminal 50 executes the processes of steps S21 to S24 again. For example, the portable terminal 50 can obtain the incident level after the countermeasure for each of a plurality of regions in the following second RGB image, which is an image obtained by photographing teeth and dental plaque that fluoresce when irradiated with light in a wavelength range including blue light after it is determined that the user has responded to the countermeasure information.

[0286] In step S24 that is executed again, when the control unit 52 determines that there is no external light for each of the plurality of regions, it determines Yes in step S28, and when it determines that there is external light in at least one region for each of the plurality of regions, it determines No in step S28.

[0287] When the external light is not blocked (No in S28), the control unit 52 outputs other countermeasure information different from the countermeasure information output in step S26 (S29). The other countermeasure information is, for example, information corresponding to at least one of the number of regions in the plurality of regions where the incident level after the countermeasure exceeds a specified value and the positions of the regions in the plurality of regions.

[0288] When the control unit 52 outputs the first countermeasure information in step S26, it can output the second countermeasure information in step S29, and when it outputs the second countermeasure information in step S26, it can output the third countermeasure information in step S29. For example, the control unit 52 can determine the second countermeasure information different from the first countermeasure information based on the incident level after the countermeasure for each of the plurality of regions, and display the second countermeasure information.

[0289] In addition, the control unit 52 can determine other countermeasure information based on the difference between at least one of the number of areas where external light is determined in step S24 and the position of the area (in other words, the incident direction of the external light), and at least one of the number of areas where external light is determined in step S28 and the position of the area (in other words, the incident direction of the external light). For example, when at least one of the following conditions is met: the first countermeasure information is output in step S26 and the number of areas where external light is determined in step S28 is greater than or equal to the number of areas where external light is determined in step S24, and the incident direction of the external light in step S28 is the same as that in step S24, the third countermeasure information is output as other countermeasure information. When at least one of the following conditions is met: the number of areas where external light is determined in step S28 is less than the number of areas where external light is determined in step S24, and the incident direction of the external light in step S28 and step S24 has changed, the second countermeasure information is output as other countermeasure information.

[0290] In addition, the method for determining other countermeasure information is not limited to the above method, and other methods can also be used to determine it. For example, other countermeasure information can be determined based on a table that correlates the countermeasure information that has been output, at least one of both the number and position of the areas where external light is determined before and after the output of the countermeasure information, and other countermeasure information.

[0291] After step S29 is executed and when it is determined as "No" in step S27, the process returns to step S27 and continues.

[0292] In addition, when the external light is blocked (Yes in S28), the control unit 52 performs the processing after step S30.

[0293] The processing of step S30 is the same as the processing shown in step S13 of Figure 8 and the processing shown in step S31 is the same as the processing shown in step S16 of Figure 8 , so the description is omitted.

[0294] (Other Embodiments)

[0295] The intraoral camera system and the like related to one or more aspects of the present disclosure have been described above based on the embodiments and the like. However, the present disclosure is not limited to these embodiments and the like. Within the scope not exceeding the gist of the present disclosure, the various modifications that can be conceived by those skilled in the art and executed on the present embodiment, as well as the modes constructed by combining the constituent elements in different embodiments, can be included in the scope of the present disclosure.

[0296] For example, in the above-described embodiments and the like, the portable terminal 50 is given as an example of the user's information terminal. However, the information terminal may also be a desktop information terminal.

[0297] In addition, the intraoral camera 10 in the above-described embodiments and the like can be mounted on, for example, an intraoral cleaner or the like.

[0298] In addition, in the above-described embodiments and the like, an example of the notification method for false detection is display. However, the notification method is not limited to display. For example, it can be notified by emitting sound or light (e.g., warning light). In this case, the portable terminal 50 may include a sound-emitting device such as a speaker or a light-emitting device such as an LED.

[0299] In addition, in the above-described embodiments and the like, an example is described in which image processing such as AWB is performed by the image processing unit 32 of the intraoral camera 10. However, it can also be performed by, for example, the control unit 52 of the portable terminal 50 or the like. And a part or all of the processing performed by the control unit 52 of the portable terminal 50 can also be performed by the control unit 33 of the intraoral camera 10 or the like.

[0300] In addition, in the modification example 2 of the above-described embodiment, an example is described in which countermeasure information is determined and displayed based on the determination result of the presence or absence of external light in each of a plurality of regions. However, it is not limited thereto, and countermeasure information for suppressing the influence of external light can be displayed without dividing the RGB image into a plurality of regions. As described in the embodiment, for example, the control unit 52 can display countermeasure information corresponding to the incident level calculated from the entire region of the RGB image. The control unit 52 can, for example, determine the first countermeasure information when the incident level is less than the first level, determine the second countermeasure information when the incident level is equal to or greater than the first level and less than the second level higher than the first level, and determine the third countermeasure information when the incident level is equal to or greater than the second level.

[0301] In addition, the above description has given an example in which the portable terminal 50 according to the above-described embodiment not only changes the display form of the tartar detection result according to the incident level of the external light incident on the photographing area of the RGB image, but also displays countermeasure information corresponding to the incident level for suppressing the influence of the external light. However, the present disclosure is not limited thereto, and it can be implemented by an image processing method or the like that can display countermeasure information corresponding to the incident level for suppressing the influence of the external light without performing the process of changing the display form of the tartar detection result. For example, the present disclosure can be implemented as an image processing method or the like as follows. The image processing method is an image processing method for displaying a tartar area based on an RGB image obtained by photographing teeth and tartar that have undergone a fluorescence reaction when light in a wavelength range including blue light is irradiated onto the teeth. In this image processing method, the incident level of the external light incident on the photographing area of the RGB image is obtained, and countermeasure information corresponding to the incident level for suppressing the influence of the external light is displayed. Accordingly, the external light incident on the oral cavity can be efficiently suppressed.

[0302] In addition, in the modification example 2 of the above-described embodiment, an example of displaying countermeasure information has been described. However, for example, an RGB image (for example, an RGB image clearly showing an area with external light) reflecting the determination result of the presence or absence of external light can be further displayed. In this case, the RGB image can be displayed as a horizontally flipped image. Accordingly, the incident direction of the external light in the image seen by the user is the same as the actual incident direction, so the user can easily and intuitively grasp the incident direction.

[0303] In addition, in the above-described embodiment and the like, each component can be configured by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0304] In addition, the order in which each step in the flowchart is executed is an example shown for specifically explaining the present disclosure, and can also be an order other than the above. For example, Figure 8 the process of step S14 shown can be executed between steps S12 and S13. Moreover, when the result of step S14 is "No", since there is incident external light, the tartar detection process of step S13 can be omitted. And a part of the above steps can be executed simultaneously (in parallel) with other steps, or a part of the above steps can be not executed.

[0305] The division of functional blocks in the block diagram is merely an example. Multiple functional blocks can be implemented as a single functional block, or a single functional block can be divided into multiple blocks, or some functions can be moved to other functional blocks. Furthermore, the functions of multiple functional blocks with similar functions can be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

[0306] In addition, the portable terminal 50 involved in the above-mentioned embodiments and the like can be implemented as a single device or by multiple devices. In the case where the portable terminal 50 is implemented by multiple devices, the various components of the portable terminal 50 can be arbitrarily distributed to multiple devices. For example, at least part of the functions of the portable terminal 50 can be implemented by the intraoral camera 10 (for example, the signal processing unit 30). In the case where the portable terminal 50 is implemented by multiple devices, the communication method between these multiple devices is not particularly limited and can be wireless communication or wired communication. Furthermore, wireless communication and wired communication can be combined between devices.

[0307] In addition, each component described in the above embodiments can be implemented as software, typically as an integrated circuit, i.e., LSI. These can be made into a single chip, or part or all of them can be included in a single chip. Here, it is called LSI, but depending on the degree of integration, it is sometimes also called IC, system LSI, super LSI, or extra-large LSI. Moreover, the method of integrated circuitization is not limited to LSI, and can be implemented using a dedicated circuit (a dedicated circuit that executes a dedicated program) or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconstruct the connection or setting of the circuit unit inside the LSI. Furthermore, with the advancement of semiconductor technology or other derived technologies, when an integrated circuit technology that can replace LSI emerges, of course, this technology can be used to integrate the components.

[0308] A system LSI is a highly multifunctional LSI that integrates multiple processing units on a single chip. Specifically, it is a computer system consisting of a microprocessor, ROM, RAM, and other components. The ROM stores computer programs. The microprocessor operates according to the computer program, allowing the system LSI to achieve its functions.

[0309] In addition, one embodiment of the present disclosure may also be to enable a computer to execute Figure 8 or Figure 14 The image processing method shown includes a computer program having characteristic steps.

[0310] In addition, for example, the program can be a program for causing a computer to execute. Further, one aspect of the present disclosure can also be a non-transitory computer-readable recording medium storing such a program. For example, the recording medium storing such a program can be distributed or circulated. For example, the distributed program is installed in a device having another processor, and the processor executes the program, whereby the device can execute the above-described various processes.

[0311] Industrial Applicability

[0312] The present disclosure can be applied to an intraoral camera system.

[0313] Description of Reference Numerals

[0314] 10 Intraoral camera

[0315] 10a Head

[0316] 10b Handle portion

[0317] 10c Neck

[0318] 20 Hardware portion

[0319] 21 Imaging portion

[0320] 22 Sensor portion

[0321] 23 Lighting portion

[0322] 23A First LED

[0323] 23B Second LED

[0324] 23C Third LED

[0325] 23D Fourth LED

[0326] 24 Operation portion

[0327] 30 Signal processing portion

[0328] 31 Camera control portion

[0329] 32 Image processing portion

[0330] 33, 52 Control portion

[0331] 34 Lighting control portion

[0332] 35 Memory portion

[0333] 40 Communication portion

[0334] 50 Portable terminal

[0335] 51 Acquisition portion

[0336] 53 Display control portion

[0337] 54 touch screen

[0338] Regions of R1, R2, R3, R4, R11, R12, R13, R14, R21, R22, R23, R24, R25, R26, R27, R28, R29

Claims

1. An image processing method is an image processing method for displaying a dental calculus area based on an RGB image. The RGB image is an image obtained by photographing the teeth and dental calculus that have undergone a fluorescence reaction when light in a wavelength range containing blue light is irradiated on the teeth. In the image processing method, obtain the incident level of external light incident on the photographing area of the first RGB image, perform a dental calculus detection process based on the first RGB image, make the display form of the dental calculus detection result different according to the incident level of the external light incident on the photographing area of the first RGB image.

2. The image processing method according to claim 1, when the incident level is not below a specified value, display an image obtained by overlapping an image showing the dental calculus detection result with a display showing the incidence of external light.

3. The image processing method according to claim 2, when the incident level is below the specified value, display an image obtained by overlapping an image showing the dental calculus detection result with a display showing no incidence of external light.

4. The image processing method according to claim 1, when the incident level is not below a specified value, prohibit the display of the dental calculus detection result.

5. The image processing method according to any one of claims 1 to 4, the incident level is obtained based on the detection result of a detector that detects external light incident on the photographing area of the first RGB image.

6. The image processing method according to any one of claims 1 to 4, the incident level is calculated based on the number of first red pixels having a red pixel value equal to or greater than a first pixel value in the first RGB image.

7. The image processing method according to claim 6, the incident level includes a first incident level, the number of first red pixels includes the number of first pixels of red pixels having each pixel value equal to or greater than the first pixel value, the number of green pixels having a green pixel value equal to or greater than the first pixel value includes the number of second pixels of green pixels having each pixel value equal to or greater than the first pixel value, in the image processing method, for each pixel value equal to or greater than the first pixel value, subtract the number of second pixels from the number of first pixels, calculate the first incident level based on the first cumulative value obtained by accumulating the first subtraction value, which is positive after subtracting the number of second pixels from the number of first pixels, make the display form of the dental calculus detection result different according to whether the first incident level is below a first specified value.

8. The image processing method according to claim 6, the incident level includes a first incident level, the number of first red pixels includes the number of first pixels of red pixels having a pixel value equal to or greater than the first pixel value, the number of green pixels having a green pixel value lower than the first pixel value and equal to or lower than a fourth pixel value includes the number of second pixels of green pixels having a pixel value equal to or lower than the fourth pixel value, in the image processing method, Subtract the second pixel count from the first pixel count. When the first subtraction value obtained by subtracting the second pixel count from the first pixel count is positive, calculate the first incident level based on the first subtraction value. Depending on whether the first incident level is below a first specified value, change the display form of the calculus detection result.

9. The image processing method according to claim 7. The incident level includes a second incident level. The second red pixel count having a red pixel value greater than or equal to a second pixel value that is higher than the first pixel value includes a third pixel count of red pixels having each pixel value greater than or equal to the second pixel value. In the image processing method. For each pixel value greater than or equal to the second pixel value, subtract the second pixel count from the third pixel count. Calculate the second incident level based on a second cumulative value obtained by accumulating the second subtraction value, which is positive, from the third pixel count minus the second pixel count. Depending on whether the second incident level is below a second specified value, change the display form of the calculus detection result.

10. The image processing method according to claim 7. The incident level includes a third incident level. Generate a V image composed of V values obtained by performing HSV transformation on each pixel value of the first RGB image. Calculate the third incident level based on the cumulative value of the number of pixels having a V value greater than or equal to a third pixel value. Furthermore, depending on whether the third incident level is below a third specified value, change the display form of the calculus detection result.

11. The image processing method according to claim 10. When the first incident level is below the first specified value and above a fourth specified value lower than the first specified value, and when the third incident level is below the third specified value and above a fifth specified value lower than the third specified value, display an image obtained by overlapping a display prompting shading on the image showing the calculus detection result.

12. The image processing method according to claim 6. Generate a V image composed of V values obtained by performing HSV transformation on each pixel value of the first RGB image. Calculate the incident level based on the cumulative value of the number of pixels having a V value greater than or equal to a first pixel value.

13. The image processing method according to any one of claims 2 to 4. The specified value is set according to the illumination environment of the space where the first RGB image is captured.

14. The image processing method according to any one of claims 2 to 4. Divide the first RGB image into a plurality of regions. For each of the divided plurality of regions, obtain the incident level of external light incident on the region. Depending on the judgment result of whether the incident level of each of the plurality of regions is below the specified value, change the display form of the calculus detection result.

15. The image processing method according to any one of claims 2 to 4. Display countermeasure information for suppressing the influence of the external light corresponding to the incident level.

16. The image processing method according to claim 15, divide the first RGB image into a plurality of regions, for each of the divided plurality of regions, obtain the incident level of the external light incident on the region, display countermeasure information corresponding to at least one of the number of regions in the plurality of regions where the incident level exceeds the specified value and the position of the region in the plurality of regions.

17. The image processing method according to claim 16, further includes a determination unit that determines whether the user has responded to the countermeasure information, obtain the incident level after countermeasures for each region of the plurality of regions in the second RGB image, where the second RGB image is an image obtained by photographing the teeth and dental calculus that fluoresce when the teeth are irradiated with light in a wavelength range including blue light after it is determined that the user has responded to the countermeasure information, display other countermeasure information different from the countermeasure information according to the incident level after countermeasures for each region of the plurality of regions.

18. The image processing method according to claim 17, the other countermeasure information is information corresponding to at least one of the number of regions in the plurality of regions where the incident level after countermeasures exceeds the specified value and the position of the region in the plurality of regions.

19. The image processing method according to claim 16, judge the direction of incidence of the external light according to the positional relationship of the regions in the plurality of regions where the incident level exceeds the specified value, the countermeasure information includes information about the direction of incidence of the external light.

20. An image processing system for displaying a dental calculus region based on an RGB image, where the RGB image is an image obtained by photographing the teeth and dental calculus that fluoresce when the teeth are irradiated with light in a wavelength range including blue light, the image processing system includes: an obtaining unit that obtains the incident level of the external light incident on the photographing region of the RGB image; a dental calculus detection unit that performs dental calculus detection processing based on the RGB image; and a display control unit that makes the display form of the dental calculus detection result different according to the incident level of the external light incident on the photographing region of the RGB image.

21. A program for causing a computer to execute the image processing method according to any one of claims 1 to 4.

Images