Image processing method, image processing device, and program

Abstract

This image processing method is for displaying a dental plaque region on the basis of an image obtained by photographing teeth and dental plaque undergoing a fluorescence reaction as a result of being irradiated with light including a wavelength band of blue light, said method comprising: detecting a natural tooth region from the region of the teeth on the basis of the image (S201-S206); detecting plaque regions on the basis of the image (S207); and displaying a plaque region exclusively in the natural tooth region (S105). For example, the image processing method may involve, in detecting a natural tooth region, detecting a first region that is (i) a region, among all pixel regions of the image, having a luminance value equal to or greater than a predetermined first threshold or (ii) a region, among all pixel regions of the image, having a green pixel value equal to or greater than a predetermined second threshold, and detecting a natural tooth region on the basis of the first region.

Description

Image processing method, image processing device and program

[0001] The present disclosure relates to an image processing method, an image processing device, and a program.

[0002] Devices for detecting dental plaque based on images of teeth in the oral cavity are known. For example, Patent Document 1 discloses a technique for storing care areas in the oral cavity and the amount of plaque detected corresponding to the care areas. Patent Document 2 discloses a technique for correcting tooth images.

[0003] JP 2013-042906 A JP 2009-37414 A

[0004] In such a device for detecting dental plaque, it is desirable to improve the accuracy of detecting dental plaque.

[0005] Therefore, the present disclosure provides an image processing method or image processing device that can improve the accuracy of plaque detection.

[0006] An image processing method according to one aspect of the present disclosure is an image processing method for displaying a plaque region based on an image of a tooth and plaque that are fluorescently reacting when the tooth is irradiated with light including a wavelength range of blue light, the method detecting a natural tooth region within the tooth region based on the image, detecting a plaque region based on the image, and displaying the plaque region limited to the natural tooth region.

[0007] The present disclosure can provide an image processing method or image processing device that can improve the accuracy of plaque detection.

[0008] FIG. 1 is a perspective view of an intraoral camera in an intraoral camera system according to an embodiment. FIG. 2 is a cross-sectional view schematically illustrating a photographing optical system incorporated in the intraoral camera in the intraoral camera system according to an embodiment. FIG. 3 is a schematic configuration diagram of the intraoral camera system according to an embodiment. FIG. 4 is a diagram illustrating an operation flow of the intraoral camera system according to an embodiment. FIG. 5 is a functional block diagram of a mobile terminal according to an embodiment. FIG. 6 is a flowchart of image generation processing according to an embodiment. FIG. 7 is a diagram illustrating an example of an RGB image (fourth RGB image) according to an embodiment. FIG. 8 is a diagram illustrating an example of a first natural tooth region according to an embodiment. FIG. 9 is a diagram illustrating an example of a gingival region according to an embodiment. FIG. 10 is a diagram illustrating an example of a second natural tooth region according to an embodiment. FIG. 11 is a diagram illustrating an example of a first plaque region according to an embodiment. FIG. 12 is a diagram illustrating an example of a mask image according to an embodiment. FIG. 13 is a diagram illustrating an example of a second tooth region according to an embodiment. FIG. 14 is a diagram illustrating an example of a display image according to an embodiment.

[0009] When blue light is irradiated into the oral cavity to detect dental plaque, the blue light is reflected by the surface of prosthetic appliances made of gold or silver, etc. This can lead to the red wavelengths contained in the blue light being mistakenly detected as red fluorescence from dental plaque. Furthermore, when a user photographs a tooth surface with a handheld intraoral camera, it is difficult to distinguish between the red fluorescence of dental plaque and the light reflected by the surface of the prosthetic appliance.

[0010] In view of this problem, the present disclosure detects natural tooth regions excluding artificial tooth regions such as prostheses from among tooth regions, and enables plaque detection only in the natural tooth regions, thereby preventing erroneous detection of plaque caused by prostheses, etc.

[0011] An image processing method according to one aspect of the present disclosure is an image processing method for displaying a plaque region based on an image of a tooth and plaque that are fluorescently reacting when the tooth is irradiated with light including a wavelength range of blue light, the method detecting a natural tooth region within the tooth region based on the image, detecting a plaque region based on the image, and displaying the plaque region limited to the natural tooth region.

[0012] This makes it possible to prevent the prosthesis or the like from being mistakenly detected as dental plaque, thereby improving the accuracy of dental plaque detection.

[0013] For example, the image processing method may detect the natural tooth region by detecting a first region that is (i) a region in the entire pixel region of the image where the brightness value is equal to or greater than a predetermined first threshold, or (ii) a region in the entire pixel region of the image where the green pixel value is equal to or greater than a predetermined second threshold, and then detect the natural tooth region based on the first region.

[0014] According to this, the image processing method can accurately detect natural tooth regions using brightness values ​​or green pixel values.

[0015] For example, in detecting the natural tooth region, the image processing method may detect, within the first region, a region in which the differences between the green pixel value, the red pixel value, and the blue pixel value are less than a predetermined third threshold, as the natural tooth region.

[0016] According to this, the image processing method can accurately detect natural tooth regions using the differences between green pixel values, red pixel values, and blue pixel values.

[0017] For example, in detecting the natural tooth region, the image processing method may detect a gum region based on the hue of the image, and detect an area obtained by excluding the gum region from the first area as the natural tooth region.

[0018] According to this, the image processing method can prevent the gum area from being mistakenly determined as a natural tooth area, thereby improving the accuracy of detecting the natural tooth area.

[0019] For example, in the detection of the natural tooth region, the image processing method may detect, as the natural tooth region, a region obtained by excluding from the first region any region whose area is smaller than a predetermined area.

[0020] According to this, the image processing method can prevent high-brightness areas caused by reflected light or the like other than natural tooth areas from being mistakenly determined to be natural tooth areas, thereby improving the accuracy of detecting natural tooth areas.

[0021] For example, in the detection of the natural tooth region, the image processing method may detect, as the natural tooth region, a region obtained by extending the boundary of the first region outward by a predetermined amount.

[0022] According to this, the image processing method can prevent the plaque region from being excessively displayed by expanding the natural tooth region.

[0023] Furthermore, an image processing device according to one aspect of the present disclosure is an image processing device that displays a plaque region based on an image of a tooth and plaque that are fluorescently reacting when the tooth is irradiated with light including a wavelength range of blue light, and includes: a natural tooth region detection unit that detects a natural tooth region within the tooth region based on the image; a plaque region detection unit that detects a plaque region based on the image; and a display unit that displays the plaque region limited to the natural tooth region.

[0024] This allows the image processing device to prevent erroneous detection of a prosthesis or the like as dental plaque, thereby improving the accuracy of detecting dental plaque.

[0025] A program according to one aspect of the present disclosure is a program for causing a computer to execute the image processing method.

[0026] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0027] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0028] The inventors have provided the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0029] 1 is a perspective view of an intraoral camera in an intraoral camera system according to the present embodiment. As shown in Fig. 1, the intraoral camera 10 has a toothbrush-shaped housing that can be handled with one hand, and the housing includes a head portion 10a that is placed in the user's oral cavity when photographing the dentition, a handle portion 10b that the user grips, and a neck portion 10c that connects the head portion 10a and the handle portion 10b.

[0030] 2 is a cross-sectional view showing the imaging optical system 12 incorporated in the intraoral camera 10. In this embodiment, the imaging optical system 12 of the intraoral camera 10 is incorporated in the head portion 10a and the neck portion 10c as shown in FIG. 2. The imaging optical system 12 includes an image sensor 14 and a lens 16 arranged on its optical axis LA.

[0031] The imaging element 14 is an imaging device such as a C-MOS (Complementary Metal-Oxide-Semiconductor) sensor or a CCD (Charge Coupled Device) element, and an image of the tooth D is formed by the lens 16. The imaging element 14 outputs a signal (image data) corresponding to the formed image to the outside.

[0032] The lens 16 is, for example, a condenser lens, and forms an image of the tooth D incident thereon on the imaging element 14. The lens 16 may be a single lens or a lens group made up of a plurality of lenses.

[0033] In this embodiment, the photographing optical system 12 further includes a mirror 18 that reflects the image of the tooth D toward the lens 16, a blue light cut filter (blue blocking element) 20 arranged between the mirror 18 and the lens 16, and an aperture 24 arranged between the lens 16 and the image sensor 14.

[0034] The mirror 18 is disposed on the optical axis LA of the photographing optical system 12 so as to reflect the image of the tooth D that has passed through the entrance 12 a of the photographing optical system 12 toward the lens 16 .

[0035] The blue light cut filter 20 is a filter that cuts out light components of blue wavelengths contained in light incident on the image sensor 14. When light including a blue wavelength range is irradiated onto teeth to detect dental plaque, if the light including the blue wavelength range is increased to enhance the excitation fluorescence of the plaque, the entire first RGB image will appear blue. In this state, blue pixel values ​​become dominant compared to red and green pixel values, and the effect of making plaque areas easier to distinguish by performing image processing (exposure control processing and white balance adjustment processing) described below may be reduced. To address this issue, the blue light cut filter 20 cuts out light including the blue wavelength range from the light before it enters the image sensor 14.

[0036] The diaphragm 24 is a plate-like member with a through-hole on the optical axis LA of the imaging optical system 12, and realizes a deep focal depth, which allows the depth direction of the oral cavity to be brought into focus, thereby obtaining an image of the row of teeth with clear contours.

[0037] The intraoral camera 10 is also equipped with multiple LEDs 26A-26D as lighting devices that irradiate light onto the tooth D to be photographed during photography. The multiple LEDs 26A-26D are, for example, blue light-emitting diodes (LEDs). As shown in FIG. 1 , in this embodiment, the multiple LEDs 26A-26D are arranged to surround the entrance 12a. To prevent the gums G or the like from coming into contact with the LEDs 26A-26D and resulting in a lack of illumination light, a translucent cover 28 that covers the multiple LEDs 26A-26D and the entrance 12a is provided on the head unit 10a. Some of the multiple LEDs 26A-26D may be white LEDs. Using white LEDs for some of the multiple LEDs 26A-26D can brighten the first RGB image and improve the balance of blue pixel values ​​relative to red and green pixel values.

[0038] Furthermore, in this embodiment, the intraoral camera 10 has a composition adjustment mechanism 30 and a focus adjustment mechanism 32 as shown in FIG.

[0039] The composition adjustment mechanism 30 is composed of a housing 34 that holds the image sensor 14 and the lens 16, and an actuator 36 that moves the housing 34 in the direction of extension of the optical axis LA. The actuator 36 adjusts the position of the housing 34 to adjust the angle of view, i.e., the size of the row of teeth imaged on the image sensor 14. The composition adjustment mechanism 30 automatically adjusts the position of the housing 34 so that, for example, one entire tooth is captured in the captured image. The composition adjustment mechanism 30 also adjusts the position of the housing 34 based on a user's operation so that the angle of view desired by the user is achieved.

[0040] The focus adjustment mechanism 32 is held within the housing 34 of the composition adjustment mechanism 30 and is composed of a lens holder 38 that holds the lens 16 and an actuator 40 that moves the lens holder 38 in the direction of extension of the optical axis LA. The actuator 40 adjusts the relative position of the lens holder 38 with respect to the image sensor 14, thereby adjusting the focus, i.e., the focal point. The focus adjustment mechanism 32 automatically adjusts the position of the lens holder 38 so that, for example, a tooth located at the center of the captured image is in focus. The focus adjustment mechanism 32 also adjusts the position of the lens holder 38 based on a user's operation.

[0041] In addition, the components of the imaging optical system 12 except for the mirror 18 may be provided on the handle portion 10 b of the intraoral camera 10 .

[0042] The image output by the imaging device 14 is an RGB image in which each of the multiple pixels that make up the image has RGB sub-pixels.

[0043] The intraoral camera 10 is also equipped with multiple LEDs 26A-26D as lighting devices that irradiate the teeth to be photographed during photography. The multiple LEDs 26A-26D are, for example, blue LEDs that irradiate blue light having a peak wavelength of 405 nm. Note that the multiple LEDs 26A-26D are not limited to blue LEDs and may be any light source that irradiates light including the wavelength range of blue light.

[0044] 3 is a schematic diagram of the intraoral camera system according to the present embodiment. As shown in FIG. 3, the intraoral camera system according to the present embodiment is generally configured to capture an image of the row of teeth using an intraoral camera 10 and perform image processing on the captured image.

[0045] 3 , the intraoral camera system includes an intraoral camera 10, a mobile terminal 70, and a cloud server 80. The mobile terminal 70 is, for example, a smartphone or tablet terminal capable of wireless communication. The mobile terminal 70 includes, as an input device and an output device, a touch screen 72 capable of displaying, for example, a dentition image. The mobile terminal 70 functions as a user interface for the intraoral camera system.

[0046] The cloud server 80 is a server that can communicate with the mobile terminal 70 via the Internet or the like, and provides the mobile terminal 70 with an application for using the intraoral camera 10. For example, a user downloads the application from the cloud server 80 and installs it on the mobile terminal 70. The cloud server 80 also acquires dentition images captured by the intraoral camera 10 via the mobile terminal 70.

[0047] The intraoral camera 10 includes a central control unit 50 as the main part for controlling the system, an LED control unit 54 that controls the multiple LEDs 26A to 26D, a lens driver 56 that controls the actuator 36 of the composition adjustment mechanism 30 and the actuator 40 of the focus adjustment mechanism 32, and a position sensor 90.

[0048] The intraoral camera 10 also has a wireless communication module 58 that communicates wirelessly with the mobile terminal 70, and a power supply control unit 60 that supplies power to the central control unit 50 and other components.

[0049] The central control unit 50 of the intraoral camera 10 is mounted, for example, on the handle portion 10b of the intraoral camera 10. For example, the central control unit 50 includes a controller 62 such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) that executes various processes described below, and a memory 64 such as a RAM (Random Access Memory) or a ROM (Read Only Memory) that stores programs for causing the controller 62 to execute various processes. In addition to the programs, the memory 64 also stores a row-of-teeth image (image data) captured by the image sensor 14 and various setting data. The row-of-teeth image captured by the image sensor 14 is an example of a first RGB image.

[0050] The controller 62 transmits the row-of-teeth image output from the imaging element 14 to the mobile terminal 70 via the wireless communication module 58. The mobile terminal 70 displays the transmitted row-of-teeth image on the touch screen 72, thereby presenting the row-of-teeth image to the user.

[0051] The LED control unit 54 is mounted, for example, on the handle portion 10b of the intraoral camera 10, and controls the turning on and off of the multiple LEDs 26A to 26D based on a control signal from the controller 62. The LED control unit 54 is configured, for example, by a circuit. For example, when a user performs an operation on the touch screen 72 of the mobile terminal 70 to activate the intraoral camera 10, a corresponding signal is transmitted from the mobile terminal 70 to the controller 62 via the wireless communication module 58. Based on the received signal, the controller 62 transmits a control signal to the LED control unit 54 to turn on the multiple LEDs 26A to 26D.

[0052] The lens driver 56 is mounted on, for example, the handle portion 10b of the intraoral camera 10, and controls the actuator 36 of the composition adjustment mechanism 30 and the actuator 40 of the focus adjustment mechanism 32 based on a control signal from a controller 62 of the central control unit 50. The lens driver 56 is composed of, for example, a circuit. For example, when a user performs an operation related to composition adjustment or focus adjustment on the touch screen 72 of the mobile terminal 70, a corresponding signal is transmitted from the mobile terminal 70 to the central control unit 50 via the wireless communication module 58. Based on the received signal, the controller 62 of the central control unit 50 transmits a control signal to the lens driver 56 to adjust the composition or focus. Furthermore, for example, the controller 62 calculates a control amount of the actuator 36 or 40 required for composition adjustment or focus adjustment based on the row-of-teeth image from the imaging element 14, and transmits a control signal corresponding to the calculated control amount to the lens driver 56.

[0053] The wireless communication module 58 is mounted on, for example, the handle portion 10b of the intraoral camera 10, and performs wireless communication with the mobile terminal 70 based on a control signal from the controller 62. The wireless communication module 58 performs wireless communication with the mobile terminal 70 in accordance with an existing communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). Via the wireless communication module 58, a dentition image showing the teeth D is transmitted from the intraoral camera 10 to the mobile terminal 70, and an operation signal is transmitted from the mobile terminal 70 to the intraoral camera 10.

[0054] In this embodiment, the power supply control unit 60 is mounted on the handle portion 10b of the intraoral camera 10 and distributes power from a battery 66 to the central control unit 50, the LED control unit 54, the lens driver 56, and the wireless communication module 58. The power supply control unit 60 is configured, for example, by a circuit. In this embodiment, the battery 66 is a rechargeable secondary battery, and is wirelessly charged by an external charger 69 connected to a commercial power source via a coil 68 mounted on the intraoral camera 10.

[0055] The position sensor 90 is a sensor for detecting the posture and position of the intraoral camera 10, and is, for example, a multi-axis (here, three axes: x, y, and z) acceleration sensor. For example, the position sensor 90 may be a six-axis sensor having a three-axis acceleration sensor and a three-axis gyro sensor. For example, as shown in FIG. 1 , the z-axis coincides with the optical axis LA. The y-axis is parallel to the imaging surface and extends in the longitudinal direction of the intraoral camera 10. The x-axis is parallel to the imaging surface and perpendicular to the y-axis. Outputs of each axis of the position sensor 90 may be transmitted to the mobile terminal 70 via the central control unit 50 and the wireless communication module 58.

[0056] The position sensor 90 may be a piezo-resistive, capacitance, or thermal-sensing MEMS (Micro Electro Mechanical Systems) sensor. Although not shown, a correction circuit may be provided to correct the balance of sensitivity of the sensors for each axis, the temperature characteristics of sensitivity, or temperature drift. A band-pass filter (low-pass filter) may also be provided to remove dynamic acceleration components or noise. Noise may also be reduced by smoothing the output waveform of the acceleration sensor.

[0057] Next, the operation of the intraoral camera system will be described. Fig. 4 is a diagram showing the flow of operation of the intraoral camera system. Note that the process shown in Fig. 4 is, for example, a process performed in real time, and is performed each time one frame or multiple frames of image data are obtained.

[0058] A user uses the intraoral camera 10 to capture images of the teeth and gums in their oral cavity, generating image data (S101). This image data is obtained by, for example, capturing images of teeth that are fluorescently reacting when irradiated with light including a blue wavelength range. Next, the intraoral camera 10 transmits the captured image data to the mobile terminal 70 (S102). Note that the image data may be a video or one or more still images. If the image data is a video or multiple still images, the sensor data may be transmitted for each frame of the video or for each still image. If the image data is a video, the sensor data may be transmitted for multiple frames.

[0059] Furthermore, the image data may be transmitted in real time, or may be transmitted all at once after a series of photographs (for example, photographs of all teeth in the oral cavity) have been taken.

[0060] The mobile terminal 70 performs image processing on the received image data (S103), generates an image showing the plaque region using the image data after the image processing (S104), and displays the generated image (S105).

[0061] By using such an intraoral camera system, a user can take an image of the inside of their own oral cavity with the intraoral camera 10 and check the condition of the inside of their oral cavity displayed on the mobile terminal 70. Furthermore, since plaque areas are shown on the displayed image, the user can easily check for areas that have not been brushed when brushing their teeth.

[0062] Furthermore, the mobile terminal 70 may generate a three-dimensional model of a plurality of teeth in the oral cavity from a plurality of captured image data, and may display an image based on the generated three-dimensional model.

[0063] Although an example in which the mobile terminal 70 processes images of teeth will be described here, some or all of this processing may be performed by the intraoral camera 10. The mobile terminal 70 is an example of an image processing device (dental plaque detection device).

[0064] 5 is a functional block diagram of the mobile terminal 70. The mobile terminal 70 includes an acquisition unit 101, a generation unit 102, and a display unit 103.

[0065] The acquisition unit 101 acquires image data (first RGB image) transmitted from the intraoral camera 10. The acquisition unit 101 may acquire sensor data in addition to image data from the intraoral camera 10. The first RGB image is an image obtained by the intraoral camera 10 capturing an image of a tooth that is undergoing a fluorescent reaction by irradiating the tooth with light including a wavelength range of blue light. Here, blue light is an example of irradiated light of a predetermined wavelength that excites a fluorescent substance contained in dental plaque. The fluorescent substance is, for example, porphyrin.

[0066] The generation unit 102 generates a display image showing a plaque region using the image data (first RGB image) acquired by the acquisition unit 101. The generation unit 102 includes an image processing unit 111, a natural tooth region detection unit 112, a plaque region detection unit 113, and an image generation unit 114.

[0067] The image processing unit 111 performs image processing as pre-processing on the first RGB image. Specifically, the image processing unit 111 generates a second RGB image by performing inverse gamma conversion (inverse gamma correction) on the first RGB image. Next, the image processing unit 111 generates a third RGB image by performing white balance adjustment processing on the second RGB image. Next, the image processing unit 111 generates a fourth RGB image by performing gamma conversion (gamma correction) on the third RGB image.

[0068] The white balance adjustment process is a process for adjusting the hue of white in an image, etc. For example, the image processing unit 111 extracts, from among a plurality of second RGB pixels constituting the second RGB image to be processed, a plurality of pixels whose RGB values ​​satisfy the following formulas 1 and 2:

[0069] min(R,G,B)≦Ths,and,max(R,G,B)<Thmax (Formula 1)

[0070] Thl≦Y≦Thu (Formula 2)

[0071] In Equation 1, min(R, G, B) represents the minimum value among the pixel values ​​of the three RGB sub-pixels (i.e., the red pixel value, green pixel value, and blue pixel value) of the second RGB pixel.

[0072] Ths is a threshold value for excluding areas in the first RGB image that are strongly affected by reflection of the irradiating light (for example, glossy areas), and is, for example, 900 in 10-bit representation.

[0073] max(R, G, B) indicates the maximum value among the pixel values ​​of the three RGB sub-pixels (that is, the red pixel value, green pixel value, and blue pixel value) of the second RGB pixel.

[0074] Thmax indicates the maximum value that a pixel value can take, and is expressed as 1023 in 10-bit notation, for example.

[0075] In Equation 2, Thl is a threshold value indicating the lower limit of the tooth region, and Thu is a threshold value indicating the upper limit of the tooth region.

[0076] The tooth region in the second RGB image is extracted using Equation 2. That is, the multiple pixels extracted using Equations 1 and 2 are multiple pixels that make up the tooth region.

[0077] The image processing unit 111 then calculates a first red pixel average value Rave, which is the average value of multiple red pixel values ​​in the tooth region that satisfy Equations 1 and 2, a first green pixel average value Gave, which is the average value of multiple green pixel values ​​in the tooth region, and a first blue pixel average value Bave, which is the average value of multiple blue pixel values ​​in the tooth region. The image processing unit 111 then adjusts the gains of at least two color components, the red component, the green component, and the blue component, of the RGB image to be processed so that the first red pixel average value Rave, the first green pixel average value Gave, and the first blue pixel average value Bave are equal.

[0078] Specifically, the image processing unit 111 calculates the gains of the multiple red pixel values ​​(red pixel gains) by dividing the first green pixel average value Gave by the first red pixel average value Rave. The image processing unit 111 also calculates the gains of the multiple blue pixel values ​​(blue pixel gains) by dividing the first green pixel average value Gave by the first blue pixel average value Bave. The image processing unit 111 then generates a third RGB image by multiplying each red pixel of the multiple second RGB pixels constituting the second RGB image by the red pixel gain and multiplying each blue pixel of the multiple second RGB pixels by the blue pixel gain. In other words, the pixel values ​​of the multiple third RGB pixels constituting the third RGB image are calculated by multiplying the red pixel values ​​of the multiple second RGB pixels constituting the second RGB image by the red pixel gain and by multiplying the blue pixel values ​​of the multiple second RGB pixels by the blue pixel gain. It should be noted that the image processing unit 111 performs white balance adjustment processing by calculating a gain for red pixels and a gain for blue pixels based on the average green pixel value and multiplying each gain by the pixel value of the corresponding color component. However, this is not limiting, and the green pixel gain and blue pixel gain may be calculated based on the average red pixel value, or the red pixel gain and green pixel gain may be calculated based on the average blue pixel value.

[0079] If the pixel value exceeds the maximum value (1023 in the case of 10-bit representation) as a result of multiplying the gain, the image processing unit 111 replaces the pixel value with 1023.

[0080] Note that the white balance adjustment process shown here is an example, and other methods may be used. Furthermore, the image processing content shown here is an example, and other processes may be used. For example, the image processing may include exposure control processing, etc. Furthermore, only some of the above-described processes may be performed, or no image processing may be performed. In other words, the first RGB image may be used to perform a natural tooth region detection process, which will be described later.

[0081] The natural tooth region detection unit 112 detects a natural tooth region, which is a region of natural teeth, from among the tooth regions in the fourth RGB image, based on the fourth RGB image. The plaque region detection unit 113 detects a plaque region, which is a region in the fourth RGB image where plaque exists, based on the fourth RGB image. The image generation unit 114 generates a display image showing the plaque region. Specifically, the image generation unit 114 generates a display image that displays the plaque region limited to the natural tooth region. For example, the display image is an image in which the plaque region is superimposed on the image data (fourth RGB image).

[0082] The display unit 103 is a display device included in the mobile terminal 70 and displays the display image generated by the image generation unit 114 .

[0083] The image generation process (step S104 shown in FIG. 4) will be described in detail below with reference to FIG. 6 to FIG. 14. FIG. 6 is a flowchart showing the details of the image generation process.

[0084] Fig. 7 is a diagram showing an example of an RGB image (fourth RGB image). The RGB image shown in Fig. 7 is an image obtained by capturing a natural tooth 201, an artificial tooth 202, and gums 203 (gums). The artificial tooth 202 is, for example, a prosthesis made of a metal such as gold or silver. The natural tooth 201 is a natural tooth, and is the portion of the tooth excluding the artificial tooth 202.

[0085] First, the natural tooth area detection unit 112 detects a first natural tooth area using the RGB image (fourth RGB image) (S201). Specifically, the natural tooth area detection unit 112 detects, as the first natural tooth area, an area in the RGB image that satisfies both a first condition that the green pixel value (G) is equal to or greater than a predetermined first threshold and a second condition that the differences between the green pixel value (G), the red pixel value (R), and the blue pixel value (B) are less than a predetermined third threshold.

[0086] When excitation light (blue light) is irradiated onto a natural tooth, excitation fluorescence is emitted from the dentin. This excitation fluorescence passes through the enamel, causing the natural tooth to fluoresce green. Furthermore, when blue light is irradiated onto a filling from a caries treatment, it appears dark (low brightness) in an image captured by a camera, unlike when white light is irradiated. On the other hand, natural teeth covered with enamel appear bright (high brightness) in the image. Therefore, by extracting this green fluorescence according to the first condition, it is possible to identify the area of ​​the natural tooth and exclude the area of ​​the artificial tooth.

[0087] Specifically, the second condition is (1) the absolute value (abs(R-G)) of the difference between the red pixel value (R) and the green pixel value (G) is less than a threshold value rg_th, (2) the absolute value (abs(G-B)) of the difference between the green pixel value (G) and the blue pixel value (B) is less than a threshold value gb_th, and (3) the absolute value (abs(B-R)) of the difference between the blue pixel value (B) and the red pixel value (R) is less than a threshold value br_th. Note that the threshold values ​​rg_th, gb_th, and br_th may be the same value or different values.

[0088] In the first condition, the luminance value (Y) may be used instead of the green pixel value (G). The luminance value (Y) is calculated using the following equation 3.

[0089] Y=0.21*R+0.72*G+0.07*B (Formula 3)

[0090] In Equation 3, Y is the luminance value, R is the red pixel value, G is the green pixel value, and B is the blue pixel value. As shown in Equation 3, the ratio of green pixel values ​​to luminance values ​​is high, so even when luminance values ​​are used, detection can be performed in the same way as when green pixel values ​​are used.

[0091] Although the case where both the first condition and the second condition are used has been described here, only one of the first condition and the second condition may be used. For example, only the first condition may be used.

[0092] Fig. 8 is a diagram showing an example of the detected first natural tooth region 211. Note that for the sake of explanation, Fig. 8 illustrates the shapes of the teeth, etc. in addition to the first natural tooth region 211, but the detected information does not necessarily include the shapes of the teeth, etc. This also applies to Figs. 9 to 13.

[0093] 8, a region including the region of the natural tooth 201 and the region of the gum 203 is detected as a first natural tooth region 211. In other words, the first natural tooth region 211 does not include the region of the artificial tooth 202.

[0094] Note that, although an example is shown here in which the first natural tooth region 211 includes the entire area of ​​the gums 203, depending on the photographing conditions, there may be cases in which only a portion of the area of ​​the gums 203 is included in the first natural tooth region 211, or in which the area of ​​the gums 203 is not included in the first natural tooth region 211.

[0095] Next, the natural tooth area detection unit 112 generates an HSV image by converting the color space of the RGB image (fourth RGB image) into an HSV space (S202). Next, the natural tooth area detection unit 112 detects the gingival area using the HSV image (S203). Specifically, the natural tooth area detection unit 112 detects an area whose hue (H) falls within a predetermined range as the gingival area.

[0096] Fig. 9 is a diagram showing an example of a detected gum region 212. As shown in Fig. 9, the region of the gums 203 is detected as the gum region 212. Note that, although an example in which the gum region is detected is shown here, lips and the like may also be detected as the gum region 212 in addition to the gums.

[0097] Next, the natural tooth area detection unit 112 determines the area obtained by excluding the gum area 212 from the first natural tooth area 211 as the second natural tooth area 213 (S204). Fig. 10 is a diagram showing an example of the second natural tooth area 213. As shown in Fig. 10, the second natural tooth area 213 includes the area of ​​the natural tooth 201, but does not include the areas of the artificial tooth 202 and gum 203.

[0098] Next, the natural tooth area detection unit 112 determines the third natural tooth area by excluding small areas from the second natural tooth area 213 (S205). Here, a small area is, for example, an area whose area is smaller than a predetermined value. This prevents high-intensity areas other than the natural tooth area (for example, the gum area) due to reflected light from being mistakenly determined as the natural tooth area.

[0099] Next, the natural tooth area detection unit 112 determines a fourth natural tooth area by expanding the third natural tooth area (S206). Specifically, the natural tooth area detection unit 112 determines the fourth natural tooth area by expanding the boundary of the third natural tooth area outward by a predetermined amount. This allows the boundary between the tooth and gum, etc., to be added to the natural tooth area, preventing excessive display of plaque areas.

[0100] Next, the plaque region detection unit 113 detects a first plaque region (S207). Specifically, the plaque region detection unit 113 detects as the first plaque region a region in which one or more pixels among the plurality of pixels in the HSV image generated in step S202 are located, the saturation of which falls within a first predetermined range (e.g., 30 to 80 in 8-bit representation), the hue of which falls within a second predetermined range (e.g., 140 to 170 in 8-bit representation), and the brightness of which falls within a third predetermined range (e.g., 100 to 180 in 8-bit representation). Note that the first, second, and third predetermined ranges are not limited to the above-mentioned numerical ranges, and may be identified by comparing the actual plaque region, tooth region, and HSV image.

[0101] The ranges of saturation, hue, and brightness values ​​can be determined by administering a plaque staining agent and comparing the degree of staining by the plaque staining agent.

[0102] Fig. 11 is a diagram showing an example of a detected first plaque region 214. As shown in Fig. 11, the region where plaque exists is detected as the first plaque region 214. In this example, the first plaque region 214 includes the region of the artificial tooth 202. In other words, the region of the artificial tooth 202 has been erroneously detected as a plaque region.

[0103] Note that, although an example is shown here in which information indicating whether or not plaque is present (binary) is detected as the first plaque region 214, the amount of plaque (multiple values) may also be detected. For example, the accumulation level (concentration or density) of the fluorescent substance can be detected by comparing the intensity of red fluorescence per unit area of ​​the first plaque region 214. For example, the plaque region detection unit 113 may detect the accumulation level based on the value of brightness V of the HSV image of the first plaque region 214. Furthermore, the plaque region detection unit 113 may perform a gradation display according to the accumulation level.

[0104] Next, the image generating unit 114 generates a display image by extracting the first plaque region 214 from the fourth natural tooth region (S208). Specifically, the image generating unit 114 generates a mask image in which the fourth natural tooth region is represented by a value of 1 (white) and the other regions are represented by a value of 0 (black). Fig. 12 is a diagram showing an example of the mask image 215. As shown in Fig. 12, in the mask image 215, the region of the natural tooth 201 has a value of 1 (white) and the region other than the natural tooth 201 has a value of 0 (black).

[0105] Next, the image generating unit 114 calculates the second plaque region by calculating the logical product of the mask image 215 and the image showing the first plaque region 214 shown in Fig. 11. That is, the image generating unit 114 extracts, as the second plaque region, the region of the first plaque region 214 that is included in the fourth natural tooth region. In other words, the image generating unit 114 determines the second plaque region by excluding, from the first plaque region 214, the region that is included in the region other than the fourth natural tooth region.

[0106] 13 is a diagram showing an example of a second plaque region. As shown in FIG. 13, the region obtained by excluding the region of the artificial tooth 202 from the first plaque region 214 is determined as a second plaque region 216.

[0107] Next, the image generating unit 114 generates a display image in which the second plaque region 216 is superimposed on the first RGB image (FIG. 7). FIG. 14 is a diagram showing an example of the display image. As shown in FIG. 14, by displaying information indicating the plaque region on the tooth image, the user can easily identify areas that have not been brushed. Furthermore, the above processing can prevent prostheses and the like from being mistakenly detected as plaque.

[0108] Note that the processing procedure shown in Fig. 6 is an example, and the present disclosure is not limited to this. For example, the timing at which the plaque detection process (S207) is performed may be any timing before step S208. Also, it is not necessary to perform all of the processes shown in Fig. 6, and some of the processes may not be performed. For example, at least some of the processes of steps S203 to S206 may not be performed.

[0109] Furthermore, while the above describes an example in which the artificial teeth are made of metals such as gold or silver, the artificial teeth are not limited to metals and may be made of ceramics, zirconia, or the like. As described above, the natural tooth region is detected by utilizing the green fluorescence that occurs when natural teeth are irradiated with excitation light (blue light). Therefore, artificial teeth made of materials other than metals can also be similarly excluded from the natural tooth region. On the other hand, the above method does not necessarily have to exclude artificial teeth made of all materials from the natural tooth region. By using the above method, it is possible to exclude artificial teeth made of at least some materials (e.g., metal) from the natural tooth region, thereby achieving the effect of reducing false detection compared to not using the method.

[0110] In the above embodiment, the intraoral camera 10 transmits the first RGB image to the mobile terminal 70, and the first RGB image is processed in the mobile terminal 70. However, this is not limiting. The first RGB image may be transmitted to the cloud server 80, the cloud server 80 performs the above processing, and the second RGB image or the fourth RGB image resulting from the processing may be transmitted to the mobile terminal 70. In this case, the first RGB image may be transmitted from the intraoral camera 10 to the cloud server 80 without passing through the mobile terminal 70, or may be transmitted from the intraoral camera 10 to the cloud server 80 via the mobile terminal 70.

[0111] As described above, the image processing method according to this embodiment is an image processing method for displaying a plaque region based on an image (e.g., a first RGB image) of teeth and plaque that have undergone a fluorescent reaction when the teeth are irradiated with light including a wavelength range of blue light. The image processing method detects natural tooth regions within the tooth region based on the image (e.g., S104 in FIG. 4 and S201 to S206 in FIG. 6), detects plaque regions based on the image (e.g., S207 in FIG. 6), and displays the plaque region limited to the natural tooth region (e.g., S105 in FIG. 4). This makes it possible to prevent erroneous detection of prosthetic appliances, etc. as plaque. Therefore, the image processing method can improve the accuracy of plaque detection.

[0112] For example, in detecting natural tooth regions, the image processing method detects a first region (e.g., a first natural tooth region) that is (i) a region in the entire pixel region of the image where the luminance value is equal to or greater than a predetermined first threshold, or (ii) a region in the entire pixel region of the image where the green pixel value is equal to or greater than a predetermined second threshold, and detects the natural tooth region based on the first region (e.g., S201 in Fig. 6). This allows the image processing method to accurately detect natural tooth regions using the luminance value or the green pixel value.

[0113] For example, in detecting a natural tooth region, the image processing method detects, within the first region, a region in which the differences between the green, red, and blue pixel values ​​are less than a predetermined third threshold as a natural tooth region (e.g., S201 in Fig. 6 ). This allows the image processing method to accurately detect a natural tooth region using the differences between the green, red, and blue pixel values.

[0114] For example, in detecting the natural tooth region, the image processing method detects the gingival region based on the hue of the image (e.g., S203 in FIG. 6), and then detects the region obtained by excluding the gingival region from the first region as the natural tooth region (e.g., S204 in FIG. 6). This makes it possible to prevent the gingival region from being erroneously determined as the natural tooth region, thereby improving the accuracy of detecting the natural tooth region.

[0115] For example, in detecting the natural tooth region, the image processing method detects, as the natural tooth region, a region obtained by excluding regions of the first region whose area is smaller than a predetermined area (e.g., S205 in Fig. 6 ). This image processing method can prevent high-brightness regions, etc. caused by reflected light, etc. outside the natural tooth region from being erroneously determined as the natural tooth region, thereby improving the accuracy of detecting the natural tooth region.

[0116] For example, in detecting the natural tooth region, the image processing method detects the natural tooth region by extending the boundary of the first region outward by a predetermined amount (e.g., S206 in Fig. 6 ). In this way, the image processing method can prevent the plaque region from being displayed excessively by extending the natural tooth region.

[0117] Furthermore, the image processing device (e.g., mobile terminal 70) according to this embodiment is an image processing device that displays a plaque region based on an image (e.g., a first RGB image) of teeth and plaque that have undergone a fluorescent reaction when the teeth are irradiated with light including a wavelength range of blue light, and includes: a natural tooth region detection unit 112 that detects a natural tooth region within the tooth region based on the image; a plaque region detection unit 113 that detects a plaque region based on the image; and a display unit 103 that displays the plaque region limited to the natural tooth region. This allows the image processing device to prevent erroneous detection of a prosthesis or the like as plaque. Therefore, the image processing device can improve the accuracy of plaque detection.

[0118] Although the intraoral camera system according to the embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment.

[0119] For example, although the above description has been given of an example in which the intraoral camera 10 is primarily intended to photograph teeth, the intraoral camera 10 may be an oral care device equipped with a camera, such as an oral irrigator equipped with a camera.

[0120] Furthermore, each processing unit included in the intraoral camera system according to the above embodiment is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0121] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0122] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0123] The present disclosure may also be realized as an image processing method or the like executed by an intraoral camera system. The present disclosure may also be realized as an intraoral camera, a mobile terminal, or a cloud server included in the intraoral camera system.

[0124] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0125] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0126] Although the intraoral camera system according to one or more aspects has been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the present embodiments and forms constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0127] The present disclosure is applicable to intraoral camera systems.

[0128] 10 Intraoral camera 10a Head 10b Handle 10c Neck 12 Imaging optical system 12a Incident port 14 Imaging element 16 Lens 18 Mirror 20 Blue light cut filter 24 Aperture 26A, 26B, 26C, 26D LED 28 Cover 30 Composition adjustment mechanism 32 Focus adjustment mechanism 34 Housing 36, 40 Actuator 38 Lens holder 50 Central control unit 54 LED control unit 56 Lens driver 58 Wireless communication module 60 Power supply control unit 62 Controller 64 Memory 66 Battery 68 Coil 69 Charger 70 Portable terminal 72 Touch screen 80 Cloud server 90 Position sensor 101 Acquisition unit 102 Generation unit 103 Display unit 111 Image processing unit 112 Natural tooth region detection unit 113 Plaque region detection unit 114 Image generation unit 201 Natural tooth 202 Artificial tooth 203 Gingiva 211 First natural tooth region 212 Gingiva region 213 Second natural tooth region 214 First plaque region 215 Mask image 216 Second plaque region

Claims

1. An image processing method for displaying a plaque area based on an image of a tooth and plaque that are fluorescently reacting when light including a blue light wavelength range is irradiated onto the tooth, the image processing method comprising the steps of: detecting a natural tooth area within the tooth area based on the image; detecting a plaque area based on the image; and displaying the plaque area limited to the natural tooth area.

2. The image processing method according to claim 1, wherein in detecting the natural tooth area, a first area is detected which is either (i) an area in the entire pixel area of ​​the image where the brightness value is equal to or greater than a predetermined first threshold value, or (ii) an area in the entire pixel area of ​​the image where the green pixel value is equal to or greater than a predetermined second threshold value, and the natural tooth area is detected based on the first area.

3. The image processing method according to claim 2, wherein in detecting the natural tooth region, an area of ​​the first region in which the differences between green pixel values, red pixel values ​​and blue pixel values ​​are each less than a predetermined third threshold value is detected as the natural tooth region.

4. An image processing method as claimed in claim 2 or 3, wherein in detecting the natural tooth area, a gingival area is detected based on the hue of the image, and an area excluding the gingival area from the first area is detected as the natural tooth area.

5. An image processing method according to claim 2 or 3, wherein in detecting the natural tooth region, an area obtained by excluding an area smaller than a predetermined area from the first region is detected as the natural tooth region.

6. An image processing method according to claim 2 or 3, wherein in detecting the natural tooth area, an area obtained by extending the boundary of the first area outward by a predetermined amount is detected as the natural tooth area.

7. An image processing device that displays a plaque area based on an image of the tooth and plaque that are undergoing a fluorescent reaction when light including a wavelength range of blue light is irradiated onto the tooth, the image processing device comprising: a natural tooth area detection unit that detects a natural tooth area within the tooth area based on the image; a plaque area detection unit that detects a plaque area based on the image; and a display unit that displays the plaque area limited to the natural tooth area.

8. A program for causing a computer to execute the image processing method according to claim 1.

Images