Image processing method, image processing device, and program

Abstract

This image processing method includes: acquiring a first RGB image obtained by photographing a tooth and dental plaque that are fluorescent-reacted by irradiating the tooth with light including blue light (S111); acquiring color data indicating a color of a region of the tooth to be photographed (S112); and generating a second RGB image by performing image processing including first image processing on the first RGB image (S113). The first image processing is a process of adjusting a gain of at least two color components among a red component, a green component, and a blue component so that a difference between the acquired color data and color data based on a first red pixel average value, a first green pixel average value, and a first blue pixel average value within a specific region that is a tooth region in an RGB image to be processed falls within a predetermined range.

Description

Image processing method, image processing device, and program

[0001] The present disclosure relates to an image processing method, an image processing device, and a program.

[0002] Patent Document 1 discloses an intraoral camera system that captures images of teeth in the oral cavity.

[0003] Japanese Patent Application Laid-Open No. 2019-141582

[0004] In such an intraoral camera system, it is desirable to be able to accurately identify plaque regions, which are regions where plaque has adhered, in tooth images.

[0005] Therefore, the present disclosure provides an image processing method and the like that can accurately identify plaque regions in a tooth image.

[0006] An image processing method according to one aspect of the present disclosure includes: acquiring a first RGB image obtained by photographing the teeth and plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light; acquiring color data indicating the color of the tooth region being photographed; and generating a second RGB image by performing image processing including a first image processing on the first RGB image; the first image processing being a process of adjusting the gain of at least two color components among the red, green, and blue components so that a difference between the acquired color data and color data based on a first red pixel average value of multiple red pixel values ​​possessed by multiple first pixels within a specific region of the tooth region in the RGB image being processed that is a tooth region that is not affected by plaque, a first green pixel average value of multiple green pixel values ​​possessed by the multiple first pixels, and a first blue pixel average value of multiple blue pixel values ​​possessed by the multiple first pixels, falls within a predetermined range.

[0007] These comprehensive or specific aspects may be realized as a system, an apparatus, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, an apparatus, an integrated circuit, a computer program, and a recording medium. The recording medium may also be a non-transitory recording medium.

[0008] The image processing method and the like according to the present disclosure can accurately identify plaque regions in a tooth image.

[0009] FIG. 1A is a perspective view of an intraoral camera in an intraoral camera system according to an embodiment. FIG. 1B is a cross-sectional view illustrating a schematic diagram of an imaging optical system incorporated in the intraoral camera in the intraoral camera system according to an embodiment. FIG. 2 is a schematic configuration diagram of the intraoral camera system according to an embodiment. FIG. 3 is a diagram illustrating a flow of intraoral imaging operations in the intraoral camera system according to an embodiment. FIG. 4 is a functional block diagram of a mobile terminal according to an embodiment. FIG. 5 is a diagram illustrating a state in which an upper front tooth is being imaged from the buccal side. FIG. 6A is a diagram illustrating an example of a first RGB image of the front tooth imaged in the state illustrated in FIG. 5. FIG. 6B is a diagram illustrating an example of color difference data between a plaque region and a tooth region when light including a blue light wavelength range is irradiated onto the tooth without a blue light cut filter. FIG. 6C is a diagram illustrating an example of color difference data between a plaque region and a tooth region when light including a blue light wavelength range is irradiated onto the tooth using a blue light cut filter. Fig. 6D is a diagram showing an example of color difference data between a plaque region and a tooth region when light including a blue wavelength range and white light are irradiated onto the tooth without a blue light cut filter. Fig. 7A is a diagram showing an example of a second RGB image of a front tooth captured in the state shown in Fig. 5. Fig. 7B is a diagram showing an example of color difference data between a plaque region and a tooth region after exposure control processing and white balance adjustment processing when light including a blue wavelength range is irradiated onto the tooth without a blue light cut filter. Fig. 7C is a diagram showing an example of color difference data between a plaque region and a tooth region after exposure control processing and white balance adjustment processing when light including a blue wavelength range is irradiated onto the tooth using a blue light cut filter. Fig. 7D is a diagram showing an example of color difference data between a plaque region and a tooth region after exposure control processing and white balance adjustment processing when light including a blue wavelength range and white light are irradiated onto the tooth without a blue light cut filter. Fig. 8 is a diagram showing an example of a fourth RGB image of a front tooth captured in the state shown in Fig. 5. Fig. 9 is a diagram showing another example of a fourth RGB image of a front tooth captured in the state shown in Fig. 5. Fig. 10 is a flowchart of image processing in a mobile terminal. Fig. 11 is a flowchart showing details of processing for detecting natural tooth regions. Fig. 12 is a diagram showing an example of an RGB image (first RGB image).Fig. 13 is a diagram showing an example of a detected first natural tooth region, Fig. 14 is a diagram showing an example of a detected gingival region, and Fig. 15 is a diagram showing an example of a detected second natural tooth region.

[0010] (Findings that form the basis of the present disclosure) It is known that the color of human teeth varies from person to person. When performing image processing on an image of a person's teeth to identify a plaque region, if the color of the person's tooth region is not known, there is a possibility that the wrong region will be identified as the plaque region.

[0011] Therefore, the inventors have discovered an image processing method that can accurately identify plaque areas in a tooth image by performing image processing taking into account the color of the tooth area of ​​the person to be identified.

[0012] The image processing method according to the first aspect acquires a first RGB image obtained by photographing the teeth and dental plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light, acquires color data indicating the color of the tooth region of the photographed object, and generates a second RGB image by performing image processing including a first image processing on the first RGB image, wherein the first image processing is a process of adjusting the gain of at least two color components among the red, green, and blue components so that the difference between the acquired color data and color data based on a first red pixel average value of multiple red pixel values ​​possessed by multiple first pixels within a specific region that is the tooth region of the RGB image to be processed, a first green pixel average value of multiple green pixel values ​​possessed by the multiple first pixels, and a first blue pixel average value of multiple blue pixel values ​​possessed by the multiple first pixels, falls within a predetermined range.

[0013] This allows for image processing according to the color of the tooth being photographed, as the gain of the color component is adjusted based on the difference between color data based on pixels in a specific region, which is a region of the tooth where plaque is not present, and color data indicating the color of the tooth region being photographed, thereby enabling for accurate identification of the plaque region in the tooth image.

[0014] The image processing method according to the second aspect is the image processing method according to the first aspect, and in the first image processing, an area surrounded by a contour that is a boundary where the brightness of the RGB image to be processed changes from pixels darker than a predetermined brightness to pixels brighter than the predetermined brightness is detected as the specific area.

[0015] The image processing method of the third aspect is the image processing method of the first aspect, and in the first image processing, of the multiple RGB pixels that make up the specific area, a first area that is (i) an area where the brightness value is equal to or greater than a predetermined first threshold, or (ii) an area where the green pixel value is equal to or greater than a predetermined second threshold, is detected as a natural tooth area, and multiple second pixels included in the first area are set as the multiple first pixels.

[0016] An image processing method according to a fourth aspect is the image processing method according to the third aspect, wherein in the first image processing, an area excluding areas of plaque and tartar from the first area is detected as the natural tooth area.

[0017] An image processing method according to a fifth aspect is the image processing method according to the first aspect, wherein the specific region has as the plurality of first pixels a plurality of second pixels constituting the tooth region, each of which has a red pixel value, a green pixel value, and a blue pixel value, and the green pixel value of each pixel is the largest and is equal to or greater than a predetermined value.

[0018] Therefore, it is possible to identify a region of the tooth that is likely to be free of plaque as the specific region.

[0019] An image processing method according to a sixth aspect is an image processing method according to any one of the first to fifth aspects, further comprising: generating an HSV image by converting the color space of the second RGB image into an HSV space; identifying a specific pixel area in which one or more fourth pixels of the HSV image that satisfy at least one of a first predetermined range for saturation, a second predetermined range for hue, and a third predetermined range for brightness are located; and performing an enhancement process on the specific pixel area in the second RGB image to generate a third RGB image.

[0020] This allows a specific pixel region in the second RGB image to be identified as a plaque region, and saturation enhancement processing is performed on the specific pixel region, thereby generating a third RGB image in which the plaque region is easily distinguished, thereby making it easy to identify the plaque region in the tooth image.

[0021] An image processing method according to a seventh aspect is the image processing method according to the fifth or sixth aspect, further comprising: generating an HSL image by converting the color space of the second RGB image into an HSL space; identifying a specific pixel area in which one or more fourth pixels of the plurality of fourth pixels in the HSL image are located, the fourth pixels having at least one of a saturation within a first predetermined range, a hue within a second predetermined range, and a luminance within a third predetermined range; and performing an enhancement process on the specific pixel area in the second RGB image to generate a third RGB image.

[0022] The image processing method according to the eighth aspect is an image processing method according to any one of the first to seventh aspects, in which the color data is obtained based on a photographed image obtained by photographing the tooth.

[0023] Therefore, tooth color data can be easily obtained by taking an image of the tooth.

[0024] An image processing method according to a ninth aspect is the image processing method according to the eighth aspect, wherein the captured image is an image obtained by capturing an image with a camera provided in a mobile terminal.

[0025] Therefore, tooth color data can be easily obtained by taking an image of the tooth with a camera provided in the mobile terminal.

[0026] An image processing method according to a tenth aspect is the image processing method according to the eighth aspect, wherein the captured image is an image obtained by capturing an image of the inside of the oral cavity with an intraoral camera.

[0027] Therefore, tooth color data can be easily obtained by taking an image of the tooth with an intraoral camera.

[0028] An image processing method according to an eleventh aspect is an image processing method according to any one of the first to seventh aspects, in which the color data indicates a tooth color corresponding to an identifier specified by a user in a shade guide indicating tooth colors.

[0029] Therefore, tooth color data can be obtained without taking an image based on a shade guide.

[0030] The image processing device of the twelfth aspect comprises an acquisition unit that acquires a first RGB image obtained by photographing the teeth and plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light; a memory unit that stores color data indicating the color of the tooth area; and a generation unit that generates a second RGB image by performing image processing including a first image processing on the first RGB image, wherein the first image processing is a process of adjusting the gain of at least two color components among the red, green, and blue components so that the difference between the acquired color data and color data based on a first red pixel average value of multiple red pixel values ​​possessed by multiple first pixels within a specific area that is the tooth area of ​​the RGB image to be processed, a first green pixel average value of multiple green pixel values ​​possessed by the multiple first pixels, and a first blue pixel average value of multiple blue pixel values ​​possessed by the multiple first pixels falls within a predetermined range.

[0031] This allows for image processing according to the color of the tooth being photographed, as the gain of the color component is adjusted based on the difference between color data based on pixels in a specific region, which is a region of the tooth where plaque is not present, and color data indicating the color of the tooth region being photographed, thereby enabling for accurate identification of the plaque region in the tooth image.

[0032] A program according to a thirteenth aspect is a program for causing a computer to execute the image processing method according to any one of the first to eleventh aspects.

[0033] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0034] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0035] The inventors have provided the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0036] 1A is a perspective view of an intraoral camera in an intraoral camera system according to the present embodiment. As shown in FIG. 1A, the intraoral camera 10 includes a toothbrush-shaped housing that can be handled with one hand. The housing includes a head portion 10a that is placed in the user's oral cavity when photographing the dentition, a handle portion 10b that the user holds, and a neck portion 10c that connects the head portion 10a and the handle portion 10b.

[0037] The photographing optical system 12 is incorporated into the head portion 10a and the neck portion 10c and includes an image pickup device 14 and a lens arranged on the optical axis LA thereof.

[0038] 1B, in this embodiment, the imaging optical system 12 of the intraoral camera 10 is incorporated into the head portion 10a and the neck portion 10c. The imaging optical system 12 includes an image sensor 14 and a lens 16 arranged on its optical axis LA.

[0039] The image pickup element 14 is an imaging device such as a C-MOS sensor or a CCD element, and an image of the tooth D is formed by the lens 16. The image pickup element 14 outputs a signal (image data) corresponding to the formed image to the outside.

[0040] The lens 16 is, for example, a condenser lens, and forms an image of the tooth D incident thereon on the imaging element 14. The lens 16 may be a single lens or a lens group made up of a plurality of lenses.

[0041] In this embodiment 1, the photographing optical system 12 further includes a mirror 18 that reflects the image of the tooth D toward the lens 16, a blue light cut filter (blue blocking element) 20 arranged between the mirror 18 and the lens 16, and an aperture 24 arranged between the lens 16 and the image sensor 14.

[0042] The mirror 18 is disposed on the optical axis LA of the photographing optical system 12 so as to reflect the image of the tooth D that has passed through the entrance 12 a of the photographing optical system 12 toward the lens 16 .

[0043] The blue light cut filter 20 is a filter that cuts out light components of blue wavelengths contained in light incident on the image sensor 14. When light including a blue wavelength range is irradiated onto teeth to detect dental plaque, if the light including the blue wavelength range is increased to enhance the excitation fluorescence of the plaque, the entire first RGB image will appear blue. In this state, blue pixel values ​​become dominant compared to red and green pixel values, and the effect of making plaque areas easier to distinguish by performing image processing (exposure control processing and white balance adjustment processing) described below may be reduced. To address this issue, the blue light cut filter 20 cuts out light including the blue wavelength range from the light before it enters the image sensor 14.

[0044] The diaphragm 24 is a plate-like member with a through-hole on the optical axis LA of the imaging optical system 12, and realizes a deep focal depth, which allows the focus to be adjusted in the depth direction within the oral cavity, thereby obtaining an image of the row of teeth with clear contours.

[0045] The intraoral camera 10 is also equipped with a plurality of first to fourth LEDs 26A-26D as illumination devices that irradiate the tooth D to be photographed during imaging. The first to fourth LEDs 26A-26D are, for example, blue LEDs. As shown in FIG. 1A , in this first embodiment, the first to fourth LEDs 26A-26D are arranged to surround the entrance 12a. To prevent the gums G or other objects from coming into contact with the first to fourth LEDs 26A-26D and resulting in insufficient illumination light, a translucent cover 28 is provided on the head unit 10a to cover the first to fourth LEDs 26A-26D and the entrance 12a. Some of the first to fourth LEDs 26A-26D may be white LEDs. Using white LEDs for some of the first to fourth LEDs 26A-26D can brighten the first RGB image and improve the balance of blue pixel values ​​relative to red and green pixel values.

[0046] Furthermore, in this embodiment, the intraoral camera 10 has a composition adjustment mechanism 30 and a focus adjustment mechanism 32, as shown in FIG. 1B.

[0047] The composition adjustment mechanism 30 is composed of a housing 34 that holds the image sensor 14 and the lens 16, and an actuator 36 that moves the housing 34 in the direction of extension of the optical axis LA. The actuator 36 adjusts the position of the housing 34 to adjust the angle of view, i.e., the size of the row of teeth imaged on the image sensor 14. The composition adjustment mechanism 30 automatically adjusts the position of the housing 34 so that, for example, one entire tooth is captured in the captured image. The composition adjustment mechanism 30 also adjusts the position of the housing 34 based on a user's operation so that the angle of view desired by the user is achieved.

[0048] The focus adjustment mechanism 32 is held within the housing 34 of the composition adjustment mechanism 30 and is composed of a lens holder 38 that holds the lens 16 and an actuator 40 that moves the lens holder 38 in the direction of extension of the optical axis LA. The actuator 40 adjusts the relative position of the lens holder 38 with respect to the image sensor 14, thereby adjusting the focus, i.e., the focal point. The focus adjustment mechanism 32 automatically adjusts the position of the lens holder 38 so that, for example, a tooth located at the center of the captured image is in focus. The focus adjustment mechanism 32 also adjusts the position of the lens holder 38 based on a user's operation.

[0049] In addition, the components of the imaging optical system 12 except for the mirror 18 may be provided on the handle portion 10 b of the intraoral camera 10 .

[0050] The image sensor 14 is a photographing device such as a C-MOS sensor or a CCD element, and an image of the teeth is formed by a lens. The image sensor 14 outputs a signal (image data) corresponding to the formed image to the outside. The image output by the image sensor 14 is an RGB image in which each of the multiple pixels constituting the image has RGB sub-pixels.

[0051] The intraoral camera 10 is also equipped with a plurality of first to fourth LEDs 26A to 26D as lighting devices that irradiate light onto the teeth to be photographed during photography. The first to fourth LEDs 26A to 26D are, for example, blue LEDs that irradiate blue light having a wavelength with a peak at 405 nm. Note that the first to fourth LEDs 26A to 26D are not limited to blue LEDs and may be any light source that irradiates light including the wavelength range of blue light.

[0052] 2 is a schematic diagram of the intraoral camera system according to the present embodiment. As shown in FIG. 2, the intraoral camera system according to the present embodiment is generally configured to capture an image of the dentition using an intraoral camera 10 and perform image processing on the captured image.

[0053] 2 , the intraoral camera system includes an intraoral camera 10, a mobile terminal 70, and a cloud server 80. The mobile terminal 70 is, for example, a smartphone or tablet terminal capable of wireless communication. The mobile terminal 70 includes, as an input device and an output device, a touch screen 72 capable of displaying, for example, a dentition image. The mobile terminal 70 functions as a user interface for the intraoral camera system.

[0054] The cloud server 80 is a server that can communicate with the mobile terminal 70 via the Internet or the like, and provides the mobile terminal 70 with an application for using the intraoral camera 10. For example, a user downloads the application from the cloud server 80 and installs it on the mobile terminal 70. The cloud server 80 also acquires dentition images captured by the intraoral camera 10 via the mobile terminal 70.

[0055] The intraoral camera system includes a central control unit 50 as the main part that controls the system, an LED control unit 54 that controls multiple LEDs 26A to 26D, a lens driver 56 that controls the actuator 36 of the composition adjustment mechanism and the actuator 40 of the focus adjustment mechanism, and a position sensor 90.

[0056] The intraoral camera system also includes a wireless communication module 58 that communicates wirelessly with the mobile terminal 70, and a power supply control unit 60 that supplies power to the central control unit 50 and other components.

[0057] The central control unit 50 of the intraoral camera system is mounted, for example, on the handle portion 10b of the intraoral camera 10. For example, the central control unit 50 also includes a controller 62 such as a CPU or MPU that executes various processes described below, and a memory 64 such as a RAM or ROM that stores programs for causing the controller 62 to execute the various processes. In addition to the programs, the memory 64 also stores a row-of-teeth image (image data) captured by the image sensor 14, various setting data, and the like. The row-of-teeth image captured by the image sensor 14 is an example of a first RGB image.

[0058] The controller 62 transmits the row-of-teeth image output from the imaging element 14 to the mobile terminal 70 via the wireless communication module 58. The mobile terminal 70 displays the transmitted row-of-teeth image on the touch screen 72, thereby presenting the row-of-teeth image to the user.

[0059] The LED control unit 54 is mounted, for example, on the handle portion 10b of the intraoral camera 10, and turns on and off the first to fourth LEDs 26A to 26D based on a control signal from the controller 62. The LED control unit 54 is configured, for example, by a circuit. For example, when a user performs an operation on the touch screen 72 of the mobile terminal 70 to activate the intraoral camera 10, a corresponding signal is transmitted from the mobile terminal 70 to the controller 62 via the wireless communication module 58. Based on the received signal, the controller 62 transmits a control signal to the LED control unit 54 to turn on the first to fourth LEDs 26A to 26D.

[0060] The lens driver 56 is mounted, for example, on the handle portion 10b of the intraoral camera 10 and controls the actuator 36 of the composition adjustment mechanism and the actuator 40 of the focus adjustment mechanism based on a control signal from a controller 62 of the central control unit 50. The lens driver 56 is composed of, for example, a circuit. For example, when a user performs an operation related to composition adjustment or focus adjustment on the touch screen 72 of the mobile terminal 70, a corresponding signal is transmitted from the mobile terminal 70 to the central control unit 50 via the wireless communication module 58. The controller 62 of the central control unit 50 transmits a control signal to the lens driver 56 to adjust the composition or focus based on the received signal. Furthermore, for example, the controller 62 calculates control amounts of the actuators 36, 40 required for composition adjustment or focus adjustment based on the row-of-teeth image from the imaging element 14, and transmits a control signal corresponding to the calculated control amount to the lens driver 56.

[0061] The wireless communication module 58 is mounted on, for example, the handle portion 10b of the intraoral camera 10, and performs wireless communication with the mobile terminal 70 based on a control signal from the controller 62. The wireless communication module 58 performs wireless communication with the mobile terminal 70 in accordance with an existing communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). Via the wireless communication module 58, a dentition image showing the teeth D is transmitted from the intraoral camera 10 to the mobile terminal 70, and an operation signal is transmitted from the mobile terminal 70 to the intraoral camera 10.

[0062] In this embodiment, the power supply control unit 60 is mounted on the handle portion 10b of the intraoral camera 10 and distributes power from a battery 66 to the central control unit 50, the LED control unit 54, the lens driver 56, and the wireless communication module 58. The power supply control unit 60 is configured, for example, by a circuit. In this embodiment, the battery 66 is a rechargeable secondary battery, and is wirelessly charged by an external charger 69 connected to a commercial power source via a coil 68 mounted on the intraoral camera 10.

[0063] The position sensor 90 is a sensor for detecting the posture and position of the intraoral camera 10, and is, for example, a multi-axis (here, three axes: x, y, and z) acceleration sensor. For example, the position sensor 90 may be a six-axis sensor having a three-axis acceleration sensor and a three-axis gyro sensor. For example, as shown in FIG. 1 , the z-axis coincides with the optical axis LA. The y-axis is parallel to the imaging surface and extends in the longitudinal direction of the intraoral camera 10. The x-axis is parallel to the imaging surface and perpendicular to the y-axis. Outputs of each axis of the position sensor 90 may be transmitted to the mobile terminal 70 via the central control unit 50 and the wireless communication module 58.

[0064] The position sensor 90 may be a piezo-resistive, capacitance, or thermal sensing MEMS sensor. Although not shown, a correction circuit may be provided to correct the balance of the sensor sensitivity for each axis, the temperature characteristics of sensitivity, temperature drift, etc. A band-pass filter (low-pass filter) may also be provided to remove dynamic acceleration components and noise. Noise may also be reduced by smoothing the output waveform of the acceleration sensor.

[0065] Next, the intraoral imaging operation of the intraoral camera system will be described. Fig. 3 is a diagram showing the flow of the intraoral imaging operation of the intraoral camera system. Note that the process shown in Fig. 3 is, for example, a process performed in real time, and is performed each time one frame or multiple frames of image data are obtained.

[0066] A user uses the intraoral camera 10 to capture images of the teeth and gums in their oral cavity, thereby generating image data (S101). Next, the intraoral camera 10 transmits the captured image data to the mobile terminal 70 (S102). Note that the image data may be a video or one or more still images. If the image data is a video or multiple still images, the sensor data may be transmitted for each frame of the video or for each still image. If the image data is a video, the sensor data may be transmitted for each multiple frame.

[0067] Furthermore, the image data may be transmitted in real time, or may be transmitted all at once after a series of photographs (for example, photographs of all teeth in the oral cavity) have been taken.

[0068] The portable terminal 70 performs image processing on the received image data (first RGB image) (S103), and displays the image data after the image processing (S104).

[0069] By using such an intraoral camera system, a user can take an image of the inside of their own oral cavity with the intraoral camera 10 and check the condition of the inside of their oral cavity displayed on the mobile terminal 70. This allows the user to easily check the health condition of their own teeth.

[0070] Furthermore, the mobile terminal 70 may generate a three-dimensional model of a plurality of teeth in the oral cavity from a plurality of captured image data, and may display an image based on the generated three-dimensional model.

[0071] Here, an example will be described in which the mobile terminal 70 processes the tooth images, but some or all of this processing may be performed by the intraoral camera 10. The mobile terminal 70 is an example of an image processing device.

[0072] 4 is a functional block diagram of the mobile terminal 70. The mobile terminal 70 includes an acquisition unit 101, a generation unit 102, a display unit 103, and a storage unit 104.

[0073] The acquisition unit 101 acquires image data (first RGB image) transmitted from the intraoral camera 10. The acquisition unit 101 may acquire sensor data in addition to image data from the intraoral camera 10. The first RGB image is an image obtained by the intraoral camera 10 photographing a tooth that is undergoing a fluorescent reaction by irradiating the tooth with light including a wavelength range of blue light.

[0074] The generation unit 102 may generate a third RGB image by performing exposure control processing (second image processing) on ​​the first RGB image, and may generate a second RGB image by performing white balance adjustment processing (first image processing) on ​​the third RGB image.

[0075] The storage unit 104 stores color data indicating the color of the tooth region of the photographed subject. The color data may be acquired based on a photographed image obtained by photographing the tooth. Specifically, the color data may be generated based on a tooth region specified by the user in the photographed image. The photographed image may be an image obtained by photographing with a camera included in the mobile terminal 70. The photographed image may be an image obtained by photographing with a camera included in the mobile terminal 70 under natural light or white light, for example. The photographed image may be an image obtained by photographing with an intraoral camera 10 that photographs the inside of the oral cavity. The photographed image may be an image obtained by photographing with the intraoral camera 10 while irradiating white light from a white LED, for example.

[0076] The color data may also be data indicating the colors of multiple types of teeth included in a shade guide that indicates tooth colors. The acquiring unit 101 may acquire, for example, color data indicating the color of a tooth corresponding to an identifier specified by a user input among multiple types of teeth as the color data of the subject to be photographed. The multiple types of teeth are teeth with mutually different colors. The shade guide is, for example, a color chart that indicates the colors, tones, or hues that a person's teeth can have.

[0077] (Exposure Control Process) In the exposure control process, the generation unit 102 first extracts, from among the plurality of first RGB pixels (third pixels) constituting the first RGB image, a plurality of pixels whose RGB values ​​satisfy the following formulas 1 and 2.

[0078] min (R, G, B) ≦ Ths, and, max (R, G, B) < Thmax (Formula 1) Gmax - G ≦ Thb (Formula 2)

[0079] min(R, G, B) indicates the minimum value among the pixel values ​​of the three RGB sub-pixels (i.e., red pixel value, green pixel value, and blue pixel value) of the first RGB pixel.

[0080] Ths is a threshold value for excluding areas in the first RGB image that are strongly affected by reflection of the irradiating light (for example, glossy areas), and is, for example, 900 in 10-bit representation.

[0081] max(R, G, B) indicates the maximum value among the pixel values ​​of the three RGB sub-pixels (that is, the red pixel value, green pixel value, and blue pixel value) of the first RGB pixel.

[0082] Thmax indicates the maximum value that a pixel value can take. Thmax is expressed as 1023 in 10-bit notation, for example. Thmax is an example of a first threshold value.

[0083] Gmax is the maximum value of the green pixel values ​​in the first RGB image. In other words, it is the pixel value of the green pixel with the maximum pixel value among the green pixels of the first RGB pixels that make up the first RGB image. Thb is a threshold value for extracting the second green pixel from the first RGB pixels. Since a large Thb value makes the image too bright, it is set to a value of 10 or less in 10-bit representation, for example.

[0084] Glossy regions are excluded using Equation 1, and tooth regions in the first RGB image are extracted using Equation 2. That is, the multiple pixels extracted using Equations 1 and 2 are multiple second pixels constituting the tooth region. In this way, the multiple second pixels are pixels among the multiple first RGB pixels (third pixels) constituting the first RGB image that satisfy the following conditions: the pixel value max (R, G, B) of the maximum color component is smaller than the first threshold value (Thmax), and the pixel value min (R, G, B) of the minimum color component is equal to or smaller than the second threshold value (Ths).

[0085] The generation unit 102 calculates an average value of green pixels of the multiple second pixels and determines a gain by which to multiply the pixel values ​​of the three RGB sub-pixels based on the calculated average value of green pixels. The generation unit 102 determines a gain by which to multiply the pixel values ​​of the three RGB sub-pixels, for example, using the following equation 3. The gain is obtained by dividing the target pixel value by the average value of green pixels. The generation unit 102 generates a third RGB image by multiplying each of the multiple first RGB pixels constituting the first RGB image by the determined gain. More specifically, the generation unit 102 generates the third RGB image by multiplying, for each of the multiple first RGB pixels, the pixel values ​​of the three sub-pixels of the first RGB pixel by the determined gain. In other words, the pixel values ​​of the plurality of third RGB pixels constituting the third RGB image are pixel values ​​calculated by multiplying the pixel values ​​of the plurality of first RGB pixels constituting the first RGB image by the determined gain. Note that if the pixel value exceeds the maximum value (1023 in the case of 10-bit representation) as a result of multiplying the gain, the generation unit 102 replaces the pixel value with 1023.

[0086] In the above description, the average value of the green pixels of the plurality of second pixels extracted from the first RGB pixels is calculated using Equation 2, and the gain by which the pixel values ​​of the three RGB sub-pixels are multiplied is determined in accordance with the calculated average value of the green pixels. However, this is not limited to this. The average value of the red pixels of the plurality of second pixels extracted from the first RGB pixels may be calculated, and the gain by which the pixel values ​​of the three RGB sub-pixels are multiplied may be determined in accordance with the calculated average value of the red pixels. Similarly, the average value of the blue pixels of the plurality of second pixels extracted from the first RGB pixels may be calculated, and the gain by which the pixel values ​​of the three RGB sub-pixels are multiplied may be determined in accordance with the calculated average value of the blue pixels.

[0087] As described above, the exposure control process is a process of determining gains for a plurality of second pixel values ​​so that the average of a plurality of index values ​​calculated from a plurality of second pixel values ​​of a plurality of second pixels (pixels corresponding to the tooth region) included in the RGB image to be processed (here, the first RGB image) becomes a predetermined value, and applying the determined gains to a plurality of first RGB pixel values ​​of a plurality of first RGB pixels included in the first RGB image, thereby generating a third RGB image. Note that the index value may be a value calculated from the pixel values ​​of the three RGB subpixels that make up one pixel, or may be the pixel value of any one of the three subpixels. Here, the average of the plurality of index values ​​is the average of the color component having the largest pixel value among the red, green, and blue components of the first RGB image. Furthermore, the color component having the maximum average value is the color component having the maximum average value among three average values: a first red pixel average value of the multiple red pixel values ​​of the multiple first pixels constituting the first RGB image, a first green pixel average value of the multiple green pixel values ​​of the multiple first pixels, and a first blue pixel average value of the multiple blue pixel values ​​of the multiple first pixels. Note that the color component having the maximum average value does not need to be determined by calculating and comparing the first red pixel average value, the first green pixel average value, and the first blue pixel average value, and may be fixed to the green component.

[0088] In the above description, the generation unit 102 calculates the average value of the green pixels of the plurality of second pixels and determines the gain based on the calculated average value of the green pixels. However, this is not limited to this. The generation unit 102 may calculate the average value of the luminance values ​​of the plurality of second pixels as the average value of the plurality of index values ​​and determine the gain based on the calculated average value of the luminance values. In this manner, the index value may be the pixel value of any one of the three RGB subpixels that constitute one pixel, or may be a value calculated from the pixel values ​​of the three subpixels. Specifically, for each of the plurality of second pixels, the generation unit 102 calculates the luminance value of the second pixel using the subpixel values ​​of the three subpixels of the second pixel. For example, the generation unit 102 calculates the luminance value using the following Equation 3:

[0089] Y=0.21*R+0.72*G+0.07*B (Formula 3)

[0090] In Equation 3, Y is the luminance value, R is the red pixel value, G is the green pixel value, and B is the blue pixel value.

[0091] In this way, the plurality of luminance values ​​may be obtained by calculating, for each of the plurality of pixel values, the luminance values ​​based on the red pixel value, the green pixel value, and the blue pixel value included in the pixel value.

[0092] (White Balance Adjustment Process) In the white balance adjustment process, the generation unit 102 extracts, from among the third RGB pixels constituting the third RGB image to be processed, a plurality of pixels whose RGB values ​​satisfy Expression 1 and Expression 4 below.

[0093] Thl≦Y≦Thu (Formula 4)

[0094] In Equation 4, Thl is a threshold value indicating the lower limit of the tooth region, and Thu is a threshold value indicating the upper limit of the tooth region.

[0095] The tooth region in the third RGB image is extracted using Equation 4. That is, the multiple pixels extracted using Equations 1 and 4 are the multiple second pixels that make up the tooth region.

[0096] The generating unit 102 then extracts a specific region that is a tooth region. The specific region is a region of the tooth region that has a plurality of pixels whose sub-pixels have the maximum green pixel value and are equal to or greater than a predetermined value. In other words, the generating unit 102 extracts, as a plurality of first pixels that constitute the specific region, a plurality of pixels that have the maximum green pixel value among the red, green, and blue pixel values ​​of each pixel among the plurality of second pixels that constitute the tooth region and whose green pixel value is equal to or greater than a predetermined value.

[0097] The specific region may be a region surrounded by a contour that is a boundary where the brightness of the RGB image to be processed changes from pixels darker than a predetermined brightness to pixels brighter than the predetermined brightness.

[0098] The generation unit 102 then calculates a first average red pixel value Rave, which is the average value of multiple red pixel values ​​in the specific region, a first average green pixel value Gave, which is the average value of multiple green pixel values ​​in the tooth region, and a first average blue pixel value Bave, which is the average value of multiple blue pixel values ​​in the tooth region.The generation unit 102 then adjusts the gains of at least two color components, i.e., the red component, the green component, and the blue component, of the RGB image to be processed so that the difference between the acquired color data and color data based on the first average red pixel value Rave, the first average green pixel value Gave, and the first average blue pixel value Bave falls within a predetermined range.

[0099] Specifically, the gains of at least two color components are adjusted so that the ratio of the first average red pixel value Rave to the first average green pixel value Gave to the first average blue pixel value Bave is approximately the same as the ratio of the red pixels to the green pixels to the blue pixels obtained from the color data. "Approximately the same as the ratio of the red pixels to the green pixels to the blue pixels obtained from the color data" means that the value is within a predetermined range based on that ratio. The predetermined range may be, for example, a range of ±5%.

[0100] Furthermore, the generating unit 102 may perform the following third image processing on the second RGB image to emphasize the plaque region within the tooth region in the second RGB image. Specifically, the generating unit 102 generates the HSV image by converting the color space of the second RGB image into an HSV space. Then, the generating unit 102 identifies, as the plaque region, a specific pixel region in which one or more fourth pixels of the HSV image are located, the fourth pixels having at least one of a saturation within a first predetermined range (e.g., 30 to 80 in 8-bit representation), a hue within a second predetermined range (e.g., 140 to 170 in 8-bit representation), and a lightness within a third predetermined range (e.g., 100 to 180 in 8-bit representation). The first, second, and third predetermined ranges are not limited to the above-mentioned numerical ranges, and may be determined by comparing the actual plaque region and tooth region with the HSV image. The generator 102 generates the fourth RGB image by performing saturation enhancement processing on the plaque region in the second RGB image. Instead of performing saturation enhancement processing, the generator 102 may generate the fourth RGB image by replacing the plaque region with a predetermined pattern. The predetermined pattern may be, for example, graphics having a constant pixel value or graphics including a specific pattern.

[0101] Alternatively, the generating unit may generate an HSL image by converting the color space of the second RGB image into an HSL space, instead of generating an HSV image by converting the color space of the second RGB image into an HSV space. In this case, luminance is used instead of lightness, and the luminance condition falls within a third predetermined range.

[0102] The display unit 103 is a display device included in the mobile terminal 70, and displays the image after image processing by the generation unit 102. The display unit 103 may display the second RGB image or the fourth RGB image.

[0103] (Comparison of color difference data of plaque region and tooth region detected from first RGB images under various irradiation lights) (1) When the oral cavity is irradiated only with blue light (peak wavelength: 405 nm) Fig. 5 is a diagram showing a state in which a user is vertically holding the intraoral camera 10 and photographing the upper front teeth from the cheek side (lip side). Fig. 6A is a diagram showing an example of a first RGB image of the front teeth photographed in the state shown in Fig. 5 by irradiating the oral cavity with blue light (peak wavelength: 405 nm).

[0104] FIG. 6B is a diagram showing an example of color difference data of the plaque region 201 and tooth region detected in FIG. 6A . Note that, for comparison with color difference data of the plaque region 201 and tooth region detected from a first RGB image of a front tooth captured without the blue light cut filter 20 shown in FIG. 1B , FIG. 6B shows color difference data of the plaque region 201 and tooth region detected from a first RGB image captured without the blue light cut filter 20 shown in FIG. 1B , for reference. In FIG. 6B , the average color difference coordinates of the hue of the plaque region were (0.16, 0.06), and the average color difference coordinates of the hue of the tooth region were (0.25, −0.02). Note that these average color difference coordinates of the hues are for reference only and may vary depending on the intensity of blue light and the sensitivity of the camera.

[0105] As is known from quantitative visible light-induced fluorescence (QLF), bacteria in dental plaque fluoresce reddish-pink when exposed to blue light. It is also known that when teeth are irradiated with light containing wavelengths in the blue range, excited fluorescence is emitted from the dentin, which then transmits through the enamel layer and emits a green light. On the other hand, lips and gums have a bluish tint, making them easy to distinguish from each other.

[0106] (2) Cutting Light Including the Blue Wavelength Range with a Blue Light Cutoff Filter A blue light cutoff filter is an effective method for reducing noise. As shown in Fig. 1B, blue light cutoff filter 20 can cut light including the blue wavelength range from light reflected from the teeth before it enters image sensor 14.

[0107] 6C is a diagram showing an example of color difference data of a plaque region 201 and a tooth region detected from a first RGB image of a front tooth captured after cutting out light including the wavelength range of blue light using a blue light cut filter 20. For reference, in FIG. 6C, the color difference average coordinates of the hue of the plaque region are (0.058, 0.068), and the color difference average coordinates of the hue of the tooth region are (0.050, -0.004). Note that these color difference average coordinates of the hues are for reference only and may vary depending on the intensity of blue light and the sensitivity of the camera.

[0108] 6B and 6C, cutting light including the blue wavelength range using a blue-light cut filter reduces the saturation of the blue color (lightens the color). Because the plaque region fluoresces a reddish pink, it is possible to reduce the brightness of the tooth region while maintaining the brightness of the plaque region. This reduces the blue pixel value and narrows the difference between the red and green pixel values, thereby reducing the white balance gain multiplied by the red and green pixel values ​​and achieving the effect of suppressing noise.

[0109] (3) When the oral cavity is illuminated with blue light (peak wavelength 405 nm) and white light: By adding white light, the red and green pixel values ​​of the tooth region in the first RGB image are increased, and the exposure compensation gain can be kept smaller than when the oral cavity is illuminated with blue light alone. However, as the intensity of the white light increases, the difference between the hue of the plaque region and the hue of the gums and lips decreases, making it more likely that the gums and lips will be mistakenly identified as plaque. Therefore, the ratio of the intensity of light including the blue light wavelength range to the intensity of the white light must be optimized in advance to avoid false detection.

[0110] In order to irradiate the oral cavity with blue light and white light, some of the first to fourth LEDs 26A to 26D in FIG. 1 may be white LEDs.

[0111] FIG. 6D shows an example of color difference data of the plaque region 201 and tooth region detected from a first RGB image of a front tooth captured by illuminating the oral cavity with blue light (peak wavelength: 405 nm) and white light. For comparison with the color difference data of the plaque region 201 and tooth region detected from a first RGB image of a front tooth captured by filtering light including the blue wavelength range using the blue light blocking filter described above ( FIG. 6C ), FIG. 6D shows color difference data of the plaque region 201 and tooth region detected from a first RGB image captured without the blue light blocking filter 20 shown in FIG. 1B . For reference, in FIG. 6D , the average color difference coordinates of the hue of the plaque region were (0.089, 0.041), and the average color difference coordinates of the hue of the tooth region were (0.15, −0.013). Note that these average color difference coordinates of the hues are for reference only and may vary depending on the intensity of the blue light and white light and the sensitivity of the camera.

[0112] 6B and 6D, it can be seen that irradiating the oral cavity with white light in addition to blue light has the effect of suppressing the brightness of the bluish tooth region while maintaining the brightness of the plaque region. This reduces the blue pixel value and the difference between the red and green pixel values, thereby reducing the white balance gain by which the red and green pixel values ​​are multiplied, thereby achieving the effect of reducing noise.

[0113] (White balance processing) The mobile terminal 70 performs image processing (exposure control processing and white balance adjustment processing) to generate the second RGB image 210 in which the plaque region 211 is easily distinguishable, as shown in Fig. 7A. By performing image processing, the tooth region is essentially decolorized, making the plaque region 211 easier to distinguish.

[0114] 7B shows an example of color difference data between the plaque region and the tooth region after exposure control and white balance adjustment when light including the blue wavelength range is irradiated onto the tooth without a blue light cut filter. The tooth region is considered to be substantially white. For reference, the color difference average coordinates of the hue of the plaque region in FIG. 7B were (0.033, 0.071). Note that these color difference average coordinates of the hue are for reference only and may vary depending on the intensity of the blue light and the sensitivity of the camera.

[0115] Similarly, Figure 7C shows an example of color difference data between the plaque region and the tooth region after exposure control processing and white balance adjustment processing when light including the wavelength range of blue light is irradiated onto the tooth using a blue light cut filter. The tooth region is considered to be substantially white. For reference, the color difference average coordinates of the hue of the plaque region in Figure 7C were (0.033, 0.071). Note that these color difference average coordinates of the hue are for reference only and may vary depending on the intensity of blue light and the sensitivity of the camera.

[0116] Similarly, Figure 7D shows an example of color difference data for the plaque region and tooth region after exposure control processing and white balance adjustment processing when light including a wavelength range of blue light and white light are irradiated onto the tooth without a blue light cut filter. The tooth region is considered to be substantially white. For reference, the color difference average coordinates of the hue of the plaque region in Figure 7D were (0.014, 0.048). Note that these color difference average coordinates of the hue are for reference only and may vary depending on the intensity of the blue light and white light and the sensitivity of the camera.

[0117] 7B to 7D, it can be seen that the hue of the plaque region can be displayed in approximately the same reddish pink color regardless of the type of light source used to illuminate the oral cavity.

[0118] (Highlighting of plaque region) Fig. 8 is a diagram showing an example of a fourth RGB image of a front tooth captured in the state shown in Fig. 5. Fig. 9 is a diagram showing another example of a fourth RGB image of a front tooth captured in the state shown in Fig. 5.

[0119] The mobile terminal 70 acquires a first RGB image 200 from the intraoral camera 10 as shown in FIG. 6A.

[0120] Furthermore, the mobile terminal 70 can further perform saturation enhancement processing on the plaque region 211 of the second RGB image 210 to generate a fourth RGB image 220 in which the plaque region 221 is easily distinguishable, as shown in Figure 8.

[0121] On the other hand, the mobile terminal 70 can further replace the plaque area 211 of the second RGB image 210 with a predetermined pattern to generate a fourth RGB image 230 in which the plaque area 231 has been replaced with a predetermined pattern, as shown in Figure 9.

[0122] (Image Processing Flow) FIG. 10 is a flowchart of image processing in the mobile terminal 70.

[0123] The mobile terminal 70 acquires a first RGB image from the intraoral camera 10 (S111).

[0124] Next, the mobile terminal 70 acquires color data indicating the color of the tooth region of the photographed subject (S112). The acquired color data indicates the color of the tooth region of the photographed subject in a state where no plaque is present. In other words, the color data is data based on the color previously acquired for the tooth of the photographed subject.

[0125] Next, the portable terminal 70 generates a third RGB image by performing exposure control processing on the first RGB image (S113).

[0126] Next, the portable terminal 70 performs white balance adjustment processing on the third RGB image to generate a second RGB image (S114).

[0127] Next, the mobile terminal 70 identifies the plaque region in the second RGB image (S115).

[0128] Next, the mobile terminal 70 generates a fourth RGB image by performing saturation enhancement processing on the plaque region of the second RGB image or by replacing the plaque region with a predetermined pattern (S116).

[0129] [Effects] As described above, the image processing device (e.g., mobile terminal 70) according to this embodiment includes acquisition unit 101, generation unit 102, and storage unit 104. Acquisition unit 101 acquires a first RGB image obtained by photographing teeth and dental plaque that are undergoing a fluorescent reaction by irradiating the teeth with light including a wavelength range of blue light. Storage unit 104 stores color data indicating the color of the tooth region being photographed. Generation unit 102 generates a second RGB image by performing image processing, including first image processing, on the first RGB image. The first image processing is a process of adjusting the gain of at least two of the red, green, and blue color components so that the difference between the acquired color data and color data based on a first red pixel average value of multiple red pixel values ​​held by multiple first pixels in a specific region, which is a tooth region within the RGB image to be processed that is a tooth region free of plaque, a first green pixel average value of multiple green pixel values ​​held by multiple first pixels, and a first blue pixel average value of multiple blue pixel values ​​held by multiple first pixels, falls within a predetermined range.

[0130] This allows for image processing according to the color of the tooth being photographed, as the gain of the color component is adjusted based on the difference between color data based on pixels in a specific region, which is a region of the tooth where plaque is not present, and color data indicating the color of the tooth region being photographed, thereby enabling for accurate identification of the plaque region in the tooth image.

[0131] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, the specific region is a region of the tooth that has multiple pixels in which the green pixel value of each sub-pixel is the maximum and is equal to or greater than a predetermined value.

[0132] Therefore, it is possible to identify a region of the tooth that is likely to be free of plaque as the specific region.

[0133] Furthermore, in the image processing device (e.g., the mobile terminal 70) according to this embodiment, the generation unit 102 further generates an HSV image by converting the color space of the second RGB image into an HSV space, identifies specific pixel areas in which one or more fourth pixels of the multiple fourth pixels in the HSV image are located, the fourth pixels having at least one of a saturation within a first predetermined range, a hue within a second predetermined range, and a brightness within a third predetermined range, and generates a third RGB image by performing an enhancement process on the specific pixel areas in the second RGB image.

[0134] This allows a specific pixel region in the second RGB image to be identified as a plaque region, and saturation enhancement processing is performed on the specific pixel region, thereby generating a third RGB image in which the plaque region is easily distinguished, thereby making it easy to identify the plaque region in the tooth image.

[0135] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, color data is acquired based on a captured image of the teeth, so that color data of the teeth can be easily acquired by capturing an image of the teeth.

[0136] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, the captured image is an image obtained by capturing an image with a camera included in mobile terminal 70. Therefore, by capturing an image of the teeth with the camera included in mobile terminal 70, tooth color data can be easily obtained.

[0137] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, the captured image is an image obtained by capturing an image of the oral cavity with intraoral camera 10. Therefore, by capturing an image of the teeth with intraoral camera 10, tooth color data can be easily obtained.

[0138] In the image processing device (e.g., the mobile terminal 70) according to this embodiment, the color data indicates the tooth color corresponding to the identifier designated by the user in the shade guide indicating the tooth color, so that the tooth color data can be obtained based on the shade guide without capturing an image.

[0139] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, an HSV image is generated by converting the color space of the second RGB image into an HSV space, a specific pixel area is identified in which one or more fourth pixels of the multiple fourth pixels in the HSV image that satisfy at least one of the following conditions are located: saturation within a first predetermined range, hue within a second predetermined range, and brightness within a third predetermined range; and a fourth RGB image is generated by replacing the specific pixel area in the second RGB image with a predetermined pattern.

[0140] This allows a specific pixel area in the second RGB image to be identified as a plaque area and then replaced with a predetermined pattern, thereby generating a fourth RGB image in which the plaque area can be more easily distinguished, thereby making it easier to identify the plaque area in the tooth image.

[0141] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, the image processing further includes a second image processing. The second image processing is processing for determining a gain so that the average value of a plurality of luminance values ​​calculated from a plurality of second pixel values ​​possessed by a plurality of second pixels included in the RGB image to be processed becomes a predetermined value, and for generating a third RGB image by applying the determined gain to the RGB image to be processed. The first image processing is processing for the third RGB image to be processed.

[0142] According to this, since the exposure control process is performed by adjusting the gain of the luminance values ​​of the plurality of second pixels, the conditions of the plurality of luminance values ​​can be kept constant even if the shooting conditions vary. In other words, the conditions of the luminance distribution of the third RGB image to be processed in the first image process can be kept constant regardless of the shooting conditions, so that the first image process can be performed more effectively.

[0143] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, the average value of multiple brightness values ​​is the average value of the color component having the maximum average value among the red component, green component, and blue component of the first RGB image.

[0144] Furthermore, in the image processing device (e.g., the mobile terminal 70) according to this embodiment, multiple luminance values ​​are obtained by calculating, for each of multiple pixel values, the red pixel value, the green pixel value, and the blue pixel value contained in the pixel value.

[0145] Furthermore, in the image processing device (e.g., mobile terminal 70) according to this embodiment, the second pixels are pixels among the third pixels constituting the first RGB image, which satisfy the following conditions: the pixel value of the largest color component is smaller than the first threshold value, and the pixel value of the smallest color component is equal to or smaller than the second threshold value.

[0146] Therefore, the second image processing can be performed by excluding areas in the first RGB image that are strongly affected by reflection of the irradiated light.

[0147] [Modifications] Modifications of the above-described embodiment will now be described.

[0148] (Variation 1) In the above embodiment, the generation unit 102 performs exposure control processing on the first RGB image and white balance adjustment processing on the third RGB image generated by the exposure control processing. However, this is not limited to this, and the exposure control processing does not have to be performed. For example, if a first RGB image with reduced variations in luminance distribution is obtained, the exposure control processing does not have to be performed. For example, the variations in luminance distribution of the obtained first RGB image may be reduced by controlling lighting so that the shooting conditions are constant.

[0149] (Variation 2) In the above embodiment, when a user photographs the inside of the oral cavity using the intraoral camera 10, guidance for specifying the intraoral position to photograph may be displayed on the display unit 103 of the mobile terminal 70. The guidance may be output as audio from a speaker (not shown) of the mobile terminal 70. This allows the user to move the intraoral camera 10 to the specified intraoral position in accordance with the guidance so that photographing is performed at the specified intraoral position. For example, the intraoral position may be the position of a front tooth, a back tooth, or the like.

[0150] The mobile terminal 70 may then store the intraoral position indicated by the guidance in association with the second RGB image or the fourth RGB image generated by image processing performed on the first RGB image acquired while the intraoral position was being indicated.

[0151] Therefore, the intraoral position of the tooth included in the second RGB image or the fourth RGB image can be easily identified.

[0152] (Variation 3) In the above embodiment, the intraoral camera 10 transmits the first RGB image to the mobile terminal 70, and image processing is performed on the first RGB image in the mobile terminal 70. However, this is not limited to this. The first RGB image may be transmitted to the cloud server 80, the cloud server 80 performs the image processing, and the second RGB image or the fourth RGB image resulting from the image processing may be transmitted to the mobile terminal 70. In this case, the first RGB image may be transmitted from the intraoral camera 10 to the cloud server 80 without passing through the mobile terminal 70, or may be transmitted from the intraoral camera 10 to the cloud server 80 via the mobile terminal 70.

[0153] (Variation 4) In the above embodiment, the white balance adjustment process (first image processing) extracts a tooth region from the RGB image to be processed, and adjusts the gains of at least two color components among the red, green, and blue components of the RGB image to be processed so that the first average red pixel value, the first average green pixel value, and the first average blue pixel value in the tooth region are equal. However, this is not limited to this. The pixel average values ​​of each color targeted for gain adjustment are not limited to the average values ​​in the tooth region, and may be the average values ​​in the natural tooth region.

[0154] The process of detecting the natural tooth region will be described below with reference to Figures 11 to 15. Figure 11 is a flowchart showing the details of the process of detecting the natural tooth region.

[0155] Fig. 12 is a diagram showing an example of an RGB image (first RGB image). The RGB image shown in Fig. 12 is an image of a natural tooth 241, an artificial tooth 242, and gums 243 (gums). The artificial tooth 242 is, for example, a prosthesis made of a metal such as gold or silver. The natural tooth 241 is a natural tooth, and is the portion of the tooth excluding the artificial tooth 242.

[0156] First, the generating unit 102 detects a first natural tooth region using the RGB image (first RGB image) (S201). Specifically, the generating unit 102 detects, as the first natural tooth region, a region in the RGB image that satisfies both a first condition that the green pixel value (G) is equal to or greater than a predetermined first threshold and a second condition that the differences between the green pixel value (G), the red pixel value (R), and the blue pixel value (B) are less than a predetermined third threshold.

[0157] When excitation light (blue light) is irradiated onto a natural tooth, excitation fluorescence is emitted from the dentin. This excitation fluorescence passes through the enamel, causing the natural tooth to fluoresce green. Furthermore, when blue light is irradiated onto a filling from a caries treatment, it appears dark (low brightness) in an image captured by a camera, unlike when white light is irradiated. On the other hand, natural teeth covered with enamel appear bright (high brightness) in the image. Therefore, by extracting this green fluorescence according to the first condition, it is possible to identify the area of ​​the natural tooth and exclude the area of ​​the artificial tooth.

[0158] As described above, artificial teeth, such as prosthetic teeth, have low brightness, resulting in small red, green, and blue pixel values ​​in the artificial tooth region of an RGB image. Therefore, if AR and AB are the white balance gains by which the red and blue pixel values ​​are multiplied to match the green pixel value, then AR = G / R AB = G / B. Because the red, green, and blue pixel values ​​are small in the artificial tooth region, the gain variation increases when the data values ​​vary, resulting in a large hue variation in the natural tooth, which is a high-brightness region. In other words, the natural tooth, which should be displayed in white, varies significantly toward red and blue. Therefore, by extracting the natural tooth region excluding the artificial tooth region as the tooth region and performing white balance adjustment processing, the white balance can be adjusted more appropriately.

[0159] Specifically, the second condition is (1) the absolute value (abs(R-G)) of the difference between the red pixel value (R) and the green pixel value (G) is less than a threshold value rg_th, (2) the absolute value (abs(G-B)) of the difference between the green pixel value (G) and the blue pixel value (B) is less than a threshold value gb_th, and (3) the absolute value (abs(B-R)) of the difference between the blue pixel value (B) and the red pixel value (R) is less than the threshold value gb_th. Note that the threshold values ​​rg_th, gb_th, and br_th may be the same value or different values.

[0160] Furthermore, in the first condition, a luminance value (Y) may be used instead of the green pixel value (G). The luminance value (Y) is calculated using the above-described formula 3. For example, an area where the luminance value is equal to or greater than a predetermined first threshold may be detected as a natural tooth area.

[0161] As shown in Equation 3, the ratio of green pixel values ​​to luminance values ​​is large, so that detection using luminance values ​​can be performed in the same manner as when green pixel values ​​are used.

[0162] Although the case where both the first condition and the second condition are used has been described here, only one of the first condition and the second condition may be used. For example, only the first condition may be used.

[0163] Fig. 13 is a diagram showing an example of a detected first natural tooth region 251. Note that, for the sake of explanation, Fig. 13 illustrates the shapes of teeth, etc. in addition to the first natural tooth region 251, but the detected information does not necessarily include the shapes of teeth, etc. This also applies to Figs. 14 and 15.

[0164] 13, a region including the region of the natural tooth 241 and the region of the gum 243 is detected as a first natural tooth region 251. In other words, the first natural tooth region 251 does not include the region of the artificial tooth 242.

[0165] Note that, although an example is shown here in which the first natural tooth region 251 includes the entire area of ​​the gums 243, depending on the photographing conditions, there may be cases in which only a portion of the area of ​​the gums 243 is included in the first natural tooth region 251, or in which the area of ​​the gums 243 is not included in the first natural tooth region 251.

[0166] Next, the generation unit 102 generates an HSV image by converting the color space of the RGB image (first RGB image) into an HSV space (S202). Next, the generation unit 102 detects a gingival region using the HSV image (S203). Specifically, the generation unit 102 detects a region whose hue (H) falls within a predetermined range as the gingival region.

[0167] Fig. 14 is a diagram showing an example of a detected gum region 252. As shown in Fig. 14, the region of the gums 243 is detected as the gum region 252. Note that, although an example in which the gum region is detected is shown here, lips and the like may also be detected as the gum region 252 in addition to the gums.

[0168] Next, the generating unit 102 determines the region obtained by excluding the gingival region 252 from the first natural tooth region 251 as the second natural tooth region 253 (S204). Fig. 15 is a diagram showing an example of the second natural tooth region 253. As shown in Fig. 15, the second natural tooth region 253 includes the region of the natural tooth 241, but does not include the regions of the artificial tooth 242 and the gingival region 243.

[0169] Next, the generating unit 102 determines the third natural tooth region by excluding small regions from the second natural tooth region 253 (S205). Here, a small region is, for example, a region whose area is smaller than a predetermined value. This prevents high-intensity regions other than the natural tooth region (e.g., the gum region) from being mistakenly determined as the natural tooth region.

[0170] Next, the generating unit 102 determines a fourth natural tooth region by expanding the third natural tooth region (S206). Specifically, the generating unit 102 determines the fourth natural tooth region by expanding the boundary of the third natural tooth region outward by a predetermined amount. This allows the boundary between the tooth and gum, etc., to be added to the natural tooth region, preventing excessive display of plaque regions.

[0171] The generating unit 102 finally detects the fourth natural tooth area as the natural tooth area to be used in the white balance adjustment process.

[0172] In this way, the plurality of second pixels constituting the natural tooth region may be set as the plurality of first pixels within the specific region.

[0173] (Variation 5) In the above embodiment, the image data obtained by the intraoral camera 10 is an RGB image. However, it may be a CMYG image obtained using a CMYG complementary color filter of four colors: cyan, magenta, yellow, and green. The CMYG image may be converted into an RGB image by calculation using an approximation formula, and the mobile terminal 70 may perform image processing on the converted RGB image. The conversion from the CMYG image to the RGB image may be performed by the intraoral camera 10, the mobile terminal 70, or another information processing device.

[0174] As a method for calculating each RGB value from each CMYG value, for example, the following calculation formulas (Formulas 5 to 9) can be used.

[0175] Mg ≈ R + B (Equation 5) Ye ≈ R + G (Equation 6) Cy ≈ G + B (Equation 7) (Mg + Ye - Cy) / 2 ≈ {(R + B) + (R + G) - (G + B)} / 2 = R (Equation 8) (Mg + Cy - Ye) / 2 ≈ {(R + B) + (G + B) - (R + G)} / 2 = B (Equation 9) Here, Mg represents magenta, Ye represents yellow, and Cy represents cyan.

[0176] Although the intraoral camera system according to the embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment.

[0177] For example, although the above description has been given of an example in which the intraoral camera 10 is primarily intended to photograph teeth, the intraoral camera 10 may be an oral care device equipped with a camera, such as an oral irrigator equipped with a camera.

[0178] Furthermore, each processing unit included in the intraoral camera system according to the above embodiment is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0179] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0180] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0181] The present disclosure may also be realized as an image display method or the like executed by an intraoral camera system. The present disclosure may also be realized as an intraoral camera included in the intraoral camera system, a mobile terminal, or a cloud server.

[0182] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0183] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0184] Although the intraoral camera system according to one or more aspects has been described based on the embodiments, the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the present embodiments and forms constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0185] The present disclosure is applicable to intraoral camera systems.

[0186] 10 Intraoral camera 10a Head portion 10b Handle portion 10c Neck portion 12 Imaging optical system 14 Imaging element 26A First LED 26B Second LED 26C Third LED 26D Fourth LED 36, 40 Actuator 50 Central control unit 52 Image processing unit 54 LED control unit 56 Lens driver 58 Wireless communication module 60 Power supply control unit 62 Controller 64 Memory 66 Battery 68 Coil 69 Charger 70 Portable terminal 72 Touch screen 80 Cloud server 101 Acquisition unit 102 Generation unit 103 Display unit 104 Memory unit 200 First RGB image 201, 211, 221, 231 Plaque area 210 Second RGB image 220, 230 Fourth RGB image 241 Natural tooth 242 Artificial tooth 243 Gums 251 First natural tooth region 252 Gums region 253 Second natural tooth region

Claims

1. An image processing method comprising: irradiating the teeth with light including a wavelength range of blue light, photographing the teeth and dental plaque that are undergoing a fluorescent reaction, thereby obtaining a first RGB image; obtaining color data that indicates the color of the tooth region being photographed; and generating a second RGB image by performing image processing including a first image processing on the first RGB image, wherein the first image processing is a process of adjusting the gain of at least two of the red, green, and blue color components so that the difference between the obtained color data and color data that is based on a first red pixel average value of multiple red pixel values ​​of multiple first pixels in a specific region that is the tooth region of the RGB image being processed, a first green pixel average value of multiple green pixel values ​​of the multiple first pixels, and a first blue pixel average value of multiple blue pixel values ​​of the multiple first pixels falls within a predetermined range.

2. The image processing method according to claim 1, wherein in the first image processing, an area surrounded by a contour that is a boundary where the brightness of the RGB image to be processed changes from pixels darker than a predetermined brightness to pixels brighter than the predetermined brightness is detected as the specific area.

3. The image processing method of claim 1, wherein in the first image processing, a first region among the plurality of RGB pixels constituting the specific region, which is (i) a region in which the luminance value is equal to or greater than a predetermined first threshold, or (ii) a region in which the green pixel value is equal to or greater than a predetermined second threshold, is detected as a natural tooth region, and a plurality of second pixels included in the first region are set as the plurality of first pixels.

4. The image processing method according to claim 3, wherein in the first image processing, an area obtained by excluding areas of plaque and tartar from the first area is detected as the natural tooth area.

5. The image processing method according to claim 1, wherein the specific region has as the plurality of first pixels a plurality of pixels among the plurality of second pixels constituting the tooth region, each of which has a red pixel value, a green pixel value, and a blue pixel value, the green pixel value of which is the largest and is equal to or greater than a predetermined value.

6. The image processing method according to claim 5, further comprising: generating an HSV image by converting the color space of the second RGB image into an HSV space; identifying a specific pixel area in which one or more fourth pixels of the plurality of fourth pixels in the HSV image are located, the one or more fourth pixels having at least one of a saturation within a first predetermined range, a hue within a second predetermined range, and a brightness within a third predetermined range; and performing an enhancement process on the specific pixel area in the second RGB image to generate a third RGB image.

7. The image processing method according to claim 5, further comprising: generating an HSL image by converting the color space of the second RGB image into an HSL space; identifying a specific pixel area in which one or more fourth pixels of the plurality of fourth pixels in the HSL image are located, the one or more fourth pixels having at least one of a saturation within a first predetermined range, a hue within a second predetermined range, and a luminance within a third predetermined range; and performing an enhancement process on the specific pixel area in the second RGB image to generate a third RGB image.

8. An image processing method according to any one of claims 1 to 7, wherein the color data is acquired based on a photographed image obtained by photographing the tooth.

9. The image processing method according to claim 8, wherein the captured image is an image obtained by capturing an image with a camera provided in a mobile terminal.

10. The image processing method according to claim 8, wherein the captured image is an image obtained by capturing an image of the inside of the oral cavity with an intraoral camera.

11. An image processing method according to any one of claims 1 to 7, wherein the color data indicates a tooth color corresponding to an identifier designated by a user from a shade guide indicating tooth colors.

12. An image processing device comprising: an acquisition unit that acquires a first RGB image obtained by irradiating the tooth with light including a wavelength range of blue light and photographing the tooth and dental plaque that are undergoing a fluorescent reaction; a memory unit that stores color data indicating the color of the tooth region; and a generation unit that generates a second RGB image by performing image processing including a first image processing on the first RGB image, wherein the first image processing is a process that adjusts the gain of at least two color components out of the red, green, and blue components so that the difference between the acquired color data and color data based on a first red pixel average value of multiple red pixel values ​​of multiple first pixels in a specific region that is the tooth region of the RGB image to be processed, a first green pixel average value of multiple green pixel values ​​of the multiple first pixels, and a first blue pixel average value of multiple blue pixel values ​​of the multiple first pixels falls within a predetermined range.

13. A program for causing a computer to execute the image processing method according to any one of claims 1 to 7.