Image processing device, control method, and program

Abstract

This improves visibility when switching between and displaying multiple images from different perspectives. [Solution] The image processing device includes detection means for detecting the position of teeth from images of multiple viewpoints including a dental arch; viewpoint switching means for switching the viewpoint of the displayed image to an image of a different viewpoint; identification means for identifying teeth of interest to the user in the displayed image; and control means for displaying the teeth identified by the identification means before viewpoint switching in a manner that allows the user to continue observing them even after viewpoint switching.

Description

Technical Field

[0001] The present invention relates to a technique for improving visibility when switching and displaying a plurality of images with different viewpoints.

Background Art

[0002] In the five-image method, which is one of the methods for photographing the inside of a patient's oral cavity in dentistry, five oral cavity images with different viewpoints are photographed, and a dentist examines while referring to the oral cavity images. Patent Document 1 describes a method of arranging and displaying a three-dimensional image obtained by photographing a specific tooth inside a patient's oral cavity with a stereo camera and two-dimensional images such as a planar image and a cross-sectional image based on the three-dimensional image.

Prior Art Documents

Patent Documents

[0003] [[ID=二十一]]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, alignment of the teeth in the three-dimensional image and the two-dimensional image is performed based on the positional relationship between the camera and the patient. However, for images with different viewpoints taken by the five-image method, since the positional relationship between each viewpoint and the camera is unknown, when the user switches and displays a plurality of images with different viewpoints, the user has to search for the teeth that were observed before the viewpoint switch from the image after the viewpoint switch.

[0005] The present invention has been made in view of the above problems, and an object thereof is to realize a technique for improving visibility when switching and displaying a plurality of images with different viewpoints.

Means for Solving the Problems

[0006] To solve the above problems and achieve the objective, the image processing apparatus of the present invention includes: detection means for detecting the position of teeth from images of multiple viewpoints including a dental arch; viewpoint switching means for switching the viewpoint of the displayed image to an image of a different viewpoint; identification means for identifying teeth of interest to the user in the displayed image; and control means for displaying the teeth identified by the identification means before viewpoint switching in a manner that allows the user to continue observing them even after viewpoint switching. [Effects of the Invention]

[0007] According to the present invention, visibility can be improved when switching between and displaying multiple images with different viewpoints. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram illustrating the hardware and functional configurations of the image processing device according to Embodiments 1 and 2. [Figure 2] A diagram illustrating the initial screen immediately after launching the application according to Embodiments 1 and 2. [Figure 3] A flowchart showing the control process based on the initial screen according to Embodiments 1, 2, and 3. [Figure 4] A diagram illustrating the application screen after image registration according to Embodiments 1, 2, and 3. [Figure 5] A diagram illustrating application screens in the standalone image display mode according to Embodiments 1, 2, and 3. [Figure 6] A diagram showing the correspondence between the viewpoint switching buttons and images in Embodiments 1, 2, and 3. [Figure 7] A flowchart illustrating the control process in the standalone image display mode according to Embodiment 1. [Figure 8] A diagram illustrating the application screen before viewpoint switching according to Embodiment 1. [Figure 9] A flowchart illustrating the control process in the standalone image display mode according to Embodiment 2. [Figure 10] A diagram illustrating the application screen before and after viewpoint switching according to Embodiment 2. [Figure 11] A flowchart illustrating the control process in the standalone image display mode according to Embodiment 3. [Figure 12] A diagram illustrating the application screen before and after viewpoint switching according to Embodiment 3. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] [Embodiment 1] Embodiment 1 describes an example in which, when a user observes oral images taken using the five-image method, and one of the oral images taken using the five-image method is displayed in an enlarged state, the user switches to an image from a different viewpoint, and the tooth the user is focusing on is enlarged and centered on the screen.

[0011] <Device configuration> First, with reference to Figure 1, the configuration and functions of the image processing apparatus according to this embodiment will be described.

[0012] Figure 1(a) is a block diagram illustrating the hardware configuration of the image processing device according to this embodiment.

[0013] Each function of the image processing apparatus in this embodiment is realized by the hardware of the information processing apparatus 100 and the application software (hereinafter referred to as "application") 150 that operates on the information processing apparatus 100.

[0014] The information processing device 100 includes personal computers (desktop PCs, notebook PCs), tablet PCs, smartphones, and cloud computers.

[0015] The information processing apparatus 100 includes a control unit 101, a non-volatile memory 102, a work memory 103, a recording medium 104, an operation unit 105, and a display unit 106, and each unit is connected so that data can be transmitted and received via an internal bus 107.

[0016] The control unit 101 includes a processor (CPU) that performs arithmetic processing and control processing of the information processing apparatus 100.

[0017] The non-volatile memory 102 is a flash ROM in which data can be rewritten, and stores an application 150, an OS, other software programs executed by the control unit 101, and parameters necessary for the operation of the application 150 described later.

[0018] The work memory 103 is a RAM into which a program read from the non-volatile memory 102, constants, variables, etc. for executing the program are loaded. Also, the work memory 103 is used as a display memory that temporarily stores oral cavity images and display data of a GUI (Graphical User Interface) output to the display unit 106.

[0019] The control unit 101 controls each component of the information processing apparatus 100 by loading and executing the program stored in the non-volatile memory 102 into the work memory 103.

[0020] The recording medium 104 can be read from and written to by the control unit 101. The recording medium 104 is, for example, a built-in memory included in the information processing apparatus 100, a solid state drive (SSD), a hard disk drive (HDD), an optical disk drive, etc. that are detachably connected to the information processing apparatus 100, and stores an application 150, image files, etc.

[0021] The operation unit 105 is an operating component such as a keyboard, mouse, or touch panel that receives various operations from the user and outputs operation information to the control unit 101.

[0022] The display unit 106 displays oral images and the GUI (Graphical User Interface) of the application 150. The display unit 106 is a display device such as a liquid crystal display or an organic EL display. The display unit 106 may be integrated into the information processing device 100 or it may be an external device connected to the information processing device 100.

[0023] Figure 1(b) is a block diagram illustrating the functional configuration of the image processing apparatus according to this embodiment.

[0024] Application 150 has software for utilizing the basic functions of the OS installed on the information processing device 100. The OS of the information processing device 100 may also have software for implementing the control processing in this embodiment.

[0025] Application 150 includes a position / state detection unit 151, a viewpoint switching unit 152, a tooth identification unit 153, and a display format changing unit 154.

[0026] The position / state detection unit 151 detects the position and state of teeth from oral images from multiple viewpoints, including the dentition. The oral images are visible light images captured by an imaging device such as a digital camera.

[0027] The viewpoint switching unit 152 switches the currently displayed image to an image with a different viewpoint according to user operation.

[0028] The tooth identification unit 153 identifies the tooth that the user viewing the displayed image is interested in.

[0029] The display format changing unit 154 changes the displayed image to a format that allows the user to continue observing the tooth that the user is interested in, even after switching to an image from a different viewpoint.

[0030] <Application behavior> Figure 2 illustrates the initial screen 200 of application 150 immediately after launch.

[0031] The operating component 201 is a folder selection button.

[0032] Display section 202 is a thumbnail list display area, where thumbnails of image files in the folder selected by the user are displayed in a list along with the file name and the date and time they were taken.

[0033] The display area 203 is an image display area, and its display modes include a five-image display mode and a single-image display mode. Immediately after the application 150 is launched, the display mode of the image display area 203 is the five-image display mode, and five frames for displaying images taken using the five-image method are displayed. The top frame is the occlusal view (maxilla), the bottom frame is the occlusal view (mandible), the center frame is the front view, the right frame is the left lateral view, and the left frame is the right lateral view.

[0034] Figure 3 is a flowchart illustrating the control process of application 150 based on the initial screen 200.

[0035] The process shown in Figure 3 is realized when the control unit 101 executes the application 150. The process shown in Figure 3 starts when the application 150 is launched in the information processing device 100, and the initial screen 200 of Figure 2 is displayed on the display unit 106.

[0036] In step S301, the control unit 101 waits until it receives an operation from the user on the initial screen 200, and if it receives a user operation, it proceeds to step S302.

[0037] In step S302, the control unit 101 determines whether or not a termination operation has been performed by the user. If a termination operation has been performed by the user, the control unit 101 terminates the application 150; otherwise, it proceeds to step S303. The termination operation shall be performed by closing the initial screen 200 of the application 150 or by using a menu.

[0038] In step S303, the control unit 101 determines whether the folder selection button 201 has been operated by the user via touch or mouse cursor. If the folder selection button 201 has been operated by the user, the control unit 101 proceeds to step S304; otherwise, it proceeds to step S307.

[0039] In step S304, the control unit 101 displays a folder selection screen (not shown) on the display unit 106. On the folder selection screen, the user can select a folder from the recording medium 104 or other accessible storage device where image files are stored.

[0040] In step S305, the control unit 101 proceeds to step S306 if a folder is selected by the user, otherwise proceeds to step S301.

[0041] In step S306, the control unit 101 retrieves a list of image files contained in the folder selected in step S305 from the storage location and updates the display content of the thumbnail list display area 202. In this embodiment, an image file is defined as a file with the extension "jpg", "png", or "gif".

[0042] In this embodiment, the folder where the image files are stored is selected in steps S303 to S305, but the method for obtaining a list of image files is not limited to this. For example, the information processing device 100 may obtain image files from an external system such as an image server that it can access by specifying conditions such as patient ID and date.

[0043] In step S307, the control unit 101 determines whether or not an image registration operation has been performed by the user. If an image registration operation has been performed by the user, the control unit 101 proceeds to step S308; otherwise, it proceeds to step S310. The image registration operation is an operation to register an image file to be displayed in the image display area 402. For example, it is an operation to drag and drop one of the thumbnails displayed in the thumbnail list display area 202 into one of the frames in the image display area 203. Alternatively, the image registration operation may be implemented by displaying a dialog box, or by arranging menus or buttons.

[0044] In step S308, the control unit 101 updates the display content of the image display area 203 in response to the image registration operation in step S307. For example, if the thumbnail of "IMG_0001.JPG" in the thumbnail list display area 202 is dragged and dropped into the "upper jaw" area of ​​the image display area 203, the image is updated to "IMG_0001.JPG".

[0045] In this embodiment, the user manually registers images in steps S307 to S308, but automatic registration is also possible. For example, if the user specifies five images, the system may determine the viewpoint of the captured images based on predetermined file naming rules and shooting order, and automatically register them in five frames in the image display area 203. Alternatively, the system may determine the viewpoint of the captured images using machine learning or the like, and automatically register them.

[0046] In step S309, the control unit 101, acting as a position / state detection unit 151, detects the position and number of all teeth in the image displayed in step S308. In this embodiment, the position of the teeth is inferred by inputting the image data into a learning model created using machine learning such as deep learning. As an inference result, the number of each tooth included in the image (e.g., upper right 1st tooth, lower left 8th tooth) and its coordinates (X coordinates of the left and right ends and Y coordinates of the top and bottom ends) are obtained. Alternatively, the position of the teeth may be inferred using other machine learning methods or template matching based on the color and shape of the teeth. Furthermore, if detection fails, the position of each tooth in an average five-image method may be used as a default value.

[0047] Furthermore, inference processing can be performed by an image processing processor such as a GPU (Graphics Processing Unit). A GPU is a processor capable of performing a large number of multiply-accumulate operations and has the computational processing power to perform matrix operations of neural networks and other operations in a short time. In addition, the inference processing may be performed by the processor (CPU) of the control unit 101 and the GPU working together, or by either the processor (CPU) of the control unit 101 or the GPU.

[0048] In step S310, the control unit 101 determines whether an image selection operation has been performed by the user, such as double-clicking on any of the images in the image display area 203. If an image selection operation has been performed, the control unit 101 proceeds to step S311; otherwise, it proceeds to step S301.

[0049] In step S311, the control unit 101 switches the display mode of the image display area 203 to the single image display mode. Details of the control process for the single image display mode will be described later in Figure 7.

[0050] Figure 4 illustrates the screen 400 of application 150 after five images have been registered in step S308 of Figure 3.

[0051] Display area 401 is the thumbnail list display area. It is similar to display area 202 of screen 200 in Figure 2, but the image files displayed in image display area 402 are highlighted by changing their color, allowing the user to easily see that images are registered.

[0052] The display section 402 is the image display area. The screen 400 in Figure 4 is in five-image display mode, and five images are displayed in their respective frames by the image registration operation in step S308 of Figure 3.

[0053] Figure 5 illustrates the screen 500 of application 150 when the display mode of the image display area 203 is switched to the single image display mode in step S311 of Figure 3.

[0054] The operating component 501 is a five-image display mode switching button, which allows switching the display mode from single-image display mode to five-image display mode.

[0055] The display area 502 is an image display area, and the image double-clicked in step S310 of Figure 3 is displayed across the entire area.

[0056] The operating components 503 to 506 are viewpoint switching buttons, which allow the image displayed in the image display area 502 to be switched to an image from a different viewpoint.

[0057] Figure 6 shows the correspondence between the viewpoint switching buttons 503-506 in Figure 5 and the images that can be switched using each button. For example, if the image displayed in the image display area 402 is a frontal view, pressing button 503 will switch to an occlusal view (maxillary) image. The "-" pattern indicates that there is no image to switch to, so the button is disabled by graying it out or similar.

[0058] Figure 7 is a flowchart illustrating the control process of application 150 when the display mode of image display area 203 is single image display mode.

[0059] In step S701, the control unit 101 waits until it receives an operation from the user on the screen 500 in Figure 5. If it receives a user operation, it proceeds to step S702.

[0060] In step S702, the control unit 101 determines whether or not the user has performed a termination operation. If the user has performed a termination operation, the control unit 101 terminates the application 150; otherwise, it proceeds to step S703. The termination operation is performed by closing the screen 500 in Figure 5 or by using a menu.

[0061] In step S703, the control unit 101 determines whether the user has operated any of the viewpoint switching buttons 503 to 506. If the user has operated any of the viewpoint switching buttons 503 to 506, the control unit 101 proceeds to step S704; otherwise, it proceeds to step S710.

[0062] In step S704, the control unit 101 determines whether the image in the image display area 502 is enlarged or not. If the image in the image display area 502 is enlarged, the control unit 101 proceeds to step S705; otherwise, it proceeds to step S708.

[0063] In step S705, the control unit 101 obtains the magnification ratio of the image in the image display area 502. Let the magnification ratio be X%. The control unit 101 also obtains the tooth number N (where N is an integer) of the tooth closest to the center of the displayed image from the detection result of step S309 in Figure 3, as the tooth identification unit 153.

[0064] In step S706, the control unit 101, acting as the tooth identification unit 153, determines whether or not a tooth with the number N exists in the detection results of step S309 in Figure 3 for the image after the viewpoint switch. If it exists, the process proceeds to step S707; otherwise, the process proceeds to step S708.

[0065] In step S707, the control unit 101, acting as a display mode changing unit 154, acquires the image after the viewpoint change and magnifies it by a magnification of X% centered on the coordinates of the tooth with number N. As a result, the image after the viewpoint change is displayed magnified with tooth N at the center, allowing the user to continue focusing on tooth N.

[0066] In step S708, the control unit 101, acting as a viewpoint switching unit 152, acquires the image after the viewpoint has been switched.

[0067] In step S709, the control unit 101 displays the image acquired in step S707 or S708 in the image display area 502.

[0068] In step S710, the control unit 101 determines whether the user has performed any zooming or panning operations in the image display area 502. If zooming or panning operations have been performed in the image display area 502, the control unit 101 proceeds to step S711; otherwise, it proceeds to step S712. Zooming can be performed, for example, by scrolling with the mouse wheel while operating the control key on the keyboard. Panning can be performed by dragging and dropping the image. Alternatively, buttons or scroll bars for zooming and panning may be provided, and these may be operated with the mouse to create a user interface.

[0069] In step S711, the control unit 101 updates the display in the image display area 502 in response to the operation in step S710.

[0070] In step S712, the control unit 101 determines whether the user has operated the five-panel display mode switching button 501. If the five-panel display mode switching button 501 has been operated, the control unit 101 proceeds to step S713; otherwise, it proceeds to step S701.

[0071] In step S712, the control unit 101 switches the display mode of the image display area 502 to the five-image display mode.

[0072] Figure 8 illustrates the screen transitions of application 150 when the viewpoint is switched.

[0073] Figure 8(a) illustrates screen 800a before the viewpoint switch, after the zoom operation has been performed in steps S710 to S711 of Figure 7.

[0074] The rectangular frame 801 indicates the position of tooth N closest to the center of the displayed image, and it may or may not be displayed.

[0075] In the example shown in Figure 8(a), a frontal view image is displayed, and tooth N is assumed to be the upper left canine tooth.

[0076] Figure 8(b) illustrates screen 800b after the viewpoint switch, which is displayed in step S709 of Figure 7.

[0077] The rectangular frame 811 indicates the position of tooth N, and it may or may not be displayed.

[0078] For example, if the viewpoint switching button 503 in Figure 5 is operated in step S703 in Figure 7, the frontal view image is displayed as in Figure 5 before the viewpoint switch, and based on the correspondence in Figure 6, the occlusal view (maxilla) is displayed after the viewpoint switch, as shown in Figure 8(b). In Figure 8(b), the upper left canine tooth, which was displayed in the center of the image in Figure 8(a), is detected from the occlusal view (maxilla) image, and an image magnified at the same magnification as in Figure 8(a) is displayed with it as the center.

[0079] As explained above, according to this embodiment, when one of the oral images captured using the five-image method is displayed in an enlarged view, and the display is switched to an image from a different viewpoint, the same tooth can be displayed in the center of the screen while maintaining the same magnification. As a result, the user can continue to observe the tooth they were focusing on before and after switching viewpoints without losing sight of it.

[0080] [Embodiment 2] In Embodiment 1, an example was described in which, while one of the oral images captured using the five-image method was being displayed in an enlarged view, the display was switched to an image from a different viewpoint, and the same tooth was enlarged and centered on the screen. In this case, the tooth of interest to the user is not necessarily located in the center of the screen. Furthermore, the user may not necessarily want the image to be enlarged after switching viewpoints. In this embodiment, an example is described in which, while one of the oral images captured using the five-image method is being displayed, a tooth specified by the user is highlighted without being enlarged after switching viewpoints, and an example is described in which the appearance of the tooth after switching viewpoints is displayed before switching viewpoints.

[0081] The device configuration according to this embodiment is the same as that shown in Figure 1 of Embodiment 1. Furthermore, the initial screen configuration, control processing based on the initial screen, screen configuration after image registration, screen configuration in single image display mode, and the correspondence between the viewpoint switching button and the image are the same as those shown in Figures 2 to 6 of Embodiment 1.

[0082] Figure 9 is a flowchart illustrating the control process of application 150 when the display mode of image display area 203 is single image display mode.

[0083] Steps S901 to S903 are the same as steps S701 to S703 in Figure 7.

[0084] In step S904, the control unit 101 proceeds to step S905 if any tooth is selected in step S912, which will be described later; otherwise, it proceeds to step S908.

[0085] In step S905, the control unit 101, as the tooth identification unit 153, obtains the number N of the selected tooth.

[0086] Step S906 is the same as step S706 in Figure 7.

[0087] In step S907, the control unit 101, acting as a display mode changing unit 154, acquires the image after the viewpoint change and the position of tooth N in the image after the viewpoint change, and sets the position of the frame to be displayed. As a result, a frame is displayed on tooth N in the switched image, allowing the user to continue to focus on tooth N.

[0088] Steps S908 to S911 are the same as steps S708 to S711 in Figure 7.

[0089] In step S912, the control unit 101, acting as a tooth identification unit 153, determines whether any of the teeth displayed in the image display area 502 have been selected by the user. If a tooth has been selected, the process proceeds to step S913; otherwise, the process proceeds to step S914. The operation of selecting a tooth is, for example, by double-clicking the desired tooth with a mouse. Alternatively, if the display unit 106 is equipped with a gaze detection unit capable of detecting the user's gaze, the system may be configured to select a tooth that has been gazed at for a predetermined period of time.

[0090] In step S913, the control unit 101 updates the display in the image display area 502 according to the tooth selected in step S912.

[0091] Steps S914 to S915 are the same as steps S712 to S713 in Figure 7.

[0092] Figure 10 shows an example of screen transitions in application 150 when the viewpoint is switched while a tooth is selected.

[0093] Figure 10(a) illustrates screen 1000a before the viewpoint switch, after the tooth selection operation has been performed in steps S912-S913 of Figure 9.

[0094] The rectangular frame 1001 shows the position of tooth N selected in step S912 of Figure 9.

[0095] The operating components 1002-1005 are viewpoint switching buttons, and their functions are the same as those of buttons 503-506 in Figure 5.

[0096] The operating component 1006 is a mouse cursor.

[0097] The display section 1007 is a pop-up window. Only when the mouse cursor 1006 is over one of the viewpoint switching buttons 1002 to 1005, a cropped image of tooth N from the image of the viewpoint to be switched will be displayed in the pop-up window. Note that the pop-up window 1007 may have a different configuration. For example, when the mouse cursor 1006 is over the rectangular frame 1001, the pop-up window 1007 may be configured to display cropped images of tooth N from all other viewpoints, and the viewpoint can be switched by clicking on any of these images. Alternatively, the pop-up window 1007 may be configured to be displayed by, for example, right-clicking the mouse.

[0098] In the example shown in Figure 10(a), a frontal view image is displayed, and tooth N is the upper left canine tooth. Figure 10(b) illustrates screen 1000b after the viewpoint switch, which is displayed in step S909 of Figure 9.

[0099] The rectangular frame 1011 indicates the position of tooth N.

[0100] Operating components 1012-1015 are viewpoint switching buttons, and their functions are the same as buttons 1002-1005.

[0101] If the viewpoint switching button 1002 in Figure 10(a) is operated in step S903 in Figure 9, the frontal view image is displayed before the viewpoint switch as shown in Figure 10(a), and the occlusal view (maxilla) is displayed after the viewpoint switch as shown in Figure 10(b), based on the correspondence in Figure 6. In Figure 10(b), the upper left canine tooth selected in Figure 10(a) is detected from the occlusal view (maxilla) image, and a frame is displayed around the detected area.

[0102] As described above, according to this embodiment, when displaying any of the oral images captured using the five-image method, a tooth specified by the user can be highlighted without magnification after switching viewpoints. This allows the user to continue observing the tooth they were focusing on before and after switching viewpoints without losing sight of it. Furthermore, by presenting the appearance of the tooth after switching viewpoints before switching viewpoints, the user can decide whether or not to perform a viewpoint switch.

[0103] [Embodiment 3] Embodiments 1 and 2 described an example in which the position and number of teeth are detected from oral images taken using the five-image method and used for display when switching viewpoints. This embodiment describes an example in which the position and condition of teeth are detected from each image and displayed before and after switching viewpoints. Furthermore, while Embodiment 1 described an example in which the magnification is maintained before and after switching viewpoints, Embodiment 3 describes an example in which the area ratio between the teeth and the screen is maintained.

[0104] The device configuration according to this embodiment is the same as that shown in Figure 1 of Embodiment 1. Furthermore, the initial screen configuration, control processing based on the initial screen, screen configuration after image registration, screen configuration in single image display mode, and the correspondence between the viewpoint switching button and the image are the same as those shown in Figures 2 to 6 of Embodiment 1.

[0105] Figure 11 is a flowchart illustrating the control process of application 150 when the display mode of image display area 203 is single image display mode.

[0106] Steps S1101 to S1104 are the same as steps S901 to S904 in Figure 7.

[0107] In step S1105, the control unit 101 obtains the area ratio S of the selected tooth N occupying the entire image display area 502. Specifically, from the detection result of step S309 in Figure 3, the original area A of the selected tooth N is obtained, and the magnification X% of the image display area 502 and the total area B of the image display area 502 are obtained (area ratio S = (A × X) / B). Note that S may also be the ratio of the height or width of the tooth N area to the entire image display area 502, in addition to the area ratio.

[0108] Step S1106 is the same as step S906 in Figure 9.

[0109] In step S1107, the control unit 101, acting as a display mode changing unit 154, obtains the image after the viewpoint change and the original area A' of tooth N in the image after the viewpoint change, and enlarges it so that the area ratio with the entire image display area 502 becomes S. As a result, the image after the viewpoint change is enlarged and displayed with tooth N at the center, allowing the user to continue to focus on tooth N.

[0110] Steps S1108 to S1112 are the same as steps S908 to S912 in Figure 9.

[0111] In step S1113, the control unit 101, acting as a position / state detection unit 151, detects the state of the tooth selected in step S1112 in images from all viewpoints. In this embodiment, the state of the tooth is inferred by inputting image data into a learning model created using machine learning such as deep learning. As an inference result, the state of the target tooth (caries, prosthesis, implant, gingival recession, etc.) and its coordinates (X coordinates of the left and right ends and Y coordinates of the top and bottom ends) are obtained for each viewpoint. Note that the process of detecting the state of the tooth may also be triggered by other operations, such as when the image is registered in the image display area 502.

[0112] In step S1114, the control unit 101 updates the display in the image display area 502. Details will be described later.

[0113] Steps S1115 to S1116 are the same as steps S914 to S915 in Figure 9.

[0114] Figure 12 illustrates the screen transitions in application 150 when the viewpoint is switched while a tooth is selected.

[0115] Figure 12(a) illustrates screen 1200a before the viewpoint switch, after the tooth selection operation has been performed in steps S1112 to S1114 of Figure 11.

[0116] The rectangular frame 1201 shows the position of tooth N selected in step S1112 of Figure 11.

[0117] The operating components 1202-1205 are viewpoint switching buttons, and their functions are the same as those of 503-506 in Figure 10.

[0118] The rectangular frame 1206 shows the state of tooth N detected in the currently displayed image.

[0119] Indicator 1207 is a character and graphic that indicates the state of tooth N, which is not detected in the currently displayed image but was detected in an image that is not currently displayed. The system may be configured so that clicking on indicator 1207 switches the viewpoint to the image in which the target state was detected.

[0120] In the example in Figure 12(a), a frontal view image is displayed, and tooth N is assumed to be the upper left canine tooth. Furthermore, it is assumed that gingival recession is detected in the frontal view image of the upper left canine tooth, but no caries is detected, and caries is detected in the occlusal view (maxilla) image, but no gingival recession is detected. When the frontal view image is displayed, the location of the gingival recession is indicated by a rectangular frame 1206, and an index 1207 is displayed below the rectangular frame 1201 to indicate that caries detected in a position not visible in the frontal view is visible in the occlusal view (maxilla). Similarly, if there is a condition that is not detected in the frontal view image but is detected in the right lateral view, an index 1207 is displayed to the left of the rectangular frame 1201.

[0121] Figure 12(b) illustrates screen 1200b after the viewpoint switch, which is displayed in step S1109 of Figure 11.

[0122] The rectangular frame 1211 indicates the position of tooth N.

[0123] Rectangular frame 1212 and index 1213 are the same as rectangular frame 1206 and index 1207, respectively.

[0124] If the viewpoint switching button 1202 in Figure 12(a) is operated in step S1103 in Figure 11, the frontal view image is displayed before the viewpoint switch as shown in Figure 12(a), and after the viewpoint switch, the occlusal view (maxilla) is displayed as shown in Figure 12(b), based on the correspondence in Figure 6. In Figure 12(b), the upper left canine tooth selected in Figure 12(a) is detected from the occlusal view (maxilla) image, and an enlarged image is displayed with the detected area at the center so that the ratio of the tooth to the screen area is the same as in Figure 12(a). Furthermore, when the occlusal view (maxilla) is displayed, the location of the caries is indicated by a rectangular frame 1212, and an indicator 1213 is displayed above the rectangular frame 1211 to show that gingival recession detected in a position not visible from the occlusal view (maxilla) is visible from the frontal view.

[0125] As described above, according to this embodiment, when displaying any of the oral images captured using the five-image method, a tooth specified by the user can be enlarged and displayed while maintaining its area ratio relative to the screen after switching viewpoints. This allows the user to continue observing the tooth they were focusing on before and after switching viewpoints without losing sight of it. In addition to showing the state of the tooth detected at the currently displayed viewpoint, the presence of tooth states detected at different viewpoints, along with their directions, can be shown to the user as a candidate viewpoint to switch to in order to observe the state of the tooth.

[0126] In the embodiments described above, control examples for switching between viewpoints of multiple oral images were explained, but the embodiment is not limited to these examples. For example, if the oral images from multiple viewpoints include visible light images taken by an imaging device and X-ray images taken by an X-ray CT scanner, this embodiment can also be applied to control switching between visible light images and X-ray images.

[0127] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0128] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0129] The disclosures herein include the following image processing apparatus, control methods, and programs. [Item 1] A detection means for detecting the position of teeth from images from multiple viewpoints, including the dental arch, A viewpoint switching means that switches the viewpoint of the currently displayed image to an image with a different viewpoint, A means for identifying the tooth that the user is interested in in the displayed image, An image processing apparatus characterized by having a control means that displays the teeth identified by the identification means before the viewpoint switch in a manner that allows the user to continue observing them even after the viewpoint switch. [Item 2] It has a display means for displaying images from multiple viewpoints, The display means is switchable between a first display mode that displays images from multiple viewpoints or a second display mode that displays any one of the images from multiple viewpoints. The image processing apparatus according to item 1, characterized in that when one of the images of the plurality of viewpoints is selected in the first display mode, the control means transitions to the second display mode and displays the selected image. [Item 3] The viewpoint switching means can switch the viewpoint of the image being displayed in the second display mode to an image with a different viewpoint. The image processing apparatus according to item 2, characterized in that when the selected image is displayed enlarged and the view is switched to an image from a different viewpoint, the control means enlarges and displays the teeth identified from the displayed image by the identification means in the center of the screen. [Item 4] The image processing apparatus according to any one of items 1 to 3, characterized in that the identifying means identifies a tooth that is displayed in the position closest to the center of the screen when the displayed image is enlarged, reduced, or moved. [Item 5] The image processing apparatus according to any one of items 1 to 3, characterized in that the identifying means identifies a tooth displayed at the position where the user's gaze is detected. [Item 6] The image processing apparatus according to any one of items 1 to 3, characterized in that the identifying means identifies a tooth selected in the displayed image. [Item 7] The viewpoint switching means is capable of switching the viewpoint of the selected tooth to a different viewpoint. The image processing apparatus according to item 6, characterized in that the control means switches the displayed image to an image of the different viewpoint that includes the selected tooth when the viewpoint of the selected tooth is switched to a different viewpoint. [Item 8] The image processing apparatus according to item 6, characterized in that when the control means switches the viewpoint of the displayed image to an image with a different viewpoint, the tooth selected in the displayed image is displayed in the center of the screen while maintaining the area ratio with respect to the screen. [Item 9] The control means is characterized by highlighting the selected tooth in the displayed image, as described in any one of items 6 to 8. [Item 10] The image processing apparatus according to item 7, characterized in that the control means displays the appearance of the tooth after the viewpoint has been switched, before switching the viewpoint of the selected tooth to a different viewpoint. [Item 11] The detection means is capable of detecting the state of the tooth selected in the displayed image, The image processing apparatus according to any one of items 6 to 10, characterized in that the control means displays the state of teeth detected from the displayed image, and also displays the state of teeth detected from an image from a different viewpoint than the displayed image. [Item 12] The image processing apparatus according to any one of items 1 to 11, characterized in that the images from multiple viewpoints are oral images taken using the five-image method. [Item 13] The images from the aforementioned multiple viewpoints include visible light images and X-ray images captured by an imaging device. The image processing apparatus according to any one of items 1 to 11, characterized in that the viewpoint switching means switches between the visible light image and the X-ray image instead of switching between the images of the multiple viewpoints. [Item 14] A step of detecting the position of teeth from images from multiple viewpoints, including the dental arch, The steps include switching the viewpoint of the currently displayed image to an image with a different viewpoint, The steps include identifying the tooth the user is interested in in the displayed image, A control method for an image processing apparatus, characterized by comprising the step of displaying the teeth identified by the identifying means before the viewpoint switch in a form that allows the user to continue observing them even after the viewpoint switch. [Item 15] A program for causing a computer to function as one of the means of an image processing apparatus described in any of items 1 through 13. [Explanation of Symbols]

[0130] 100...Image processing device, 151...Control unit, 151...Position / state detection unit. 152...Viewpoint switching unit, 153...Target tooth identification unit, 154...Display mode changing unit.

Claims

1. A detection means for detecting the position of teeth from images from multiple viewpoints, including the dental arch, A viewpoint switching means that switches the viewpoint of the currently displayed image to an image with a different viewpoint, A means for identifying the tooth that the user is interested in in the displayed image, An image processing apparatus characterized by having a control means that displays the teeth identified by the identification means before the viewpoint switch in a manner that allows the user to continue observing them even after the viewpoint switch.

2. It has a display means for displaying images from multiple viewpoints, The display means is switchable between a first display mode that displays images from multiple viewpoints or a second display mode that displays any one of the images from multiple viewpoints. The image processing apparatus according to claim 1, characterized in that when one of the images of the plurality of viewpoints is selected in the first display mode, the control means transitions to the second display mode and displays the selected image.

3. The viewpoint switching means can switch the viewpoint of the image being displayed in the second display mode to an image with a different viewpoint. The image processing apparatus according to claim 2, characterized in that when the selected image is displayed enlarged and the view is switched to an image from a different viewpoint, the control means enlarges and displays the teeth identified from the displayed image by the identification means in the center of the screen.

4. The image processing apparatus according to claim 1, characterized in that the identifying means identifies a tooth that is displayed in the position closest to the center of the screen when the displayed image is enlarged, reduced, or moved.

5. The image processing apparatus according to claim 1, characterized in that the identifying means identifies a tooth displayed at the position where the user's gaze is detected.

6. The image processing apparatus according to claim 1, characterized in that the identifying means identifies a tooth selected in the displayed image.

7. The viewpoint switching means is capable of switching the viewpoint of the selected tooth to a different viewpoint. The image processing apparatus according to claim 6, wherein the control means switches the displayed image to an image of the different viewpoint that includes the selected tooth when the viewpoint of the selected tooth is switched to a different viewpoint.

8. The image processing apparatus according to claim 6, characterized in that when the control means switches the viewpoint of the displayed image to an image with a different viewpoint, it displays the tooth selected in the displayed image in the center of the screen while maintaining the area ratio with respect to the screen.

9. The image processing apparatus according to claim 6, characterized in that the control means highlights the selected tooth in the displayed image.

10. The image processing apparatus according to claim 7, characterized in that the control means displays the appearance of the tooth after the viewpoint has been switched, before switching the viewpoint of the selected tooth to a different viewpoint.

11. The detection means is capable of detecting the state of the tooth selected in the displayed image, The image processing apparatus according to claim 6, characterized in that the control means displays the state of the teeth detected from the displayed image, and also displays the state of the teeth detected from an image from a different viewpoint than the displayed image.

12. The image processing apparatus according to claim 1, characterized in that the images from the multiple viewpoints are oral images taken using the five-image method.

13. The images from the aforementioned multiple viewpoints include visible light images and X-ray images captured by an imaging device. The image processing apparatus according to claim 1, characterized in that the viewpoint switching means switches between the visible light image and the X-ray image instead of switching between the images of the plurality of viewpoints.

14. A step of detecting the position of teeth from images from multiple viewpoints, including the dental arch, The steps include switching the viewpoint of the currently displayed image to an image with a different viewpoint, The steps include identifying the tooth the user is interested in in the displayed image, A control method for an image processing apparatus, characterized by comprising the step of displaying the teeth identified before the viewpoint switch in a form that allows the user to continue observing them even after the viewpoint switch.

15. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 13.

Images