Image processing apparatus and image processing method, and program

Abstract

The present invention provides an image processing device that can acquire changes over time for each of several types of features (size, color, roughness, etc.) in the affected area, thereby contributing to a more accurate diagnosis. [Solution] The image processing device 100 uses a first examination image and a second examination image taken of the same affected area at different times to extract the affected area region with the first difference acquisition means 113 and the area outside the affected area with the second difference acquisition means 114. The first difference acquisition means 113 acquires the first difference of the area outside the affected area, and the second difference acquisition means 114 acquires the second difference of the affected area. The third difference acquisition means 115 acquires a third difference, which is the actual difference of the affected area at different times, based on the first and second differences.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] In recent years, informatization has advanced in medical institutions such as hospitals, and an increasing number of medical institutions have introduced electronic medical records for managing patient diagnostic information. On electronic medical records, patient diagnostic histories, medication information, surgical information, affected part images, X-ray images and the like are centrally managed, making it easy to share and reuse data. In addition, the recent increase in storage capacity has made it possible to capture a large number of digital images of affected parts and the like and store them in electronic medical records.

[0003] In such an environment, a usage method of photographing a patient using a digital camera and storing the photographed image in an electronic medical record as a medical care record is frequently performed. In particular, in the fields of dermatology, surgery, and nursing care, wounds, surgical scars, and pressure ulcers (bedsores) of patients are periodically photographed with a digital camera, and temporal changes in the affected parts are observed.

[0004] When observing temporal changes, it is necessary to compare the affected part regions in the images. The affected part itself may change over time due to healing or deterioration, and there exists a conventional technique that uses the periphery of the affected part as a reference for comparison of temporal changes, instead of performing comparison based on the affected part region itself. For example, Patent Document 1 discloses a technique of identifying the affected part region in an image using feature values of a normal skin region around the affected part, and grasping a change in the size of the affected part region.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, while the prior art disclosed in Patent Document 1 can grasp changes in the size of the affected area, it has the problem of overlooking other characteristics of the affected area in skin diseases, such as changes in skin color and roughness. For a doctor to make an accurate diagnosis of a skin disease, it is necessary to grasp not only changes in the size of the affected area, but also changes in skin color and roughness.

[0007] This disclosure has been made in view of the above-mentioned problems, and aims to provide an image processing device that can acquire changes over time for each of several types of features (size, color, degree of roughness, etc.) in the affected area, thereby contributing to a more accurate diagnosis. [Means for solving the problem]

[0008] The image processing apparatus of the present disclosure is an image processing apparatus that uses a first examination image and a second examination image taken of the same affected area at different times, and comprises a first extraction means, a second extraction means, a first difference acquisition means, a second difference acquisition means, and a third difference acquisition means. The first extraction means extracts the affected area included in the first examination image and the second examination image, respectively. The second extraction means extracts the area outside the affected area, other than the affected area, included in the first examination image and the second examination image, respectively. The first difference acquisition means acquires a first difference in feature quantities between the area outside the affected area in the first examination image and the area outside the affected area in the second examination image. The second difference acquisition means acquires a second difference in feature quantities between the affected area in the first examination image and the affected area in the second examination image. The third difference acquisition means acquires a third actual difference of the affected area at different times, based on the first difference and the second difference. [Effects of the Invention]

[0009] According to this disclosure, an image processing device is realized that can acquire changes over time for each of several types of features (size, color, roughness, etc.) in the affected area, thereby contributing to a more accurate diagnosis. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the configuration of the image processing apparatus according to this embodiment. [Figure 2] This flowchart shows the entire process of the image processing method using the image processing apparatus according to this embodiment. [Figure 3] Figure 2 is a flowchart detailing the various processes shown. [Figure 4] This is a schematic diagram related to medical imaging. [Figure 5] Figure 2 is a schematic diagram showing an example of how the selected examination image in S203 is displayed. [Figure 6] This is a schematic diagram illustrating the mechanism for detecting structures on the skin (specific organs or areas) that show little change over time, other than the affected area, in medical images. [Figure 7] This figure shows an example of a table for managing the normal skin color in the normal skin area and the skin color in the affected area. [Figure 8] This is a schematic diagram showing the waveform of skin irregularities in a medical examination image. [Figure 9] This figure shows an example of a correspondence table illustrating the relationship between disease names and change items. [Modes for carrying out the invention]

[0011] The embodiments of this disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential, and the features may be combined in any way. Furthermore, in the drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted. The dimensions and structural details shown in each embodiment are not limited to those shown in the text and drawings.

[0012] [Configuration of the image processing device] The configuration of the image processing apparatus 100 according to this embodiment will be described below. Figure 1 is a block diagram showing the configuration of the image processing device 100 according to this embodiment, where (a) shows an example of the overall configuration of the image processing device 100 and (b) shows an example of the main configuration of the CPU 101.

[0013] As shown in Figure 1(a), the image processing device 100 includes a CPU 101, a display 102, a mouse 103, a keyboard 104, a WIFI module 105, memory 106, and an SSD 107. The display 102 may be external and does not necessarily have to be included in the image processing device 100.

[0014] As shown in Figure 1(a), the CPU 101 controls the entire image processing unit 100 by controlling each component of the image processing unit 100. The CPU 101 also implements the various functions of the image processing unit 100 by loading the program recorded on the SSD 107 into the memory 106 and executing it. Alternatively, instead of the CPU 101 controlling the entire image processing unit 100, multiple hardware components could share the processing to control the entire unit.

[0015] Memory 106 is the working memory required for the operation of CPU 101. SSD 107 stores various programs executed by CPU 101. In addition to programs, SSD 107 also stores data of images of affected areas showing diseases and a database that manages the characteristics of the diseases.

[0016] The WIFI module 105 is responsible for communication with a device separate from the image processing apparatus 100, and in the example of FIG. 1, it performs communication with the digital camera 110. The WIFI module 105 is used when importing an image captured by the digital camera 110 into the image processing apparatus 100.

[0017] The image processing apparatus 100 includes a mouse 103 and a keyboard 104 as input interfaces. The display 102 is a display unit used to display results of programs executed by the CPU 101 and the like.

[0018] As shown in FIG. 1(b), the CPU 101 includes a first extraction unit 111, a second extraction unit 112, a first difference acquisition unit 113, a second difference acquisition unit 114, and a third difference acquisition unit 115. The CPU 101 further includes an automatic selection unit 116, a difference selection unit 117, an actual size association unit 118, a disease name management unit 119, and a disease name reconsideration proposal unit 120. These units are implemented by the functions of the CPU 101.

[0019] In the present embodiment, the image processing apparatus 100 performs various processes using a first medical examination image and a second medical examination image obtained by imaging the same affected part at different points in time. In the image processing method to be described later, the "current medical examination image" is exemplified as the second medical examination image, and the "previous medical examination image", which is a diagnostic image older than the second medical examination image, is exemplified as the first medical examination image.

[0020] The first extraction unit 111 extracts affected part regions respectively included in the first medical examination image and the second medical examination image. The affected part region refers to a region in the medical examination image where the features of the recorded disease are exhibited. The second extraction unit 112 extracts non-affected part regions respectively included in the first medical examination image and the second medical examination image. The non-affected part region refers to a region other than the affected part region where the features of the disease recorded in the medical examination image are exhibited. The non-affected part region may be a structure (organ or site) with little temporal change, or a site where temporal change may occur, depending on the feature that is the target for acquiring a third difference described later.

[0021] The first difference acquisition means 113 acquires the first difference in feature quantities between the area outside the affected area of ​​the first examination image and the area outside the affected area of ​​the second examination image. The second difference acquisition means 114 acquires the second difference in feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image. Feature quantities are predetermined amounts depending on the target for acquiring the third difference, which will be described later. For example, these may be the number of pixels that satisfy predetermined conditions in the first and second examination images, the difference in color values ​​between the first and second examination images, or the degree of graininess between the first and second examination images.

[0022] The third difference acquisition means 115 acquires a third difference, which is the actual difference of the affected area at different points in time, based on the first and second differences. The third difference is the actual difference between the first examination image and the second examination image of the affected area in terms of multiple types of features (including at least two of the size, color, and roughness of the affected area).

[0023] The automatic selection means 116 automatically selects the first examination image before the user selects the first examination image, provided that the second examination image is newer than the first examination image and the second examination image is selected.

[0024] The difference selection means 117 allows selection of which of the multiple types of third differences to display on the display 102. The selected information may also be displayed on an external display device other than the display 102.

[0025] The actual size association means 118 acquires the actual size information of the subject. Assume that the actual size information acquisition means of the digital camera 110 acquires the actual size of the affected area in either the first examination image or the second examination image. In this case, the actual size association means 118 acquires the actual size of the affected area in the other examination image based on the third difference calculated by the third difference calculation means and associates it with the actual size of the affected area in the first examination image.

[0026] The disease name management means 119 is used to set disease names related to the affected area, and when a disease name is set, a third difference representing the characteristics of the disease name is displayed on the display 102. The selected information may also be displayed on an external display device other than the display 102.

[0027] The disease name reconsideration suggestion means 120 prompts the user to reconsider the disease name set in the disease name management means 119 if there are changes other than the characteristics of the disease name set in the disease name management means 119. In this case, a message prompting reconsideration is displayed on the display 102 as appropriate. The selected information may also be displayed on an external display device other than the display 102.

[0028] [Image processing method] The following describes an image processing method using the image processing device 100 according to this embodiment. In this embodiment, two images of a skin disease taken at different dates and times (the current diagnostic image and the previous diagnostic image) are used to measure the amount of change over time in various characteristics of the skin disease (size of the affected area, color of the affected area, and degree of roughness of the affected area), and a process is performed to acquire the differences over time.

[0029] Figure 2 is a flowchart showing the entire process of the image processing method using the image processing device 100. When the measurement process for changes over time is started in S201, the process proceeds to S202, where the examination image obtained during the current examination, in which the skin disease was photographed, is determined. Next, the process proceeds to S203, where the examination image obtained during the previous examination, in which the same skin disease was photographed, is determined. Once the two examination images for comparison are available, the process proceeds to S204, where the amount of change over time of various characteristics of the affected area between the current examination image and the previous examination image is obtained. Once the amount of change over time has been obtained, the process proceeds to S205, where the amount of change over time of the obtained characteristics is managed and utilized. After that, the process proceeds to S206, where the measurement process for the amount of change over time is terminated.

[0030] Details of each of S202 to S205 in Figure 2 will be explained below. Each figure in Figure 3 is a flowchart detailing the various processes shown in Figure 2. Figure 4 is a schematic diagram of a medical examination image, where (a) shows the screen 400 of display 102 and (b) shows an example of saving a medical examination image. The screen 400 in Figure 4(a) displays the selection and display of the medical examination image when calculating changes over time, as well as the amount of change over time of various characteristics.

[0031] (Details of the S202 process) First, let's explain the processing of S202. The current examination image is displayed in the image display area 401 on screen 400 in Figure 4(a). The user presses the file selection button 403 to select the examination image to be displayed in the image display area 401. When the file selection button 403 is pressed, a file selection dialog box appears, allowing the user to select a file from the SSD 107. The user can use the file selection dialog box to specify the examination image they want to use for calculating changes over time from the current examination image.

[0032] When saving medical images, the user designates a specific folder on SSD107 to manage the images. Within that designated folder, a folder representing the patient ID is created. Under the patient ID folder, a folder for the patient's consultation date is prepared, and the medical images taken on that consultation date are saved within the consultation date folder.

[0033] The medical image selected in the file selection dialog launched by the file selection button 403 is displayed in the image display area 401. Additionally, information about the selected medical image is displayed in the image information display area 405.

[0034] Furthermore, if the image processing device 100 and the digital camera 110 are connected using the WIFI module 105, the medical examination image taken by the digital camera 110 can be displayed in the image display area 401, and the information of that medical examination image can be displayed in the image information display area 405. The above is a detailed explanation of the S202 process.

[0035] (Details of the S203 process) Next, we will explain the processing of S203. The previous examination image is displayed in the image display area 402 on screen 400 in Figure 4(a). The user presses the file selection button 404 to select the examination image displayed in the image display area 402. When the file selection button 404 is pressed, a file selection dialog box appears, allowing the user to select a file from the SSD 107. The user can use the file selection dialog box to specify the examination image from the previous examination images that they want to use for calculating changes over time.

[0036] The medical image selected in the file selection dialog launched by the file selection button 404 is displayed in the image display area 402. Additionally, information about the selected medical image is displayed in the image information display area 406.

[0037] If the current examination image is selected, the image processing device 100 may automatically determine the previous examination image before the user presses the file selection button 404. If the current examination image is determined, the automatic selection means 116 can automatically select the most recent examination image of the same affected area taken before the current examination date as the previous examination image. The determination of the same affected area can be done using common techniques such as similar image search. Of course, if the automatically selected examination image is not appropriate, the user can press the file selection button 404 to select a more appropriate image. The above is a detailed explanation of the S203 process.

[0038] If the user executes process S203 before process S202, and in S202 the user attempts to take a medical image with the digital camera 110, the screen of the digital camera 110 may display the medical image selected in S203. An example of the display screen in that case is shown in Figure 5. Image 501 is a thumbnail of the medical image selected in S203.

[0039] (Details of the S204 process) Next, we will explain the process of S204. The details of S204 will be explained using the flowchart in Figure 3(a). When the measurement process for changes over time is started in S301, the process proceeds to S302 to identify the affected area. Next, it proceeds to S303 to calculate the scale between the current examination image and the previous examination image. Next, it proceeds to S304 to calculate the area ratio of the affected area using the current examination image and the previous examination image. Next, it proceeds to S305 to calculate the amount of change in the color of the affected area over time using the current examination image and the previous examination image. Next, it proceeds to S306 to quantify the degree of roughness of the affected area using the current examination image and the previous examination image, and calculate the amount of change over time. After that, it proceeds to S307 to end the measurement process for the amount of change over time.

[0040] Thus, in S204, the temporal changes in various characteristics (size, color, and roughness) of the affected area are acquired. Specifically, the temporal change in the size of the affected area is calculated in S303 and S304, the temporal change in the color of the affected area is calculated in S305, and the temporal change in the roughness of the affected area is calculated in S306.

[0041] Details of each process S302 to S306 in Figure 3(a) will be explained below.

[0042] (Details of the S302 process) When the measurement process for changes over time is started in S301, the process proceeds to S302. In S302, the first extraction means 111 extracts and identifies the affected area that appears in both the current examination image and the previous examination image from each image. The affected area refers to the area in the examination image that exhibits the characteristics of the disease recorded. Areas of normal skin that do not exhibit the characteristics of the disease are not included in the affected area. Incidentally, there is no particular specification for the method of identifying the affected area. AI may be used to segment the affected area. Alternatively, the doctor may specify the affected area in the affected area image during the initial examination, and the affected area in the current examination image and the previous examination image may be determined using the characteristic quantities of that specified affected area. The above details the processing of S302.

[0043] (Details of the S303 process) After S302, proceed to S303 to calculate the scale between the current examination image and the previous examination image.

[0044] Figure 3(b) is a flowchart detailing the process of S303. When the process of calculating the scale of the affected area image is started in S311, the process proceeds to S312. In S312, the second extraction means 112 extracts structures on the skin (predetermined organs or parts) that do not change much over time, other than the affected area, as the non-affected area for the current examination image and the previous examination image. In this extraction process, the user may manually specify the affected area and the second extraction means 112 may extract it, or the second extraction means 112 may automatically specify the affected area and extract it. Here, the user may also modify the results of the affected area extraction by the second extraction means 112 as appropriate.

[0045] Each figure in Figure 6 is a schematic diagram illustrating the mechanism for detecting structures on the skin (specific organs or areas) that show little change over time, other than the affected area, in a medical examination image. Figure 6(a) is a diagram showing a list of skin structures that change little over time and are recognized as targets for extraction by the second extraction means 112. Information on these structures is recorded in the SSD 107. The second extraction means 112 uses the information in Figure 6(a) to extract skin structures that change little over time from the current examination image and the previous examination image. In addition, the image processing device 100 may also record information on skin structures that change little over time and their detection methods in its program, separately from Figure 6(a), and operate the second extraction means 112 based on this information.

[0046] Figure 6(b) shows the results of extracting the affected area and skin structures that show little change over time from the affected area image. An affected area 601 exists on the patient's hand 600. The method for identifying the affected area 501 is the same as the method shown in S302.

[0047] When detecting structures on the skin that do not change much over time, other than the affected area, the second extraction means 112 first refers to the information in Figure 6(a) in the current and previous examination images to recognize the structures on the skin to be extracted. Then, for example, it extracts the structures on the skin that were selected for extraction within the range of the hand 600 in Figure 6(b). When extracting structures on the skin, the affected area, such as the affected area 601, is excluded. In addition, to extract structures on the skin that do not change much over time, AI may be used, or information in which the user has specified the features and location of the objects to be extracted in advance, such as at the initial examination, may be used.

[0048] When extracting skin structures that show little change over time by referring to the list in Figure 6(a), it is possible that multiple types of structures (organs or body parts) may be found simultaneously. If multiple are found, it is advisable to prioritize using those with a low probability of change, such as skin contours or tattoos. Alternatively, a priority order may be noted for each extraction target in the list in Figure 6(a) beforehand, and extraction may be performed in order of priority.

[0049] In Figure 6(b), moles 602, 603, 604, blood vessels 610, and 611 are extracted as target organs. Note that since the size of moles can change, the second extraction means 112 may extract multiple structures, in this case moles, as mole regions 605, for both the current and previous examination images.

[0050] The above is a detailed explanation of the processing in S312. Next, we proceed to S313. In S313, the first difference acquisition means 113 compares the results extracted in S312 and calculates the size ratio between the current examination image and the previous examination image.

[0051] In S313, the first difference acquisition means 113 calculates the number of pixels that the structure (organ or body part) extracted in S312 spans among the pixels constituting each examination image, using the number of pixels as a feature. The first difference acquisition means 113 calculates the ratio of the number of pixels in the current examination image to that of the previous examination image. This ratio of the number of pixels becomes the ratio of the size in the current examination image and the previous examination image. For example, if the structure in the current examination image is 100 pixels and the structure in the previous examination image is also 100 pixels, then both diagnostic images are judged to have captured the structure (organ or body part) at the same size.

[0052] In the above calculation method, the ratio of the two diagnostic images is calculated using the number of pixels, but other calculation methods may also be used. The digital camera 110 may be equipped with a function to measure the length or area of ​​a subject. For example, if the size of a structure (organ or part) is calculated using this function, the first difference acquisition means 113 may also calculate the ratio of the two diagnostic images based on the values ​​obtained by the measurement function of the digital camera 110.

[0053] Once the ratio of both diagnostic images is calculated in S313, the process proceeds to S314, and the calculation of the scale of each diagnostic image is completed. The above is a detailed explanation of the S303 process.

[0054] (Details of processing S304) After S303, proceed to S304 to calculate the area ratio of the affected region using the current examination image and the previous examination image.

[0055] Figure 3(c) is a flowchart detailing the process of S304. When the calculation of the area ratio of the affected area is started in S321, the process proceeds to S322. In S322, the second difference acquisition means 114 calculates the number of pixels in the affected area in the current examination image and the previous examination image as features. The method for identifying the affected area is the same as the method shown in S302.

[0056] After S322, the process proceeds to S323. In S323, the third difference acquisition means 115 acquires the actual amount of change in the size of the affected area over time. For example, even if the size of the affected area is actually decreasing over time, if the digital camera is brought close to the affected area to take a picture, the number of pixels constituting the affected area may increase. Therefore, in S323, the area ratio of the affected area is calculated using the ratio of the sizes of both examination images obtained in S303 and the number of pixels of both examination areas obtained in S322. This area ratio represents the actual amount of change in the size of the affected area over time and indicates the actual difference of the affected area at different points in time between the previous examination and the current examination.

[0057] Specifically, first, the third difference acquisition means 115 calculates the enlargement rate of the affected area from the value obtained by dividing the size of the affected area in the previous diagnostic image by the size of the affected area in the current diagnostic image. Next, the third difference acquisition means 115 divides the enlargement rate of the affected area by the ratio of the sizes of the current examination image and the previous examination image obtained in S303. As a result, the area ratio of the affected area in the current examination image and the previous examination image, that is, the actual amount of change in the size of the affected area over time, is obtained.

[0058] In the calculation method described above, the area ratio of the two affected areas is calculated using the number of pixels, but other calculation methods may also be used. The digital camera 110 may be equipped with a function to measure the length and area of ​​a subject. For example, if the size of a structure (organ or part) is calculated using this function, the third difference acquisition means 115 may also calculate the area ratio of the two affected areas based on the values ​​obtained by the measurement function of the digital camera 110.

[0059] Furthermore, the digital camera 110 may be equipped with a measurement function (actual size information acquisition means) for acquiring information on the actual size of the subject. In this case, the actual size information acquisition means may acquire the actual size of the affected area for one of the diagnostic images, either the previous diagnostic image or the current diagnostic image, and the actual size information may be recorded. In such cases, the actual size information association means 118 acquires the actual size of the affected area image in the other diagnostic image based on the actual amount of change in the size of the affected area over time calculated by the third difference acquisition means 115, and associates it with the actual size of the affected area image in the first diagnostic image. The acquired actual sizes may be displayed in the image display area 401 or the image display area 402, respectively. In this way, by appropriately utilizing the measurement function of the digital camera 110, a pair of diagnostic images can be efficiently acquired and managed.

[0060] After S323, proceed to S324 to complete the calculation of the change in area of ​​the affected region. The above is a detailed explanation of the S304 process.

[0061] (Details of processing S305) After S304, proceed to S305, where the amount of change in the color of the affected area over time is calculated using the current examination image and the previous examination image.

[0062] Figure 3(d) is a flowchart detailing the process of S305. When the process of calculating the amount of change in the color of the affected area over time is started in S331, the process proceeds to S332.

[0063] In S332, the second extraction means 112 extracts areas with normal skin color (normal skin areas) from the current and previous examination images. The normal skin areas are predetermined areas of the patient's skin in the examination images, excluding the affected area obtained in S302 and structures (organs / parts) outside the affected area obtained in S312 that do not change much over time. In this extraction process, the user may manually specify the normal skin areas and have the second extraction means 112 extract them, or the second extraction means 112 may automatically specify and extract the normal skin areas. The user may also modify the results of the normal skin area extraction by the second extraction means 112 as appropriate.

[0064] The above is a detailed explanation of the processing in S332. Next, we proceed to S333. In S333, the first difference acquisition means 113 calculates the difference value of the normal skin color information in the normal skin region between the previous examination image and the current examination image, using the normal skin color information as a feature quantity.

[0065] Figure 7 shows an example of a table for managing normal skin color in the normal skin area and skin color in the affected area. Table 700 organizes skin color information. The information in Table 700 is recorded in SSD107.

[0066] In this embodiment, Table 700 contains six types of skin color information. In Table 700, the first type is the row for "Previous Skin 701," which records the normal skin color information of the normal skin area captured in the previous examination image. The second type is the row for "Previous Affected Area 702," which records the skin color information of the affected area captured in the previous examination image. The third type is the row for "Current Skin 703," which records the normal skin color information of the normal skin area captured in the current examination image. The fourth type is the row for "Current Affected Area 704," which records the skin color information of the affected area captured in the current examination image. The fifth type is the row for "Current Skin - Previous Skin 705," which records the difference between the normal skin color information of the normal skin area captured in the current examination image and the normal skin color information of the normal skin area captured in the previous examination image. The sixth type is the row for "Current Affected Area - Previous Affected Area 706," which records the difference between the skin color information of the affected area in the current examination image and the skin color information of the affected area in the previous examination image.

[0067] Specifically, the first difference acquisition means 113 acquires information to fill in rows 701 and 703 in table 700 from the previous examination image and the current examination image, and acquires the difference value of normal skin color information in the normal skin area in row 705.

[0068] In the example in Table 700, the previous skin tone (701) had values ​​of red=190, green=130, and blue=61. The current skin tone (703) has values ​​of red=210, green=130, and blue=61. Therefore, the difference between the current skin tone and the previous skin tone (705) is red=20, green=0, and blue=0. In this case, it can be seen that the current skin tone was captured with a stronger red color than the previous skin tone.

[0069] The above is a detailed explanation of the processing in S333. Next, we proceed to S334. In S334, the second difference acquisition means 114 calculates the difference value of the color information in the affected area between the previous examination image and the current examination image, using the color information in the affected area as a feature.

[0070] In the example in Table 700, the previous affected area 702 had red=216, green=130, and blue=61. The current affected area 704 has red=236, green=130, and blue=61. Therefore, the difference between the current affected area and the previous affected area 706 is red=20, green=0, and blue=0. In this case, it can be seen that the current affected area appears redder than the previous affected area. The above is a detailed explanation of the processing of S334.

[0071] Next, we proceed to S335. In S335, the third difference acquisition means 115 acquires the actual amount of change in the color of the affected area over time. In the example in Table 700, the third difference acquisition means 115 calculates the actual amount of change in the color of the affected area over time by subtracting the color information value of "current skin - previous skin" 705 from the color information value of "current affected area - previous affected area" 706. This actual amount of change in the color of the affected area over time represents the actual difference of the affected area at different points in time between the previous examination and the current examination.

[0072] In the example in Table 700, the values ​​for red, green, and blue are all 0, meaning the actual change in the color of the affected area over time is 0. In other words, when comparing only the color components of the affected area, the affected area in the current examination image appears to be redder than the affected area in the previous examination image, but the normal skin area also shows the same amount of increase in redness. Therefore, this means that the current examination image appears redder overall than the previous examination image due to changes in the environment during shooting or changes in camera settings.

[0073] After S335, proceed to S336 to complete the calculation of the amount of color change in the affected area. The above is a detailed explanation of the S305 process.

[0074] (Details of the S306 process) After S305, proceed to S306 to calculate the change in the degree of roughness of the affected area using the current examination image and the previous examination image.

[0075] Figure 3(e) is a flowchart detailing the process of S306. When the calculation of the change in the degree of roughness of the affected area over time is started in S341, the process proceeds to S342.

[0076] In S342, the second extraction means 112 extracts areas representing the degree of roughness of normal skin (normal skin areas) from the current and previous examination images. The normal skin areas are the same as the areas described in S332. In this extraction process, the user may manually specify the normal skin areas and the second extraction means 112 may extract them, or the second extraction means 112 may automatically specify and extract the normal skin areas. The user may also modify the results of the normal skin area extraction by the second extraction means 112 as appropriate.

[0077] Furthermore, since normal skin regions have been extracted from both the current and previous examination images in S332 as described above, the processing in S342 may be omitted, and the results of the normal skin region extraction obtained in S332 may be reused.

[0078] The above is a detailed explanation of the process in S342. Next, we proceed to S343. In S343, the first difference acquisition means 113 calculates the difference value of the degree of roughness of normal skin in the normal skin region between the previous examination image and the current examination image, using the degree of roughness of normal skin in the normal skin region as a feature quantity.

[0079] Figure 3(f) is a flowchart detailing the processing of S343. Each figure in Figure 8 is a schematic diagram showing the waveform of skin irregularities in the examination image. When the calculation of the difference value of the degree of roughness is started in S351, the process proceeds to S352. In S352, before representing the unevenness of the skin as a waveform, the first difference acquisition means 113 adjusts the scale of the current examination image and the previous examination image to be the same. The adjustment is made based on the size ratio between the images obtained in S303. By adjusting the scale, the waveform of the unevenness of normal skin in the normal skin area will be as shown in Figure 8(a) for the previous examination image, and as shown in Figure 8(c) for the current examination image.

[0080] Next, in S353, the first difference acquisition means 113 analyzes the waveform of normal skin in the normal skin region of the current examination image and the previous examination image by frequency decomposition. It is not necessary to decompose it finely; it is sufficient to isolate only the frequency band that represents the characteristics of skin roughness in skin diseases.

[0081] Next, in S354, the first difference acquisition means 113 calculates the difference value as a feature quantity between the frequency of the degree of roughness of normal skin in the normal skin region of the current examination image and the frequency of the degree of roughness of normal skin in the normal skin region of the previous examination image. This difference value is denoted as difference value A.

[0082] Once the difference value A is calculated in S354, the process proceeds to S355, completing the calculation of the scale for each diagnostic image. The above is a detailed explanation of the processing in S343.

[0083] Next, proceed to S344. In S344, the second difference acquisition means 114 uses the degree of roughness in the affected area as a feature quantity and calculates the difference value of the degree of roughness between the previous examination image and the current examination image. The method for calculating the difference value of the degree of roughness at this time is the same as in S342, and in Figure 3(f), the affected area is used as the calculation target instead of the normal skin area when performing each process.

[0084] In S344, the waveform of the unevenness in the affected area is, for example, as shown in Figure 8(b) in the previous examination image, and for example, as shown in Figure 8(d) in the current examination image. Here, the second difference acquisition means 114 calculates the difference value as a feature quantity between the frequency of the degree of roughness in the affected area in the current examination image and the frequency of the degree of roughness in the affected area in the previous examination image. This difference value is denoted as difference value B.

[0085] Next, we proceed to S345. In S345, the third difference acquisition means 115 acquires the actual amount of change over time in the roughness of the affected area. Difference value A represents noise caused by the difference in the shooting environment, obtained using normal skin. Therefore, the third difference acquisition means 115 calculates the actual amount of change over time in the roughness of the affected area by subtracting difference value A from difference value B. This actual amount of change over time in the roughness of the affected area represents the actual difference in the affected area at different points in time between the previous examination and the current examination.

[0086] For example, if the waveform amplitude of the affected area in the previous examination image is larger than that of the affected area in the current examination image, it can be said that the degree of roughness has decreased. Conversely, if the waveform amplitude of the affected area in the previous examination image is smaller than that of the affected area in the current examination image, it can be said that the degree of roughness has increased.

[0087] In the above explanation, the degree of roughness was determined by the amplitude of the waveform, but the digital camera 110 may be equipped with a function that can measure and record the distance to the subject, etc. The third difference acquisition means 115 may also acquire the change in roughness of the affected area between the previous examination image and the current examination image based on the results of measuring the degree of roughness of the skin using this function.

[0088] After S345, proceed to S346 to complete the calculation of the change in the degree of roughness of the affected area. The above is a detailed explanation of the process in S306. After completing the process in S306, the process proceeds to S307, and the processes shown in Figure 3(a), i.e., the process in S204 in Figure 2, are completed.

[0089] As described above, this embodiment enables the acquisition of changes over time for each of several types of features (size, skin color, roughness, etc.) in the affected area, thereby realizing an image processing device 100 that contributes to a more accurate diagnosis.

[0090] In this embodiment, as shown in Figure 4(a), a change display area 410 is displayed on the screen 400 of the display 102. The change display area 410 is provided with checkboxes 411, 412, and 413 that indicate whether or not to display the results of changes over time in various characteristics (size, color, and roughness) acquired for the affected area on the screen 400 of the display 102. Checkbox 411 relates to the display state of the change over time in the size of the affected area, checkbox 412 relates to the display state of the change over time in the color of the affected area, and checkbox 413 relates to the display state of the change over time in the roughness of the affected area. When the user checks one of the checkboxes 411, 412, or 413 to turn it on, the difference selection means 117 displays the size change display item on the screen 400 in the change display area 410. On the other hand, if the user checks one of the checkboxes 411, 412, or 413 to turn it off, the difference selection means 117 clears the display of the size change display item in the change display field 410.

[0091] Thus, in this embodiment, the user can appropriately display only the results of the changes over time of the features they wish to check (size, color, degree of roughness) on the display 102, which is convenient for the user.

[0092] Furthermore, when the mouse hovers over each character of a change item displayed in the change display area 410, the results of the time-dependent changes in the characteristics related to that display item may be displayed on the display 102. For example, consider the case where the mouse hovers over a display item related to the results of the time-dependent changes in size. In this case, the current examination image displayed in the image display area 401 and the previous examination image displayed in the image display area 402 may be redisplayed at the same scale, and the affected area may be filled with, for example, blue. Also, consider the case where the mouse hovers over a display item related to the results of the time-dependent changes in color. In this case, the RGB values ​​may be adjusted so that the normal skin color of the current examination image displayed in the image display area 401 and the previous examination image displayed in the image display area 402 are displayed with the same RGB values.

[0093] The disease name column 420 in Figure 4(a) can display the disease name of the patient who was the subject of the diagnostic image. The disease name to be displayed may be a disease name entered by a physician into the image processing device 100, or it may be a disease name inferred from the diagnostic image by medical AI, for example.

[0094] If a disease name is displayed in the disease name field 420, the characteristic change items for that disease name may be forcibly displayed in the change display field 410. In this case, if a disease name is set in the disease name management means 119, the characteristic change items for that disease name will be displayed in the change display field 410. Change items that are not characteristic of the disease name may not be automatically displayed in the change display field 420. This configuration prevents the omission of necessary change items for the disease name due to user selection, and contributes to an accurate diagnosis of the affected area.

[0095] Figure 9 shows an example of a correspondence table illustrating the relationship between disease names and change items. The information in this correspondence table is recorded, for example, in SSD107. If there are changes other than the characteristics of the disease name set in the disease name column 420, the system can prompt a reconsideration of the diagnosis, as shown in comment 430 in Figure 4(a). In this case, the disease name reconsideration suggestion means 120 will prompt a reconsideration of the disease name set in the disease name management means 119 if there are changes in the affected area other than the characteristics of the disease name set in the disease name management means 119. This configuration prevents inaccurate judgments about the disease name related to the affected area and contributes to an accurate diagnosis of the affected area.

[0096] Furthermore, in addition to the disease name column 420, a medication column may also be displayed on screen 400 as shown in Figure 4(a). In this case, it is also possible to display items that are affected by the type of medication in the higher-level column.

[0097] Note that the sequence of processes S304 and S305, S305, and S306 in Figure 3(a) do not necessarily have to be executed in this order, and the order may be changed as appropriate. Also, it is not necessary to perform all of the sequences of processes S304 and S305, S305, and S306 as needed. For example, it is possible to execute one or two of the sequences of processes S304 and S305, S305, and S306.

[0098] (Other embodiments) In the embodiments described above, a computer program for controlling the image processing device 100 is stored in a storage medium such as an SSD 107. This computer program is a program for realizing the various functions of the CPU 101 of the image processing device 100. This program realizes the functions of the first extraction means 111, the second extraction means 112, the first difference acquisition means 113, the second difference acquisition means 114, and the third difference acquisition means 115. Furthermore, this program realizes functions such as the automatic selection means 116, the difference selection means 117, the actual size association means 118, the disease name management means 119, and the disease name reconsideration suggestion means 120. Specifically, each step corresponds to the program as shown in Figure 2 (S201-S206), Figure 3 (S301-S307, S311-S314, S321-S324, S331-S336, S341-S346, S351-S355), etc. Then, the CPU 101, acting as a computer, reads and executes computer programs from storage media such as the SSD 107. This allows the CPU 101 to control the image processing unit 100.

[0099] While preferred embodiments of this disclosure have been described above, this disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0100] The disclosure of various embodiments includes the following configurations, methods, and programs. (Composition 1) An image processing device that uses a first examination image and a second examination image taken of the same affected area at different times, A first extraction means for extracting the affected area contained in the first and second examination images, respectively, A second extraction means for extracting areas outside the affected area, which are included in the first and second examination images, respectively, A first difference acquisition means for acquiring a first difference in feature quantities between the area outside the affected area of ​​the first examination image and the area outside the affected area of ​​the second examination image, A second difference acquisition means for acquiring a second difference in the feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image, A third difference acquisition means for acquiring the actual third difference of the affected area at different points in time, based on the first difference and the second difference; Having, Image processing device. (Configuration 2) The third difference acquisition means acquires the third differences of multiple types of features, The image processing apparatus described in Configuration 1. (Composition 3) The aforementioned multiple types of characteristics include at least two of the size, color, and degree of roughness of the affected area. The image processing apparatus described in Configuration 2. (Composition 4) The system further includes a difference selection means that can select which of the third differences of the aforementioned multiple types of features to display. The image processing apparatus described in Configuration 2. (Composition 5) The second extraction means uses a region containing multiple structures as the region outside the affected area. An image processing device as described in any one of configurations 1 to 4. (Composition 6) The system further includes a disease name management means for which a disease name is set for the affected area, The disease name management means, when the disease name is set, displays the third difference that represents the characteristics of the disease name. An image processing device as described in any one of configurations 1 to 5. (Composition 7) The system further includes a disease name reconsideration suggestion means that prompts a reconsideration of the disease name set in the disease name management means when changes other than those characteristic of the disease name set in the disease name management means occur in the affected area. The image processing apparatus described in configuration 6. (Composition 8) The second examination image is a more recent image than the first examination image. The system further includes an automatic selection means that automatically selects the first examination image before the user selects the first examination image, while the second examination image is selected. An image processing device as described in any one of configurations 1 to 7. (Composition 9) The automatic selection means searches for the past first examination image that is closest to the selected second examination image. The image processing apparatus described in configuration 8. (Composition 10) A means for acquiring actual size information to obtain the actual size information of the subject, When the actual size information acquisition means acquires the actual size of the affected area in either the first or second examination image, the actual size information association means acquires the actual size of the affected area in the other examination image based on the third difference acquired by the third difference acquisition means and associates it with the actual size of the affected area in the first examination image. It further has, An image processing device as described in any one of configurations 1 to 9. (Method 1) An image processing method using a first examination image and a second examination image taken of the same affected area at different times, The steps include extracting the affected area contained in the first and second examination images, respectively, A step of extracting the area outside the affected area, which is included in the first and second examination images, respectively, The steps include obtaining a first difference in feature quantities between the area outside the affected area in the first examination image and the area outside the affected area in the second examination image, The steps include obtaining a second difference in the feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image, A step of obtaining an actual third difference of the affected area at a different point in time, based on the first difference and the second difference; Having, Image processing methods. (Program 1) Computers, The first and second examination images, taken at different times, of the same affected area, are used. A first extraction means for extracting the affected area contained in the first and second examination images, respectively, A second extraction means for extracting areas outside the affected area, which are included in the first and second examination images, respectively, A first difference acquisition means for acquiring a first difference in feature quantities between the area outside the affected area of ​​the first examination image and the area outside the affected area of ​​the second examination image, A second difference acquisition means for acquiring a second difference in the feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image, A third difference acquisition means for acquiring the actual third difference of the affected area at different points in time, based on the first difference and the second difference. To make it function as program. [Explanation of symbols]

[0101] 100 Image Processing Devices 101 CPU 111 First extraction means 112 First extraction means 113 First difference acquisition means 114 Second difference acquisition means 115 Third difference acquisition means 116 Automatic selection means 117 Means of selecting differences 118 Means of associating actual size 119 Disease name management means 120 Means for proposing reconsideration of disease name

Claims

1. An image processing device that uses a first examination image and a second examination image taken of the same affected area at different times, A first extraction means for extracting the affected area contained in the first and second examination images, respectively, A second extraction means for extracting areas outside the affected area, which are included in the first and second examination images, respectively, A first difference acquisition means for acquiring a first difference in feature quantities between the area outside the affected area of ​​the first examination image and the area outside the affected area of ​​the second examination image, A second difference acquisition means for acquiring a second difference in the feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image, A third difference acquisition means for acquiring the actual third difference of the affected area at different points in time, based on the first difference and the second difference, Having, Image processing device.

2. The third difference acquisition means acquires the third differences of multiple types of features, The image processing apparatus according to claim 1.

3. The aforementioned multiple types of characteristics include at least two of the size, color, and degree of roughness of the affected area. The image processing apparatus according to claim 2.

4. The system further includes a difference selection means that can select which of the third differences of the aforementioned multiple types of features to display. The image processing apparatus according to claim 2.

5. The second extraction means uses a region containing multiple structures as the region outside the affected area. The image processing apparatus according to claim 1.

6. The system further includes a disease name management means for which a disease name is set for the affected area, The disease name management means, when the disease name is set, displays the third difference that represents the characteristics of the disease name. The image processing apparatus according to claim 1.

7. The system further includes a disease name reconsideration suggestion means that prompts a reconsideration of the disease name set in the disease name management means when changes other than those characteristic of the disease name set in the disease name management means occur in the affected area. The image processing apparatus according to claim 6.

8. The second examination image is a newer image than the first examination image. The system further includes an automatic selection means that automatically selects the first examination image before the user selects the first examination image, while the second examination image is selected. The image processing apparatus according to claim 1.

9. The automatic selection means searches for the past first examination image that is closest to the selected second examination image. The image processing apparatus according to claim 8.

10. A means for acquiring actual size information to obtain the actual size information of the subject, When the actual size information acquisition means acquires the actual size of the affected area in either the first or second examination image, the actual size information association means acquires the actual size of the affected area in the other examination image based on the third difference acquired by the third difference acquisition means and associates it with the actual size of the affected area in the first examination image. It further has, The image processing apparatus according to claim 1.

11. An image processing method using a first examination image and a second examination image taken of the same affected area at different times, The steps include extracting the affected area contained in the first and second examination images, respectively, A step of extracting the area outside the affected area, which is included in the first and second examination images, respectively, The steps include obtaining a first difference in feature quantities between the area outside the affected area in the first examination image and the area outside the affected area in the second examination image, The steps include obtaining a second difference in the feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image, A step of obtaining a third actual difference of the affected area at a different point in time, based on the first difference and the second difference; Having, Image processing methods.

12. Computers, The first and second examination images, taken at different times, of the same affected area, are used. A first extraction means for extracting the affected area contained in the first and second examination images, respectively, A second extraction means for extracting areas outside the affected area, which are included in the first and second examination images, respectively, A first difference acquisition means for acquiring a first difference in feature quantities between the area outside the affected area of ​​the first examination image and the area outside the affected area of ​​the second examination image, A second difference acquisition means for acquiring a second difference in the feature quantities between the affected area of ​​the first examination image and the affected area of ​​the second examination image, A third difference acquisition means for acquiring the actual third difference of the affected area at different points in time, based on the first difference and the second difference. To make it function as program.

Images