Imaging support device and method, imaging device, image processing device and method, program and storage medium

Abstract

To more easily obtain images that allow for a comparison of brightness and chromaticity for an injury obtained by imaging a portion of the same patient at different imaging dates.SOLUTION: An imaging support device comprises: detection means which detects an affected portion from images repeatedly captured by imaging means; extraction means which extracts a reference region in which brightness and chromaticity in predetermined multiple images are aligned from regions excluding the affected portion detected by the detection means in the images; and control means which controls the display means so as to superimpose and display the image captured by the imaging means, a first indicator representing the region of the affected portion, and a second indicator representing the reference region.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an imaging support device and method, an imaging device, an image processing device and method, a program, and a storage medium, and in particular to a technique for imaging images in the medical field. [Background technology]

[0002] Conventionally, in the medical field, an image processing device has been proposed that photographs a reference color sample image together with a patient's skin and adjusts the display color (see Patent Document 1). Also, a color imaging device has been proposed that calculates the white balance of the most recently photographed image based on the white balance of the most recently photographed image after the power is turned on and the white balance data of past images (see Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-19050 [Patent Document 2] Japanese Patent Application Publication No. 10-290469 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, a reference color sample must be photographed along with the patient's skin. If the reference color sample is not photographed, it is difficult to compare the brightness and chromaticity of images of injuries or illnesses taken of the same part of the patient if the images were taken on different dates and times.

[0005] Furthermore, in Patent Document 2, if the light source used when the latest image was taken was different from that used when the previous image was taken, it was difficult to obtain an image with the desired white balance adjustment.

[0006] The present invention has been made in consideration of the above problems, and has as its object to more simply obtain images that allow easy comparison of brightness and chromaticity for injuries and diseases photographed on different dates and times of the same patient's body part. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the imaging support device of the present invention comprises a detection means for detecting an affected area from images repeatedly captured by an imaging means, an extraction means for extracting a reference area that aligns the brightness and chromaticity in a plurality of predetermined images from an area in the image excluding the affected area detected by the detection means, and a control means for controlling a display means to superimpose the image captured by the imaging means, a first indicator indicating the affected area, and a second indicator indicating the reference area. Effect of the Invention

[0008] According to the present invention, it is possible to more easily obtain images that allow easy comparison of brightness and chromaticity between images of lesions captured on different dates and times of the same patient. [Brief description of the drawings]

[0009] [Figure 1] FIG. 4 is a diagram showing an example of an injury / disease characteristics table in the first embodiment. [Diagram 2] 1 is a block diagram showing a schematic functional configuration of an image capturing apparatus according to an embodiment of the present invention. [Diagram 3] 4A to 4C are diagrams illustrating examples of color filters according to an embodiment. [Figure 4] FIG. 2 is a rear view of the digital still camera according to the first embodiment. [Diagram 5] 5 is a flowchart showing an image recording process in the first embodiment. [Figure 6] FIG. 2 is a conceptual diagram showing the structure of a RAW file in the first embodiment. [Figure 7] 1 is a flowchart showing imaging assist figure display processing according to a first embodiment; [Figure 8]5 is a flowchart showing a reference area detection process in the first embodiment. [Figure 9] 5 is a flowchart showing a development process in the first embodiment. [Figure 10] 4A and 4B are views showing an example of an affected area and a reference area on an image in the first embodiment. [Figure 11] 6A and 6B are diagrams showing examples of another affected area and reference area on an image in the first embodiment. [Figure 12] FIG. 11 is a block diagram showing the configuration of a medical system according to a second embodiment. [Figure 13] 10 is a flowchart showing an imaging assist figure display process according to the second embodiment. [Figure 14] FIG. 11 is a rear view of a digital still camera according to a second embodiment. [Figure 15] 10 is a flowchart showing an imaging assist figure display process according to the second embodiment. [Figure 16] FIG. 11 is a rear view of a digital still camera according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <First embodiment> [Terminology definition] The medical system in this embodiment uses electronic medical records, and each patient to be managed is assigned a patient management number that uniquely identifies the patient, and patient information such as name, age, sex, medical history, etc. is registered in advance and linked to the patient management number. When obtaining patient information, the patient information can be obtained using the patient management number as a key.

[0012] In addition, in this embodiment, examination information refers to information related to an examination, and is assigned a test management number in advance that can uniquely identify each examination. The examination information includes information on the patient, the purpose of the examination, the examination area, and the examination location.

[0013] The purpose of the examination indicates the illness or injury that is the subject of the examination, and examples of the illness or injury include atopic dermatitis, bedsores, injuries, gingivitis, and freckles. The illness or injury that is the subject of the examination may be a suspected illness or injury, and may be different from the illness or injury that is the final diagnosis. When there are multiple purposes of the examination, the purpose of the examination includes multiple illnesses or injuries.

[0014] The examination site refers to sites such as the head, chest, breasts, abdomen, waist, buttocks, back, shoulders, elbows, knees, hands, and legs. The examination location refers to a location within a hospital facility, such as an examination room, a photography room, a bathroom, or a room in an inpatient ward, for example.

[0015] A RAW image is image data that is obtained by receiving light with an optical sensor covered with a color filter, photoelectrically converting the electric charge obtained, converting it to analog-digital data, and recording it with lossless compression or without compression. A RAW file is a file that is a RAW image with parameters required for playback and image processing, and information on shooting conditions such as ISO sensitivity, aperture value, and shutter speed as attribute information. Furthermore, a RAW file contains both a developed reduced image and a developed unreduced image. The file structure will be described later.

[0016] In addition, the series of image processing steps that converts a RAW image into an output that resembles the subject as seen by the human eye is called development processing.

[0017] White balance correction is essentially a process of balancing the overall colors so that an achromatic subject is displayed as an achromatic subject image, but in this embodiment, white balance correction refers to a process of adjusting the overall balance of colors, and does not necessarily require an achromatic subject to become achromatic. Also, the white balance coefficient is a coefficient that is multiplied by the pixels of each color to adjust the color balance.

[0018] In addition, in a digital still camera that operates as an imaging device according to one embodiment of the present invention, a figure that is superimposed on the subject image displayed in live view on the rear LCD screen of the camera during imaging and that assists in imaging will hereinafter be referred to as an "imaging assist figure."

[0019] The reference region is an area for determining values ​​to be used as indexes of brightness and chromaticity when performing image processing on the entire image in the imaging device of this embodiment, and is an area of ​​normal tissue, known background, or equipment. Information regarding the reference region is hereinafter referred to as "reference region information." The process of determining the reference region and the reference region information will be described later.

[0020] [Injury and Illness Characteristics Table] FIG. 1 is a conceptual diagram showing an example of an injury / disease characteristics table 100 used in this embodiment, which is stored in the imaging device or is stored so that the imaging device can obtain it from the outside via a communication line in this embodiment.

[0021] The injury / illness column 101 indicates the type of injury / illness, and indicates the injury / illness or observation target that is the subject of the examination. The part dependency column 102 indicates the part dependency of each injury or illness, and indicates whether the probability of the injury or illness varies depending on the part. "Yes" indicates that the probability of the injury or illness varies depending on the part, and "No" indicates that the probability of the injury or illness does not vary.

[0022] The site-specific onset probability column 103 indicates the onset probability for each site when the onset probability of an injury or illness varies depending on the site. However, in this embodiment, sites with extremely low onset probabilities are omitted from the description.

[0023] The movement / enlargement / reduction column 104 indicates the movement and enlargement / reduction of the injury or illness, and indicates whether the range of the injury or illness will move, enlarge or reduce. "Yes" indicates that the range of the injury or illness will move, enlarge or reduce, and "No" indicates that the range of the injury or illness will not move, enlarge or reduce.

[0024] The threshold distance field 105 indicates the threshold of how far the reference region should be from the diseased region, which varies depending on the nature of the injury.

[0025] The equipment / background column 106 indicates the presence or absence of equipment or background that is likely to be used due to injury or illness. The DATA field 107 indicates data on instruments and backgrounds that are likely to be used due to injuries and illnesses, and holds data for detecting known instruments and backgrounds. The target color luminance column 108 indicates the target luminance and chromaticity values ​​in the reference region.

[0026] [Imaging device] 2 is a block diagram showing a schematic functional configuration of a digital still camera that operates as an imaging device including an imaging support device according to an embodiment of the present invention. By executing a predetermined control program in the digital still camera, imaging processing described below is realized and the digital still camera functions as an imaging device. Note that, although a case where the imaging support device is included in the imaging device will be described here as an example, it is also possible to configure the imaging support device with a computer and make it into a system that remotely controls the imaging device, for example.

[0027] In FIG. 2, the imaging unit 200 includes an imaging element such as a CMOS sensor or a CCD. The imaging element is composed of a plurality of pixels arranged in a matrix, and in this embodiment, as shown in FIG. 3, for example, the imaging element is covered with a color filter of a primary color Bayer array consisting of three colors, in which red (R), green (G1), green (G2), and blue (B) are arranged in a matrix direction repeatedly with red (R), green (G1), green (G2), and blue (B) as one unit. Note that G1 and G2 are the same green filter, but are represented as G1 and G2 in order to process them separately in the luminance and chromaticity correction described later. The imaging element converts the incident optical image of the subject into an electrical signal, and each pixel outputs one of an R signal, a G1 signal, a G2 signal, and a B signal according to the color of the filter it is covered with. Then, the imaging unit 200 performs analog-to-digital conversion of the obtained electrical signal to generate image data and output it.

[0028] The CPU 201 controls the entire digital still camera. The ROM 202 stores the operation processing procedures of the CPU 201 (for example, programs for processing when the digital still camera is turned on and basic input / output processing). The RAM 203 functions as the main memory of the CPU 201, and various programs including a control program for implementing the processing described below are loaded from the ROM 202 etc. into the RAM 203 and executed by the CPU 201. The RAM 203 also provides a work area when the CPU 201 executes various processes. Furthermore, the RAM 203 stores inspection information.

[0029] The display unit 204 performs various displays under the control of the CPU 201. For example, information written in an electronic medical record, images captured by the imaging unit 200, etc. are displayed in a predetermined layout. The input unit 205 is configured by one or a combination of operation members such as switches and dials for performing various operations, a touch panel, pointing by line of sight detection, a voice recognition device, and the like.

[0030] 4 is a rear view of the digital still camera in this embodiment. The input unit 205 includes a power button 401, a release button 402, an up button 403, a right button 404, a down button 405, a left button 406, and an enter button 407.

[0031] The power button 401 switches the power ON / OFF. When the photographer presses the button while the digital still camera is off, the CPU 201 determines that the photographer has instructed the camera to turn on the power and turns the power on. When the photographer presses the button while the camera is on, the CPU 201 determines that the photographer has instructed the camera to turn off the power and turns the power off.

[0032] When the photographer presses the release button 402, the CPU 201 determines that an instruction to capture a still image has been given.

[0033] The display 408 is, for example, a liquid crystal display or the like, and constitutes a part of the display unit 204. A touch panel may be arranged on the surface of the display 408, in which case the display 408 and the touch panel constitute the input unit 205. When the photographer touches any point on the display 408 with a finger or the like, the CPU 201 determines that an input instruction has been received from the photographer, determines the content of the operation from the pressed position and movement, and performs processing according to the determined content.

[0034] A rectangle 409 is an imaging assist figure indicating an affected area, and a rectangle 410 is an imaging assist figure indicating a reference area. The imaging assist figures 409 and 410 will be described later.

[0035] Returning to the explanation of FIG. 2, the media drive 206 allows a removable storage medium to be mounted and enables data to be stored in the mounted storage medium and data stored in the storage medium to be read out. The network interface 207 is an interface for connecting to a network 210 via a communication line 209. Through the network interface 207, data is sent to and received from a server computer, a personal computer, etc. (not shown). The system bus 208 comprises an address bus, a data bus, and a control bus, and connects the above-mentioned units together.

[0036] The image processing unit 211 performs image processing on the image data generated by the imaging unit 200. The CPU 201 temporarily stores the image data generated by the imaging unit 200 and attribute information at that time in the RAM 203. Then, as necessary, the image processing unit 211 performs a series of image processing so as to obtain image data that matches the visual characteristics of humans. When performing this series of image processing, processing is performed using a white balance coefficient or a color luminance correction coefficient that is set as described below.

[0037] In the file generating unit 212, under the control of the CPU 201, a compression process is performed on the image data as necessary, and the image data and attribute information, which is information required for reproducing the image data, are recorded together as a file.

[0038] [Image data storage processing] Fig. 5 is a flow chart showing an image capturing process and a storage process of the captured image data in a digital still camera having the configuration shown in Fig. 2. In this embodiment, the digital still camera is used in a medical system, and the subject to be captured includes an affected area.

[0039] In S501, the CPU 201 acquires the examination purpose. The examination purpose may be input using the input unit 205 or selected from preset items, or examination information may be stored in the RAM 203 in advance and read out.

[0040] Next, in S502, the CPU 201 loads the image data (here, RAW image data) generated by the imaging unit 200 into the RAM 203.

[0041] In S503, the CPU 201 controls the image processing unit 211 to perform reduction processing and development processing on the RAW image data in the RAM 203. Note that the processing performed in S503 will be described later with reference to FIG.

[0042] In S504, the CPU 201 superimposes the imaging assist figure on the reduced image reduced in S503 and displays the imaging assist figure on the display unit 204. The display processing of the imaging assist figure will be described later.

[0043] In S505, the CPU 201 determines whether or not an image capturing operation has been performed. The image capturing operation is, for example, pressing a release button included in the input unit 205. If it is determined that an image capturing operation has not been performed, the process returns to S502, and the captured image and the image capturing assist figure are repeatedly superimposed on each other. On the other hand, if it is determined that an image capturing operation has been performed, the process proceeds to S506, where the latest RAW image data acquired during the image capturing operation is stored in the RAM 203, and the process proceeds to S507.

[0044] In S507, the CPU 201 detects a reference area by a process described later using the reduced image obtained in S503, and judges whether or not the reference area in the RAW image data stored in the RAM 203 includes a number of pixels equal to or greater than a threshold value. If it is judged that a reference area with a number of pixels equal to or greater than the threshold value is included, the process proceeds to S508. On the other hand, if it is judged that the reference area is not included, in S510, a message indicating that the reference area is not included is displayed on the display unit 204, and the process proceeds to S508.

[0045] Next, in S508, CPU 201 determines whether or not the affected area is included. Note that machine learning inference is used to detect the affected area. If it is determined that the affected area is included, the process proceeds to S510, and if it is determined that the affected area is not included, the process proceeds to S509.

[0046] In S509, CPU 201 displays on display unit 204 that the affected area is not included. This is because, even though the image is taken to record the progress of the affected area, if the affected area is not included in the image data, the purpose of the image taking may not be achieved. However, since the affected area may have already healed, it is sufficient to at least notify the user that the affected area is not included. This notification allows the user to take another image if necessary.

[0047] In S512, the CPU 201 performs development processing on the RAW image data in the RAM 203, and stores the developed image data in the RAM 203. The development processing performed here will be described later with reference to FIG. 9. After the development processing in S512, the image data of all pixels and the image data of the reduced image are held in the RAM 203. Hereinafter, the image data of all pixels will be referred to as the "original image," and the reduced image data will be referred to as the "thumbnail image."

[0048] In S513, the CPU 201 controls the file generation unit 212 to perform compression processing. In this embodiment, lossless compression is performed on the RAW image data that has not been developed and that has been read in S506, and JPEG compression is performed on the image data that has been developed in S512 and the image data that has been reduced in S503. Then, the compressed data is stored in the RAM 203.

[0049] In S514, the CPU 201 saves the compressed image data and attribute information in the RAM 203 as a RAW file in a storage medium attached by the media drive 206. Note that the attribute information is recorded including reference area information, which will be described later.

[0050] [RAW file] Fig. 6 is a schematic diagram of an image file created by the image recording method according to this embodiment, showing the RAW file saved in S514. In Fig. 6, an attribute information area 601 stores information at the time of shooting, such as image data size, shooting date and time, camera model, compression method, flash on / off, etc., as well as information required for reading, playing, and selecting images. A reduced image data area 602 stores image data of thumbnail images to be displayed when displaying a list. A developed image data area 603 stores image data of the main image, and a main image data area 604 stores RAW image data including all pixels to be stored as the image file.

[0051] [Image capture assist graphic display processing] Next, the process of displaying the capture assist figure, which is performed in S504, will be described with reference to the flowchart in FIG.

[0052] In S701, the CPU 201 performs a process of detecting an affected area of ​​the injury or disease to be examined from the reduced image generated in S503. Note that the detection of the affected area is performed using inference based on machine learning. When the affected area is detected, the coordinates of the affected area are stored in the RAM 203 as affected area information.

[0053] Next, in S702, the CPU 201 determines whether or not the affected area has been detected. If the affected area has been detected, the process proceeds to S703. If the affected area has not been detected, the process proceeds to S704, in which the CPU 201 notifies the user that the affected area has not been found, and the process proceeds to S705. In S703, the CPU 201 displays a reduced image on the display 408 as shown in FIG. 4, and also superimposes an imaging assist figure 409 (first index) indicating the detected affected area.

[0054] Next, in S705, the CPU 201 detects the reference area. When the reference area is detected, the coordinates are stored as reference area information in the RAM 203. The reference area detection process will be described later.

[0055] In S706, the CPU 201 determines whether or not the detection of the reference area has been successful, and if the detection of the reference area has been successful, the process proceeds to S707. On the other hand, if the detection of the reference area has failed, the process proceeds to S708, in which the CPU 201 notifies the user that the reference area cannot be found, and the process returns to the process in FIG.

[0056] In S707, CPU 201 displays an imaging assist figure 410 (second index) indicating the reference area superimposed on the reduced image displayed on display 408 as shown in Fig. 4, and returns to the processing in Fig. 5. Note that imaging assist figure 410 may be any figure including a predetermined number of pixels or more in order to calculate the luminance and chromaticity of the reference area, and when the reference area is divided into multiple locations, the imaging assist figure may be drawn in a location with a larger area, or multiple imaging assist figures may be drawn. However, to make it easier to photograph the affected area, the imaging assist figure is drawn in a position close to the affected area within the reference area.

[0057] In this way, when the affected area and the reference area are detected, an imaging assist figure 409 indicating the affected area and an imaging assist figure 410 indicating the reference area are displayed, respectively, allowing the user to easily adjust the imaging range to be preferable for photographing the affected area.

[0058] [Reference area detection process] Next, the reference region detection process performed in S705 will be described with reference to the flowchart in FIG.

[0059] First, in S801, the CPU 201 performs a process of detecting existing instruments and backgrounds for the injury or illness to be examined. Here, the instrument / background column 106 of the injury or illness characteristics table 100 shown in Fig. 1 is referenced to determine whether or not there is an instrument and background corresponding to the relevant injury or illness information. If there is an instrument or background, the data column 107 is further referenced to obtain data for detecting the target from the injury or illness characteristics table 100. Note that here, the injury or illness characteristics table 100 is stored in, for example, a storage medium, and is read into the RAM 203 via the media drive 206 when used.

[0060] Next, in S802, the CPU 201 determines whether known tools and background have been detected from the reduced image obtained in S503. For tools, detection is attempted using a general pattern matching process based on data obtained from the DATA column 107 of the injury / disease characteristics table 100. For the background, a distance measurement value is used, and if there is a flat area at a distance farther than the central subject, that area is detected as the background. If known tools and background are detected, the process proceeds to S810. If neither known tools nor background are detected, the process proceeds to S803.

[0061] In S803, the CPU 201 performs machine learning inference on the injury or disease that is the object of the examination, and detects the affected area of ​​the injury or disease. If the affected area has already been detected by the process of S701, the affected area information stored in the RAM 203 is used.

[0062] In S804, the CPU 201 detects an area outside the affected area and having a distance measurement value within a threshold from the affected area as a normal tissue area (partial area). The distance measurement value indicates the distance from the light receiving surface of the imaging element of the imaging unit 200 to the subject, and corresponds to all pixels that constitute the image. In this embodiment, as an example, the threshold value is set to ±20 mm. For example, when the distance measurement values ​​of the pixels corresponding to the affected area are distributed between 500 mm and 510 mm, an area outside the affected area with a distance measurement value between 480 mm and 530 mm is detected as a normal tissue area.

[0063] In S805, the CPU 201 judges whether the size of the normal tissue region is equal to or larger than a threshold. In the present embodiment, as an example, the threshold is set to 10% of the area of ​​the image. If it is judged that the size of the normal tissue region is equal to or larger than the threshold, the process proceeds to S806, and if it is judged that the size of the normal tissue region is less than the threshold, the process proceeds to S809.

[0064] In S806, the CPU 201 refers to the injury / disease characteristics table 100, and obtains a distance threshold corresponding to the injury or disease to be examined from the threshold distance column 105. Then, of the normal tissue regions detected in S804, a region that is farther away from the injury or disease region by more than the threshold is set as a reference region.

[0065] Next, in S807, the CPU 201 estimates the location of the normal tissue region extracted in S804. In this embodiment, the estimation is performed by machine learning.

[0066] In S808, the CPU 201 refers to the part dependency column 102 and the part-by-part onset probability column 103 of the injury / disease characteristics table 100, and determines whether the onset probability of the injury / disease to be examined depends on the part of the normal tissue region extracted in S808, and if so, whether the onset probability of the part is less than a threshold value. In the present embodiment, as an example, the threshold value is less than 50%.

[0067] If it is equal to or greater than the threshold, the reference region set in S806 is deemed inappropriate (invalid) as a reference region, and the process proceeds to S809, where the CPU 201 stores the result that the reference region could not be detected in the RAM 203. On the other hand, if the onset probability is independent of the site or is less than the threshold, the process proceeds to S810.

[0068] In S810, the instrument or background detected in S802 is stored in the RAM 203 as a reference area, or the reference area set in S806 is stored in the RAM 203.

[0069] In S811, the CPU 201 determines an evaluation value of the possibility of change over time according to the method for determining the reference region. In this embodiment, the evaluation value is set to 0 if there is no change and 100 if there is a change. If the evaluation value is less than 50, the possibility of change over time is considered low. The evaluation value is determined in advance according to the method for determining the reference region and is stored in the RAM 203. In the case of a known instrument or background detected in S802, the evaluation value is set to 10. Also, in the case of a normal tissue region in S808, the evaluation value is set to 20.

[0070] In S812, the CPU 201 stores, as part of the attribute information of the RAW image data, information on the method for determining the reference region, an evaluation value of the possibility of time-series change, coordinates on the image of the reference region, and information on the target luminance and chromaticity, as reference region information in the RAM 203. Note that the information on the method for determining the reference region is information indicating whether the reference region is included in a known instrument or background region, or in a normal tissue region. When the above process is completed, the process returns to FIG.

[0071] [Development processing] Next, the development processing performed in S503 and S512 in FIG. 5 will be described with reference to the flowchart in FIG.

[0072] First, in step S901 , the CPU 201 loads RAW image data to be processed into the RAM 203 . Next, in S902, the CPU 201 determines whether or not to perform reduction processing. If reduction processing is to be performed, the process proceeds to S903, and if reduction processing is not to be performed, the process proceeds to S904. In S503, it is determined that reduction processing is to be performed, and in S212, it is determined that reduction processing is not to be performed.

[0073] In S903, the CPU 201 performs a reduction process on the RAW image data read in S301. In this case, a process of thinning out pixels, a process of adding pixel signals, or both are performed. In this case, the reduction ratio is set to be the same vertically and horizontally.

[0074] Next, in S904, the CPU 201 subtracts a value that is a base charge from the luminance value constituting the pixel of the RAW image data. The value of the base charge is hereinafter referred to as an "OB (optical black) value." The OB value is a luminance value acquired in a light-shielded area of ​​the imaging element, and the OB value corresponding to each RGB color filter is recorded in advance as attribute information.

[0075] In S905, the CPU 201 determines whether the target color luminance column 108 of the injury / disease characteristics table 100 contains information on the target luminance and chromaticity in the reference area (hereinafter referred to as "target color luminance information") and whether a reference area has been set. If the reference area is a normal skin area, the target luminance and chromaticity in the reference area are values ​​indicating the reference skin color and brightness. Also, if the reference area is a known instrument or background, the values ​​are values ​​indicating the color and brightness of the instrument or background. If the target color luminance information and the reference area match, the process proceeds to S913; if not, the process proceeds to S906.

[0076] Next, in S906, the CPU 201 calculates a white balance coefficient using the attribute information. When the auto white balance (AWB) mode is set in the settings of the imaging device, the white balance coefficient is calculated inside the imaging device assuming the light source color temperature when the light source is a fluorescent light, an incandescent light, or a strobe light. In S906, the AWB mode is assumed, and the white balance coefficient associated with the AWB mode is read into the RAM 203.

[0077] When a natural object such as a landscape is photographed, the average value of each pixel tends to be achromatic. In the AWB mode of this embodiment, this is utilized to calculate the average value of the pixels constituting the subject image to be achromatic after subtracting the OB value for each RGB pixel. If the average value of the R signal after subtracting the OB value is aveR, the average value of the G1 signal is aveG1, the average value of the G2 signal is aveG2, the average value of the B signal is aveB, and the target luminance value of all the colors R, G1, G2, and B is tgtNonCol, the white balance coefficients wbRcoef, wbG1coef, wbG2coef, and wbBcoef can be calculated by the formula (1).

[0078] tgtNonCol=(aveR+aveG1+aveB+aveG2)÷4 wbRcoef=tgtNonCol÷aveR wbG1coef=tgtNonCol÷aveG1 wbG2coef=tgtNonCol÷aveG2 wbBcoef=tgtNonCol÷aveB …(1)

[0079] On the other hand, in S913, the CPU 201 calculates the luminance and chromaticity correction coefficients of the reference area. The luminance average values ​​of the color sensors of the reference area in the main image data area 604 of the RAW file shown in Fig. 3 that is the comparison source are bR, bG1, bG2, and bB for each color, and the luminance average values ​​of the color sensors targeted by the reference area are tR, tG1, tG2, and tB for each color. In this case, the calculation formulas of the color luminance correction coefficients tRcoef, tG1coef, tG2coef, and tBcoef of the luminance and chromaticity for the correction target image are shown in Equation (2).

[0080] tRcoef = tR÷bR tG1coef=tG1÷bG1 tG2coef=tG2÷bG2 tBcoef = tB÷bB …(2)

[0081] In S907, the CPU 201 controls the image processing unit 211 to multiply all pixels constituting the RAW image by the white balance coefficient for each color calculated in S906 or the color luminance correction coefficient for each color calculated in S913. If the input values ​​of each color are inR, inG1, inG2, and inB, and the coefficients of each color, wbRcoef or tRcoef, are Rcoef, wbG1coef or tG1coef, G1coef, wbG2coef or tG2coef, and G2coef, and wbBcoef or tBcoef, are Bcoef, the output values ​​of each color, outR, outG1, outG2, and outB, are shown in Equation (3).

[0082] outR = inR × Rcoef outG1=inG1×G1coef outG2=inG2×G2coef outB = inB × Bcoef …(3)

[0083] In S908, the CPU 201 controls the image processing unit 211 to perform an interpolation process (demosaic process) on all pixels constituting the RAW image. This interpolation process is necessary when the color filters arranged on the image sensor are composed of multiple colors, and is unnecessary when a value corresponding to each color is obtained for each pixel using a stacked sensor, a three-plate sensor, or the like. In this embodiment, the color filters have a Bayer array as shown in FIG. 3, and only one of the R signal, G1 signal, G2 signal, and B signal can be obtained as the pixel value of each pixel, so the average value of the missing color signals from the surrounding pixels is obtained and used as the value of that pixel.

[0084] In S909, the CPU 201 controls the image processing unit 211 to perform a luminance adjustment process. In this embodiment, the luminance adjustment process is a process of multiplying an average luminance value by a luminance coefficient so that the average luminance value becomes a target luminance determined in the imaging device. In S910, the CPU 201 controls the image processing unit 211 to perform color adjustment processing. The color adjustment processing is processing for applying a color conversion matrix in accordance with the characteristics of the optical sensor. In S911, the CPU 201 performs edge enhancement processing by controlling the image processing unit 211. In this embodiment, the edge enhancement processing is processing that applies a high-pass filter. In S912, the CPU 201 controls the image processing unit 211 to perform tone curve processing. The tone curve processing is a process that broadens the distribution of low and intermediate luminance in the luminance gradation at the time of output and suppresses the distribution of high luminance parts in accordance with the human perception characteristics.

[0085] In S914, the CPU 201 controls the image processing unit 211 to output the results of a series of image processing to the RAM 203 and store them for use in applications such as display and file generation.

[0086] [In the case of atopic dermatitis] Next, a specific example in which this embodiment is applied to photographing executives will be described with reference to FIG. 1 and in accordance with the flow chart of FIG. First, a case where the disease to be examined is atopic dermatitis will be described. Atopic dermatitis is characterized by the fact that the area of ​​the disease changes or moves. In addition, the probability of developing the disease varies depending on the part of the body. A characteristic example of this will be described.

[0087] In S801, the CPU 201 refers to the equipment / background column 106 of the injury / disease characteristics table 100 to obtain whether or not there is an equipment / background corresponding to atopic dermatitis. Since the answer is "no" in this case, in S802, the CPU 201 determines that there is no corresponding existing equipment or background, and proceeds to S803.

[0088] In S803, the CPU 201 performs machine learning inference on atopic dermatitis for the reduced image obtained in S503, and detects an affected area.

[0089] In S804, the CPU 201 detects an area outside the atopic dermatitis lesion area and within a threshold distance from the affected area as a normal tissue area. When the distance measurement value of the atopic dermatitis lesion area is 1000 mm to 1010 mm, for example, when the threshold is set to plus or minus 20 mm, the CPU 201 detects an area outside the lesion area with a distance measurement value between 980 mm and 1030 mm as a normal tissue area.

[0090] In S805, CPU 201 determines whether the size of the detected normal tissue region is equal to or larger than a threshold. In this embodiment, if the threshold is 10% of the area of ​​the image, and the width of the reduced image is 1280 pixels and the height is 960 pixels, the threshold is 122880 pixels. If it is determined that the size of the normal tissue region is equal to or larger than the threshold, the process proceeds to S806, and if it is determined that the size is less than the threshold, the process proceeds to S809. In this embodiment, it is determined that the area of ​​the normal tissue region is equal to or larger than the threshold, and the process proceeds to S806.

[0091] In S806, the CPU 201 refers to the threshold distance column 105 of the injury / disease characteristics table 100 and obtains a threshold distance of 40 mm corresponding to atopic dermatitis. The threshold distance is set to be longer because atopic dermatitis is an injury / disease that spreads and moves due to its nature. Then, of the normal tissue regions detected in S805, a region that is away from the injury / disease by the threshold distance of 40 mm or more is extracted as a reference region.

[0092] 10 shows an example of a reduced image 1001 displayed on the display 408. The purpose of the examination is atopic dermatitis, the examination area is the chest, and the examination location is an in-hospital radiography room.

[0093] Reference numerals 1002 and 1003 denote diseased areas detected in S803, and reference numeral 1004 denotes a normal tissue area detected in S804. Reference numeral 1005 denotes a conceptual line corresponding to the threshold distance. Since the threshold distance in atopic dermatitis is 40 mm, the reference area is determined so that the distance from the diseased area 1003 is 40 mm or more.

[0094] Reference numeral 1006 denotes an imaging assist figure indicating the reference region, which is displayed in S707. The imaging assist figure 1006 allows the user to check whether the reference position has been correctly detected.

[0095] Reference numeral 1007 denotes an imaging assist figure indicating the affected area, which is displayed in S703. The imaging assist figure 1007 allows the user to confirm whether the affected area has been correctly detected.

[0096] In S807, the CPU 201 estimates the part of the reference region extracted in S806. In this embodiment, the estimation is performed by machine learning, and in this case, the part is estimated to be the abdomen.

[0097] In S808, the CPU 201 refers to the abdominal onset probability column 103 of the injury / disease characteristics table 100, and refers to the abdominal onset probability of atopic dermatitis. Since the abdominal onset probability is not stored, that is, the onset probability is extremely low, it is determined to be less than the threshold value of 50%, and the process proceeds to S810. Then, the extracted reference region is stored in the RAM 203.

[0098] In addition, if the part of the reference region is estimated to be the chest, upper back, etc. in S807, the probability of onset is 50%, and therefore, it is processed as if the reference region could not be extracted in S809. In this case, the result is NO in S705, and the imaging assist figure 1006 of the reference region as shown in FIG. 10 is not displayed.

[0099] In S811, the CPU 201 determines an evaluation value of the possibility of change over time according to the method for determining the reference region. Since the reference region is a normal tissue region, the evaluation value is set to 20. Then, in S812, the CPU 201 stores, as the reference region information obtained by the above process, information on the method for determining the reference region, the evaluation value of the possibility of change over time, the coordinates on the image of the reference region, and target luminance and chromaticity information in the RAM 203 as part of attribute information of the RAW image data.

[0100] Next, we will explain the brightness and chromaticity correction processing in a case where detection of the affected area and the reference area is successful, and both the imaging assist figure 1007 of the affected area and the imaging assist figure 1006 of the reference area are displayed in the image, an imaging instruction is issued in S505, and the RAW image data is stored in RAM 203 in S506.

[0101] In S905, the CPU 201 refers to the target color luminance column 108 of the injury / disease characteristics table 100 and determines whether or not there is target color luminance information for atopic dermatitis. Since there is a value in the target color luminance column 108 for atopic dermatitis and the reference region has also been extracted, the result is YES, and the process proceeds to S913.

[0102] In S913, the CPU 201 calculates the color luminance correction coefficients of the reference area. First, the average luminance value of each color sensor of the reference area in the RAW image data corresponding to the extracted reference area is calculated for each color. As an example, bR=120, bG1=90, bG2=95, and bB=60. If the target luminance of the reference area obtained from the target color luminance field 108 is tR=254, tG1=220, tG2=220, and tB=189 for each color, the color luminance correction coefficients tRcoef, tG1coef, tG2coef, and tBcoef for the correction target image can be obtained by formula (4).

[0103] tRcoef = tR÷bR = 254 / 120 tG1coef=tG1÷bG1=220 / 90 tG2coef=tG2÷bG2=220 / 95 tBcoef=tB÷bB=189 / 60 …(4) In S907, the CPU 201 multiplies all pixels constituting the RAW image by the color luminance correction coefficients corresponding to the respective colors calculated in S913. For example, when the values ​​of the respective colors of the input RAW data are bR=120, bG1=90, bG2=95, and bB=60, the values ​​of the respective colors outR, outG1, outG2, and outB of the luminance and chromaticity correction obtained are shown in Equation (5).

[0104] outR=bR×tRcoef=120×(254 / 120)=254 outG1=bG1×tG1coef=90×(229 / 90)=220 outG2=bG2×tG2coef=95×(220 / 95)=220 outB=bB×tBcoef=60×(189 / 60)=189 …(5) In this way, the luminance and chromaticity of the reference area will match the target luminance and chromaticity regardless of the light source, so if the processing from S908 to S912 is the same, an image will be obtained in which the average luminance and chromaticity of the reference area are the same as the target luminance and chromaticity.

[0105] As described above, according to the imaging device of the present embodiment, by displaying an imaging assist figure of the reference area, it becomes easy to capture an image of the affected area and to capture an image of the affected area including the reference area that is the reference for brightness and chromaticity. That is, imaging may be performed so that the imaging assist figure of the affected area and the imaging assist figure of the reference area are displayed. When the range of an injury or disease has a characteristic that changes or moves, an area that is a threshold distance away from the injury or disease area and is a part where the injury or disease is unlikely to develop is extracted as the reference area. This makes it possible to extract an area with a low evaluation value for the possibility of time-series change as the reference area, and therefore the image can be corrected based on the reference area so that the brightness and chromaticity of the captured image become the target brightness and chromaticity.

[0106] In addition, in the case of an illness such as atopic dermatitis in which the probability of the site of onset changes depending on the age of the patient, the information in the onset probability by site column 103 in the illness characteristics table may be changed for each age according to the patient's age, or information on the probability of the site of onset according to age may be added.

[0107] In addition, if skin color differs depending on gender, age, or race, a target color luminance may be determined for each person. Since it is sufficient to compare changes in the lesion area over time for a single patient, differences in luminance and chromaticity of the captured image caused by the light source at the time of shooting can be reduced.

[0108] [In case of injury] Next, a case where the injury or disease to be examined is an injury will be described. An injury has the characteristic that the area of ​​the injury or disease does not spread or move, and the probability of the injury or disease occurring does not depend on the location. A characteristic example of this will be described.

[0109] First, in S801, the CPU 201 refers to the equipment / background column 106 of the injury / illness characteristics table 100 to obtain the presence or absence of equipment and background corresponding to the injury. Since the result here is "absent," in S802 the CPU 201 determines that there is no corresponding existing equipment or background, and proceeds to S803.

[0110] In S803, the CPU 201 performs machine learning inference on the injury for the reduced image obtained in S503, and detects the affected area. In S804, the CPU 201 detects, as a normal tissue region, a region that is outside the injured or diseased region and whose distance measurement value is within a threshold distance from the affected region.

[0111] In S805, CPU 201 judges whether the size of the detected normal tissue region is equal to or larger than a threshold. If it is judged that the size of the normal tissue region is equal to or larger than the threshold, the process proceeds to S806, and if it is judged that the size is less than the threshold, the process proceeds to S809. Here, it is assumed that the area of ​​the normal tissue region is equal to or larger than the threshold, and the process proceeds to S806.

[0112] In S806, the CPU 201 refers to the threshold distance column 105 of the injury / disease characteristics table 100 and obtains a threshold distance of 10 mm corresponding to the injury. The threshold distance is set to be short because an injury is an injury that does not spread or move due to its nature. Then, of the normal tissue regions detected in S805, a region that is away from the injury by the threshold distance of 10 mm or more is extracted as a reference region.

[0113] 11 shows an example of an image 1101 displayed on the display 408. The purpose of the examination is an injury, the part to be examined is the left leg, and the examination location is an in-hospital radiography room.

[0114] Reference numeral 1102 denotes the diseased area detected in S703, and reference numerals 1103 and 1104 denote normal tissue areas extracted in S804. Reference numeral 1105 denotes a conceptual line corresponding to the threshold distance. Since the threshold distance for an injury is 10 mm, the diseased area 1102, the normal tissue area 1103, and the reference area are determined so that the distance from the diseased area 1102 is 10 mm or more.

[0115] Reference numerals 1106 and 1107 are imaging assist figures indicating the reference region, and are displayed in S707. The user can check whether the reference position has been correctly detected by the imaging assist figures 1106 and 1107. Note that it is sufficient to capture an image so that either the imaging assist figure 1106 or 1107 is displayed.

[0116] Reference numeral 1108 denotes an imaging assist figure indicating the affected area, which is displayed in S703. The imaging assist figure 1108 allows the user to confirm whether the affected area has been correctly detected.

[0117] In S807, the CPU 201 estimates the part of the reference region extracted in S806 by machine learning. If multiple reference regions are extracted, estimation is performed for each reference region. In this embodiment, all reference regions are estimated to be the left leg.

[0118] In S808, the CPU 201 refers to the onset probability column 103 for each body part in the injury / disease characteristics table 100, and refers to the onset probability of the injured left leg. If the reference area spans multiple body parts, the onset probability is determined for each body part. Since the onset probability of the left leg is not stored, meaning that the onset probability is extremely low, it is determined to be less than the threshold value of 50%, and the process proceeds to S810. Then, the extracted reference area is stored in the RAM 203.

[0119] The processes from S809 to S810 onwards are similar to those described in the example of atopic dermatitis above, and therefore description thereof will be omitted.

[0120] As described above, according to the imaging device of the present embodiment, when the range of the injury or disease does not expand or move, the imaging device extracts the reference area including the area close to the injury or disease area. Then, an imaging assist figure indicating the reference area is drawn for each of the separated areas. This makes it easy to capture the image of the affected area including the reference area that is the reference for brightness and chromaticity in the image of the affected area. Also, as described in the present embodiment, even if the reference area is divided into multiple areas, the incidence rate of each area is referenced and determined for each area, and as a result, an area that is unlikely to change over time is extracted. This makes it easy to capture the image of the affected area including the reference area even in the case of a location with a limited area such as a limb, and the image can be corrected based on the reference area so that the brightness and chromaticity of the captured image become the target brightness and chromaticity.

[0121] [In case of gingivitis] Next, a case where the lesion to be inspected is gingivitis will be described. Gingivitis has a characteristic that a known instrument is included in the image. A characteristic example of this will be described.

[0122] First, in S801, the CPU 201 refers to the instrument / background column 106 of the injury / disease characteristics table 100 to obtain the presence or absence of an instrument and background corresponding to gingivitis. Since the presence is "present" here, the CPU 201 further refers to the DATA column 107 to obtain data for detecting the instrument. In this embodiment, data for detecting the angle of the mouth is obtained, and a process for detecting the angle of the mouth is performed by pattern matching.

[0123] In S802, the CPU 201 judges whether or not the corner of the mouth has been detected from the reduced image obtained in S503. If the corner of the mouth has been detected, the RAM 203 stores that the comparison area is a known background or an instrument, and the process proceeds to S810. If the known background or instrument has not been detected, the process proceeds to S803, and thereafter, the same process as in the case of atopic dermatitis and injury is performed.

[0124] On the other hand, if the canthus is detected, the process proceeds to S810, and the extracted canthus is stored in the RAM 203 as a reference region.

[0125] Then, in S811, the CPU 201 determines an evaluation value of the possibility of change over time according to the method for determining the reference region. Since the comparison region is a known background and tool, the evaluation value is set to 10. The subsequent processing is similar to the processing described in the example of atopic dermatitis, and therefore a description thereof will be omitted.

[0126] As described above, according to the imaging device of this embodiment, when there is a known instrument or background, that area is extracted as a reference area. Then, an imaging assist figure of the reference area is drawn together with an imaging assist figure indicating the affected area. This makes it easy to capture an image of the affected area including the reference area that serves as a reference for luminance and chromaticity, and correct the image based on the reference area so that the luminance and chromaticity of the captured image become the target luminance and chromaticity.

[0127] In this embodiment, an example has been given in which a reference area is determined using a known tool and the luminance and chromaticity are corrected. However, even if the background is the same, such as a known background cloth, drape, or wall of the imaging room, the image can be corrected in a similar manner so that the luminance and chromaticity are consistent.

[0128] In addition, in the present embodiment, an example has been described in which the imaging assist figure indicating the reference area is a rectangle, but an area that can be the reference area may be displayed as a polygon. Also, the area may be drawn using a surface such as a diagonal line, and the user may be notified by text or sound whether the reference area is included in the subject image as an area sufficient for correcting the luminance and chromaticity.

[0129] As described above, according to the imaging device of this embodiment, by supporting imaging so that both the affected area and the reference area are included when imaging the affected area based on the examination purpose selected when imaging with the imaging device, it is possible to capture images that are easy to compare images of injuries and illnesses captured of the same patient's body part on different imaging dates and times, even without reliably capturing the reference color sample. This makes it possible to determine, as the reference area, an area included in the subject image of an image in which the evaluation value of the possibility of change over time is equal to or less than a predetermined value based on the examination information, and to correct the image so that the luminance and chromaticity are consistent.

[0130] In this embodiment, the structure of a RAW file is shown, but the image file saved at the time of shooting is not limited to a RAW file, and may be in a general-purpose image format such as JPEG. Also, although the reference area information is held in the attribute information, the reference area information does not have to be stored in the attribute information of the file as long as it can be acquired in association with the image.

[0131] <Second embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, a method will be described in which, based on the examination purpose selected when photographing with an imaging device, past images of the same patient and examination purpose are obtained from an in-hospital system, and imaging is performed based on information associated with the images.

[0132] [System Configuration] FIG. 12 is a block diagram showing the configuration of a medical system in this embodiment.

[0133] The imaging device 1201 captures an affected area 1207 of a patient to be examined and generates visible light image data. In the following description, the image captured by the imaging device 1201 will be described as visible light image data. Note that the imaging device 1201 has the same functions as the imaging device described with reference to FIG. 2, and a description thereof will be omitted. When the imaging device 1201 is powered on by the photographer, it connects to the network 1202 and becomes capable of communicating with other devices in the medical system via the network 1202. The imaging device 1201 continues to maintain a communicable state until the power is turned off.

[0134] The image processing device 1206 communicates with the imaging device 1201 and acquires examination information from an electronic medical record system 1204, which will be described later.

[0135] The electronic medical record system 1204 receives a request from the electronic medical record display terminal 1203, communicates with the image processing device 1206 and the image management system 1205, and transmits and receives data associated with the creation and editing of electronic medical records. The electronic medical record system 1204 in this embodiment stores examination information of patients in the medical system in a database in an internal storage device. In response to a request to obtain patient information from the image processing device 1206, it is capable of obtaining a series of patient information for a patient using a patient management number as a key. Furthermore, in response to a request to obtain examination information from the image processing device 1206, it is capable of communicating with the image management system 1205 (described later) and obtaining examination information related to past diagnoses of injuries and illnesses.

[0136] An electronic medical record display terminal 1203 communicates with an electronic medical record system 1204, displays electronic medical records, accepts input when creating electronic medical records, and adds information to images.

[0137] The image management system 1205 receives image data captured by the imaging device 1201 via the image processing device 1206, and records the data in an internal storage device in association with the patient reference number and the date and time of capture. When an acquisition request for image data is received from the electronic medical record system 1204 or the image processing device 1206, the image management system 1205 transmits image data that meets the specified conditions to the requesting party.

[0138] The affected area 1207 of the patient indicates the object of examination in the medical system. There are cases where the state of injury or illness is diagnosed based on the appearance of symptoms throughout the body, and cases where the state of injury or illness is diagnosed based on the appearance of symptoms in part of the body. In the case of the appearance of symptoms throughout the body, a large area such as the head, trunk, lower limbs, and upper limbs is examined. It also includes cases where the alignment of teeth and the state of replacement are recorded, such as in the oral cavity, rather than injuries or illnesses.

[0139] The test information database 1208 is a database that stores test information. By issuing an instruction via the electronic medical record system 1204, it is possible to search by patient management number or examination date and time, and output the test information of the corresponding patient and examination date and time.

[0140] The facility information database 1209 is a database that stores information on medical staff and devices. It is assumed that, by instructions given via the electronic medical record system, a search can be made by patient management number to obtain information on the corresponding photographer. In addition, by assigning an ID to the imaging device 1201, when multiple imaging devices 1201 are used, it is assumed that a search can be made by imaging device ID to obtain information on the corresponding imaging device.

[0141] The image information database 1210 is a database that stores patient information and images in association with each other.

[0142] The image processing device 1206, the imaging device 1201, the electronic medical record system 1204, and the image management system 1205 communicate with each other via a network 1202. The electronic medical record display terminal 1203 and the electronic medical record system 1204 communicate with each other via an HDMI (registered trademark) cable or the like. The communication method is, for example, a network, an HDMI (registered trademark), or a USB, but is not limited to these.

[0143] In the medical system of this embodiment, the imaging device 1201, electronic medical record display terminal 1203, electronic medical record system 1204, image management system 1205, and image processing device 1206 are described as separate devices. However, the functions of two or more devices may be realized by one device, for example, by providing the configuration of the imaging device 1201 in the electronic medical record display terminal 1203.

[0144] [Image capture assist graphic display processing] 13 is a flowchart showing the imaging assist figure display process in this embodiment. In this embodiment, this process is performed by the imaging device 1201. Hereinafter, a method for detecting a reference area on an image will be described with reference to FIG.

[0145] In S1301 , the CPU 201 causes the user to input a patient reference number via the input unit 205 , acquires examination information from the electronic medical record system 1204 via the network interface 207 , and reads the information into the RAM 203 .

[0146] In S1302, the CPU 201 obtains images and examination information of the same patient, the same examination purpose, the same area, and a past examination date from the examination information read into the RAM 203 via the network interface 207 from the image management system 1205 and the electronic medical record system 1204, respectively.

[0147] In S1303, the CPU 201 judges whether there is reference area information associated with a past image from the information read into the RAM 203. If there is reference area information associated with a past image, the process proceeds to S1304. If there is no reference area information associated with a past image, the process proceeds to S1306. For example, if the patient to be examined is a first-time patient, or if the patient is not a first-time patient but is examined for a new examination purpose, there is no past image, so it is judged that there is no reference area information associated with the past image. Even if there is a past image, if the image is taken with a different model of imaging device and there is no reference area information, it is also judged that there is no reference area information associated with the past image. Even if the image is taken with the imaging device 1201 of this embodiment, it is judged that there is no reference area information for an image taken with an angle of view that does not include a reference area.

[0148] In S1304, the CPU 201 detects a reference region from the subject image using the same method as the reference region information associated with the past image. As a method for determining the reference region, when there is a known background or tool, the detection is performed using the same method as S802, and the background and tool are determined as the reference region. In addition, when the reference region is determined from a normal tissue region, the detection is performed using the same method as S803 to S808. In S1305, the CPU 201 displays the affected area and the reference area of ​​the image captured in the previous image capture, superimposed on the subject image in the live view. The display method will be described later.

[0149] On the other hand, in S1306, the CPU 201 determines a reference area within the subject image of the photographing target. Here, the reference area can be determined by the series of processes described above with reference to FIG.

[0150] In S1307, the CPU 201 displays an imaging assist figure indicating the reference area superimposed on the subject image displayed as a live view on the rear liquid crystal display of the camera. This process is similar to that of S707.

[0151] [User interface of imaging device] Fig. 14 is a rear view of the digital still camera of this embodiment, showing an example of an image displayed when the injury or disease to be examined is a bedsore. Note that the same reference numerals are used for the same components as in Fig. 4, and the description will be omitted as appropriate.

[0152] Reference numeral 1409 denotes a patient information area, which displays patient information at the time of examination, which is included in the examination information. Reference numeral 1410 denotes a previous radiography date information area, which displays the previous radiography date, which displays information on the most recent past radiography date, which is included in the examination information acquired in S1302. Reference numeral 1411 denotes the contour of the affected area in the previous radiography, which is a figure displayed in S1305. Reference numeral 1412 denotes the contour of the reference area in the previous radiography, which is a figure displayed in S1305.

[0153] The processing after the image capture is the same as that described with reference to S506 to S514 in Fig. 5 in the first embodiment. If a reference area is detected from the subject image in S1304 using the same method as the reference area information associated with the past image, the target luminance and chromaticity information included in the reference area information of the past image is used when calculating the correction coefficient in S913. The target luminance and chromaticity in the target color luminance 108 column of the injury / disease characteristics table 100 may be used, but since it is sufficient that the luminance and chromaticity are consistent between images, for example, the luminance and chromaticity of the reference area of ​​the oldest past image may be used as the target luminance and chromaticity to calculate the correction coefficient.

[0154] As described above, according to the imaging device of the second embodiment, the reference area is determined in the same manner as in the past images, and is corrected to have the same target luminance and chromaticity as the past images, so that it is possible to obtain images that make it easy to compare time-series changes related to the same injury or illness of the same patient in the same part of the body.

[0155] <Third embodiment> Next, a third embodiment of the present invention will be described. In the third embodiment, a method of displaying an assist figure when an affected area cannot be found and when a reference area cannot be found by the same method as in the past image will be described. Cases when a reference area cannot be found by the same method as in the past image include, for example, a case where an injury or disease spreads and there is no part that can be a reference area even if the image is captured with the same angle of view.

[0156] It should be noted that this embodiment is performed using the imaging device described in the first embodiment or the medical system described in the second embodiment, and a description thereof will be omitted.

[0157] [Image capture assist graphic display processing] Fig. 15 is a flowchart showing an imaging assist figure display process in this embodiment, which is performed in place of the process shown in Fig. 7 or Fig. 13. In this embodiment, this process is performed by the imaging device. In Fig. 15, the same processes as those in Fig. 7 are given the same reference numbers, and the description will be omitted as appropriate.

[0158] If the affected area detection is successful in S702, the process proceeds to S705, and if the affected area detection is unsuccessful, the process proceeds to S1503.

[0159] In S1503, CPU 201 determines whether or not the problem has been cured. When determining whether or not the problem has been cured, a screen (options) for the user to confirm is displayed on display unit 204, and the user is prompted to select an option using input unit 205. Then, it is determined whether or not the user has selected "cured." If it is determined that the problem has been cured, the process proceeds to S1505. If it is determined that the problem has not been cured, the process proceeds to S1504. In S1504, the CPU 201 allows the user to select the affected area. In the imaging device of this embodiment, machine learning is used to detect the affected area, but it is possible that the inference fails. In that case, the user is allowed to manually specify the affected area. For example, a message such as "Please surround the affected area" is displayed on the display unit 204 to prompt the user to operate. When a touch panel is installed as the input unit 205, the CPU 201 stores in the RAM 203 the trajectory of the finger on the display 408 operating as the input device, associates it with the coordinates of the image displayed on the display 408, and stores it in the RAM 203 as the affected area position. Then, the image pattern included in the affected area is tracked until a shooting operation is performed by pressing the release button.

[0160] Then, in S703, the CPU 201 displays an imaging assist figure indicating the affected area, superimposed on the image of the subject displayed as a live view on the rear liquid crystal display of the camera.

[0161] Next, in S1506, CPU 201 detects a reference area from the subject image using the same method as the reference area information associated with the previous image. In S1506, CPU 201 determines whether detection of the reference area has been successful, and proceeds to S707 if detection of the reference area has been successful, and proceeds to S1508 if detection of the reference area has been unsuccessful.

[0162] In S1508, CPU 201 determines whether to manually select the reference area. When making the determination, a message such as "Do you want to manually select an area with little time series change?" is displayed on display unit 204, and the user is prompted to select "Yes" or "No." If the user selects "Yes," CPU 201 determines that the user has instructed to manually select the reference area, and proceeds to S1511. On the other hand, if the user selects "No," CPU 201 determines that the user has instructed not to manually select, and proceeds to S1509.

[0163] In S1509, a new reference area is detected, rather than using the method for determining the reference area of ​​a previous image.

[0164] In S1510, it is determined again whether detection of the reference region has been successful, and if successful, the process proceeds to S1512, and if unsuccessful, the process proceeds to S1511.

[0165] In S1511, CPU 201 prompts the user to select a reference area. A message such as "Please surround an area outside the affected area with little change over time" is displayed on display unit 204 to prompt the user to perform the operation. If a touch panel is provided as input unit 205, CPU 201 stores in RAM 203 the trajectory of a finger on display 408 operating as an input device, associates it with the coordinates of an image displayed on display 408, and stores it in RAM 203 as a reference area. Thereafter, the image pattern included in the reference area is tracked continuously until a shooting operation is performed by pressing the release button. In S1512, the CPU 201 determines whether there are multiple candidates for the reference region. The determination is made based on whether there are multiple regions that are not consecutive and have a sufficient number of pixels for performing luminance and chromaticity correction. If it is determined that there are multiple candidates, the process proceeds to S1513, and if there is only one candidate, the process proceeds to S707.

[0166] In S1513, the CPU 201 prompts the user to select a reference region. A message such as "Multiple candidate regions that can be used as references for luminance and chromaticity have been found. Which one would you like to photograph together with the affected area?" is displayed on the display unit 204 to prompt the user to select. When a selection is made, the reference region is stored in the RAM 203 in association with the coordinates of the image displayed on the display 408. After displaying the imaging assist figure for the reference area in S707, the CPU 201 displays the image of the same part captured previously on the edge of the display unit 204 in a superimposed manner in S1415.

[0167] [User interface of imaging device] Fig. 16 is a rear view of the digital still camera of this embodiment, showing an example of an image displayed when the injury or disease to be examined is an injury. Note that the same reference numerals are used for the same components as in Figs. 4 and 14, and the description will be omitted as appropriate.

[0168] 16 is an example in which an imaging assist figure is displayed. Note that, for all buttons other than the power button 401, the operations when the digital still camera is turned on will be described.

[0169] Reference numeral 1611 denotes the contour of the affected area in the previous radiography, which is a figure displayed in S703, and 1612 denotes the contour of the reference area in the previous radiography, which is a figure displayed in S707.

[0170] Reference numerals 1613, 1614, and 1615 denote reduced images of images of the same part of the same patient taken previously, which are displayed in S1415. Images may be taken with a plurality of angles of view, and an example in which three images are taken is shown here.

[0171] Reference numeral 1616 denotes a selection frame showing the previous image that is closest to the image captured this time displayed on the display 408. The image captured with the closest angle of view is selected from the attribute information in the previous image, using the subject distance, unevenness information of the subject, sensor information detecting the camera attitude, and the subject image pattern. If a shooting operation has been performed, the reference area information associated with the closest image from among the previous images is used. Then, by correcting the brightness and chromaticity to be close to those of the past reference area, it is possible to capture an image that is easy to compare with past images. As described above, according to the third embodiment, even if an injury or illness has been cured, the brightness and chromaticity of the image are corrected based on the reference area of ​​the previous image so that the image is close to that of the previous image, making it possible to capture an image that is easy to compare. Also, by displaying the image captured previously superimposed on the edge of the screen, it is possible to support imaging with the same angle of view. Also, in cases where the posture that the patient can take at the time of imaging has changed, it is possible to support medical personnel in recording medical records.

[0172] Alternatively, when the reference area of ​​the past image is lost due to the spread of the injury or illness, etc., a means for re-extracting or selecting the reference area is provided. Then, by correcting the luminance and chromaticity of the past reference area and the newly designated reference area to be close to each other, an image that is easy to compare with the past image can be obtained. In addition, if there are multiple candidates for the reference area that is the standard for brightness and chromaticity, the user can select one. This can help the user to take pictures with an angle of view that suits their preferences and is easy for them to take pictures.

[0173] <Other embodiments> The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0174] <Summary> The disclosure of this embodiment includes the following configuration.

[0175] (Item 1) a detection means for detecting an affected area from images repeatedly captured by the imaging means; an extraction means for extracting a reference region that is determined to have uniform luminance and chromaticity from a region in the image excluding the affected area detected by the detection means; a control means for controlling a display means to superimpose and display an image captured by the imaging means, a first index indicating the affected area, and a second index indicating the reference area; An imaging support device comprising: (Item 2) 2. The imaging support device according to item 1, wherein the predetermined plurality of images are images including the same affected area and the reference region of a same patient. (Item 3) The apparatus further includes an input means for inputting the type of injury or disease to be examined, 3. The imaging support device according to item 1 or 2, wherein the extraction means extracts the reference region based on characteristics according to the type of injury or disease. (Item 4) The imaging support device described in item 3 is characterized in that, when the type of injury or illness has a characteristic in which at least one of a predetermined instrument and a background is reflected when the image is captured, the extraction means extracts at least one of the instrument and the background from the image and extracts the reference area from the extracted instrument or background. (Item 5) the extraction means extracts a partial region excluding the affected area, the partial region being within a range of a predetermined distance from the imaging means to the affected area, and extracts the reference region from the partial region, the reference region being a predetermined distance away from the affected area in the image; 4. The imaging support device according to item 3, wherein the predetermined distance from the affected area in the image is determined according to the type of the injury or disease. (Item 6) 6. The imaging support device according to item 5, wherein the extraction means extracts the reference region when a partial region including a number of pixels equal to or greater than a predetermined threshold value can be extracted. (Item 7) The method further comprises estimating a portion of the reference region, 7. The imaging support device according to item 5 or 6, characterized in that the reference area is invalidated when the probability of the disease occurring in the estimated area is higher than a predetermined probability of occurrence. (Item 8) The imaging support device described in any one of items 1 to 7, characterized in that when a plurality of reference areas are extracted, the control means displays the second indicator indicating at least one of the plurality of reference areas that is closest to the detected affected area. (Item 9) The imaging support device described in any one of items 1 to 8, further comprising a storage means for storing, when an instruction to record an image is given, the image obtained when the instruction to record is given, information on the extracted affected area, information on the position of the reference area, and information on the method of extracting the reference area in association with each other. (Item 10) 10. The imaging support device according to item 9, wherein, when there is a past image of the same affected area of ​​the same patient, the extraction means extracts the reference area using the same method as the extraction method of the reference area extracted from the past image. (Item 11) 11. The imaging support device according to item 10, wherein the control means further displays an index indicating the diseased area and the reference area in the past image. (Item 12) 10. The imaging support device according to item 9, wherein, when there are previous images of the same affected area of ​​the same patient, the control means displays reduced images of the previous images side by side. (Item 13) 13. The imaging support device according to any one of items 1 to 12, further comprising a notification unit that notifies the user that an affected area has not been detected when the detection unit has not been able to detect the affected area. (Item 14) Item 14. The imaging support device according to item 13, wherein the notification means further notifies the user that the reference area could not be extracted if the extraction means is unable to extract the reference area. (Item 15) further comprising an operation means for manually accepting designation of an affected area when the detection means is unable to detect the affected area; 15. The imaging support device according to any one of items 1 to 14, wherein, when the detection means is unable to detect the affected area, the control means displays a message indicating that the designation of the affected area is accepted. (Item 16) The imaging support device described in item 15, characterized in that if the detection means cannot detect the affected area, the control means displays an option of whether the injury or illness has been cured, and if the injury or illness has not been cured, displays a message indicating that the designation of the affected area is accepted. (Item 17) 17. The imaging support device according to item 15 or 16, wherein, when the extraction means is unable to extract the reference region, the control means displays a message indicating that the reference region designation is accepted. (Item 18) An imaging means for capturing and outputting an image; An imaging support device according to any one of items 1 to 17, An imaging device comprising: (Item 19) Item 19. The imaging device according to item 18, further comprising a storage means for storing, when an instruction is given to record an image output from the imaging means, the image obtained when the instruction to record is given, information on the extracted affected area, information on the position of the reference area, and information on a method of extracting the reference area in association with each other. (Item 20) A calculation means for calculating, when an instruction is given to record an image output from the imaging means of the imaging device according to item 18, a correction coefficient for adjusting the luminance and chromaticity of the reference region to a predetermined luminance and chromaticity for the image obtained when the instruction to record is given; a correction means for correcting the image using the correction coefficient; 13. An image processing device comprising: (Item 21) A calculation means for calculating a correction coefficient for adjusting the luminance and chromaticity of the reference area to a predetermined luminance and chromaticity for an image stored by the imaging device according to item 19; a correction means for correcting the image using the correction coefficient; 13. An image processing device comprising: (Item 22) 22. The image processing device according to item 20 or 21, wherein the predetermined luminance and chromaticity are determined according to the injury or disease to be inspected. (Item 23) A calculation means for calculating a correction coefficient for making the luminance and chromaticity of the reference region the same between a plurality of images including the same affected area of ​​the same patient and the reference region among the images stored by the imaging device according to item 19, a correction means for correcting the image using the correction coefficient; 13. An image processing device comprising: (Item 24) a detection step of detecting an affected area from images repeatedly captured by the imaging means; an extraction step of extracting a reference region that has uniform luminance and chromaticity in a plurality of predetermined images from a region in the image excluding the affected area detected in the detection step; a control step of controlling a display means to superimpose and display the image captured by the imaging means, a first index indicating the affected area, and a second index indicating the reference area; An imaging assistance method comprising: (Item 25) A calculation step of calculating a correction coefficient for adjusting the luminance and chromaticity of the reference region to a predetermined luminance and chromaticity for an image obtained when recording of an image output from the imaging means of the imaging device according to item 18 is instructed; a correction step of correcting the image using the correction coefficient; 13. An image processing method comprising: (Item 26) A calculation step of calculating a correction coefficient for the image stored by the imaging device according to item 19, which changes the luminance and chromaticity of the reference area to a predetermined luminance and chromaticity; a correction step of correcting the image using the correction coefficient; 13. An image processing method comprising: (Item 27) A calculation step of calculating a correction coefficient for making the luminance and chromaticity of the reference region the same between a plurality of images including the same affected area of ​​the same patient and the reference region among the images stored by the imaging device according to item 19; a correction step of correcting the image using the correction coefficient; 13. An image processing method comprising: (Item 28) A program for causing a computer to function as each of the means of the imaging support device described in any one of items 1 to 17. (Item 29) Item 29. A computer-readable storage medium storing the program according to item 28.

[0176] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0177] 100: injury / disease characteristic table, 200: imaging unit, 201: CPU, 202: ROM, 203: RAM, 204: display unit, 205: input unit, 211: image processing unit, 1002, 1003, 1102: diseased area, 1004, 1103, 1104: normal tissue area, 1006, 1007, 1106, 1107, 1108: imaging assist figure, 1201: imaging device, 1202: network, 1203: electronic medical record display terminal, 1204: electronic medical record system, 1205: image management system, 1206: image processing device, 1207: diseased area, 1208: examination information database, 1209: facility information database, 1210: image information database

Claims

1. a detection means for detecting an affected area from images repeatedly captured by the imaging means; an extraction means for extracting a reference region that is determined to have uniform luminance and chromaticity in a plurality of predetermined images from a region in the image excluding the affected area detected by the detection means; a control means for controlling a display means to superimpose and display an image captured by the imaging means, a first index indicating the region of the diseased area, and a second index indicating the reference region; An imaging support device comprising:

2. 2. The imaging support device according to claim 1, wherein the predetermined plurality of images are images including the same affected area of ​​a same patient and the reference region.

3. The apparatus further includes an input means for inputting the type of injury or disease to be examined, 2. The imaging support device according to claim 1, wherein the extraction means extracts the reference region based on a feature corresponding to the type of the injury or disease.

4. The imaging support device described in claim 3, characterized in that when the type of injury or illness has a characteristic in which at least one of a predetermined instrument and a background is reflected when the image is captured, the extraction means extracts at least one of the instrument and the background from the image, and extracts the reference area from the extracted instrument or background.

5. the extraction means extracts a partial region excluding the affected area, the partial region being within a range of a predetermined distance from the imaging means to the affected area, and extracts the reference region from the partial region, the reference region being a predetermined distance away from the affected area in the image; 4. The imaging support device according to claim 3, wherein the predetermined distance from the affected area in the image is determined according to the type of the injury or disease.

6. 6. The imaging support device according to claim 5, wherein the extraction means extracts the reference region when a partial region including a number of pixels equal to or greater than a predetermined threshold value can be extracted.

7. The method further comprises estimating a portion of the reference region, 6. The imaging support device according to claim 5, wherein the reference region is invalidated when the probability of the disease occurring in the estimated region is higher than a predetermined probability of the disease occurring.

8. The imaging support device according to claim 1, characterized in that, when a plurality of the reference areas are extracted, the control means displays the second indicator indicating at least one of the plurality of reference areas that is closest to the detected affected area.

9. The imaging support device according to claim 1, further comprising a memory means for storing, when an instruction to record an image is given, the image obtained when the instruction to record is given, information on the extracted affected area, information on the position of the reference area, and information on the method of extracting the reference area in association with each other.

10. 10. The imaging support device according to claim 9, wherein, when there is a past image of the same affected area of ​​the same patient, the extraction means extracts the reference area using the same method as the extraction method of the reference area extracted from the past image.

11. 11. The imaging support device according to claim 10, wherein the control means further displays indicators indicating the diseased area and the reference area in the previous image.

12. 10. The imaging support device according to claim 9, wherein, when there are previous images of the same affected area of ​​the same patient, the control means displays reduced images of the previous images side by side.

13. 2. The imaging support device according to claim 1, further comprising a notification unit that notifies a user that an affected area has not been detected when the detection unit is unable to detect the affected area.

14. 14. The imaging support device according to claim 13, wherein the notification means further notifies the user that the reference area could not be extracted when the extraction means has failed to extract the reference area.

15. further comprising an operation means for manually accepting designation of an affected area when the detection means is unable to detect the affected area; 2. The imaging support device according to claim 1, wherein, when the detection means is unable to detect the affected area, the control means displays a message to the effect that the designation of the affected area is accepted.

16. The imaging support device described in claim 15, characterized in that if the detection means is unable to detect the affected area, the control means displays an option as to whether the injury or illness has been cured, and if the injury or illness has not been cured, displays a message indicating that the designation of the affected area is accepted.

17. 16. The imaging support device according to claim 15, wherein, when the extraction means is unable to extract the reference region, the control means displays a message to the effect that the designation of the reference region is accepted.

18. An imaging means for capturing and outputting an image; The imaging support device according to any one of claims 1 to 17, An imaging device comprising:

19. 19. The imaging device according to claim 18, further comprising a storage means for storing, when an instruction is given to record the image output from the imaging means, the image obtained when the instruction to record is given, information on the extracted affected area, information on the position of the reference area, and information on a method of extracting the reference area, in association with each other.

20. a calculation means for calculating, when an instruction is given to record an image output from the imaging means of the imaging device according to claim 18, a correction coefficient for adjusting the luminance and chromaticity of the reference region to a predetermined luminance and chromaticity for the image obtained when the instruction to record is given; a correction means for correcting the image using the correction coefficient; 13. An image processing device comprising:

21. 21. The image processing apparatus according to claim 20, wherein the predetermined luminance and chromaticity are determined according to an injury or disease to be inspected.

22. a calculation means for calculating a correction coefficient for adjusting the luminance and chromaticity of the reference area to a predetermined luminance and chromaticity for an image stored by the imaging device according to claim 19; a correction means for correcting the image using the correction coefficient; 13. An image processing device comprising:

23. 23. An image processing apparatus according to claim 22, wherein the predetermined luminance and chromaticity are determined according to an injury or disease to be examined.

24. a calculation means for calculating a correction coefficient for making the luminance and chromaticity of the reference region uniform among a plurality of images including the same affected area of ​​the same patient and the reference region among the images stored by the imaging device according to claim 19; a correction means for correcting the image using the correction coefficient; 13. An image processing device comprising:

25. a detection step of detecting an affected area from images repeatedly captured by the imaging means; an extraction step of extracting a reference region that has uniform luminance and chromaticity in a plurality of predetermined images from a region in the image excluding the affected area detected in the detection step; a control step of controlling a display means to superimpose and display the image captured by the imaging means, a first index indicating the region of the diseased area, and a second index indicating the reference region; An imaging assistance method comprising:

26. a calculation step of calculating, when an instruction is given to record an image output from the imaging means of the imaging device according to claim 18, a correction coefficient for adjusting the luminance and chromaticity of the reference region to a predetermined luminance and chromaticity for the image obtained when the instruction to record is given; a correction step of correcting the image using the correction coefficient; 13. An image processing method comprising:

27. A calculation step of calculating a correction coefficient for adjusting the luminance and chromaticity of the reference area to a predetermined luminance and chromaticity for an image stored by the imaging device according to claim 19; a correction step of correcting the image using the correction coefficient; 13. An image processing method comprising:

28. a calculation step of calculating a correction coefficient for making the luminance and chromaticity of the reference region the same among a plurality of images including the same affected area of ​​the same patient and the reference region among the images stored by the imaging device according to claim 19, a correction step of correcting the image using the correction coefficient; 13. An image processing method comprising:

29. A program for causing a computer to function as each of the means of the imaging support device according to any one of claims 1 to 17.

30. 30. A computer readable storage medium storing the program according to claim 29.

Images