Image processing device, image processing method, and program

Abstract

To make it possible to quantitatively evaluate a wound that can be in various states.SOLUTION: An image processing device acquires distance information about a distance from an imaging device to a subject. In addition, the image processing device extracts an affected area of the subject from image data of the subject captured by the imaging device, and further extracts color information of the affected area. Then, the image processing device determines the state of the affected area on the basis of the distance information and the color information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for evaluating an affected area of ​​a photographed subject. [Background technology]

[0002] When a person or animal lies down, the body weight puts pressure on the contact area between the body and the floor, mattress, or bedding underneath. If the same posture continues, the contact area may become insufficient in blood circulation and the surrounding tissue may become necrotic. The state in which the tissue becomes necrotic is called a bedsore or pressure sore. Once a bedsore occurs, it is necessary to periodically evaluate and manage the condition of the bedsore. As described in Non-Patent Document 1, DESIGN-R (registered trademark), an evaluation index developed by the Academic Education Committee of the Japanese Society of Bedsores, is known as an evaluation index for bedsores. DESIGN-R is a tool for evaluating the healing process of wounds, including bedsores. Currently, evaluation of bedsores is often determined by visually inspecting the affected area and whether or not the condition of the evaluation index is confirmed. However, since manual and visual evaluation methods vary depending on the person who performs the measurement, accuracy is required. In addition, touching the affected area during work causes pain to the patient and also involves the risk of infection, so it is required to measure and evaluate each evaluation index without contact. On the other hand, the diagnostic device of Patent Document 1 discloses a method for determining the depth region corresponding to a lesion from the color of the lesion in a captured image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-112270 A [Non-patent literature]

[0004] [Non-Patent Document 1] Shorinsha Bed Sore Guidebook 2nd Edition Based on the Bed Sore Prevention and Management Guidelines (4th Edition) Edited by the Japanese Bed Sore Society ISBN13 978-4796523608 Pages 23 and 24 Summary of the Invention [Problem to be solved by the invention]

[0005] However, wounds such as bedsores can have various depths and colors. The conventional technology disclosed in Patent Document 1 may erroneously evaluate and measure wounds such as bedsores that can have various conditions. In other words, the technology of Patent Document 1 often cannot quantitatively evaluate wounds that can have various conditions.

[0006] Therefore, an object of the present invention is to enable quantitative evaluation of wounds, which may be in a variety of states. [Means for solving the problem]

[0007] The image processing device of the present invention includes an acquisition means for acquiring distance information relating to a distance from an imaging device to a subject, and an extraction means for extracting an affected area of ​​the subject from image data of the subject captured by the imaging device, and further extracting color information of the affected area. A calculation means for generating depth information corresponding to the affected area based on the distance information; The above Depth and a determining means for determining a state of the affected area based on the color information and the color information. The extraction means extracts a predetermined area that is set near the affected area but does not include the affected area from the image data, and sets at least three reference points in the predetermined area. The calculation means sets a reference plane that includes the reference points, and generates depth information corresponding to the affected area based on the reference plane. The present invention is characterized by the above. Effect of the Invention

[0008] According to the present invention, it is possible to quantitatively evaluate wounds which may be in a variety of states. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overview of an image processing system according to an embodiment; [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. [Diagram 3]FIG. 2 illustrates an example of a hardware configuration of an image processing apparatus. [Figure 4] 4 is a flowchart of the operation of the image processing system according to the embodiment. [Diagram 5] FIG. 13 is a diagram for explaining a method for calculating the area of ​​a region. [Figure 6] 13 is a diagram showing an example of a display in which the extraction result of the affected area and information indicating the size are superimposed. FIG. [Figure 7] 13 is a diagram showing an example of a display in which the extraction result of the affected area and information including the major axis and minor axis are superimposed on each other. FIG. [Figure 8] 13 is a flowchart of a process for acquiring the depth of an affected area. [Figure 9] 13 is an explanatory diagram of an example of setting a measurement standard for the extraction result and depth of an affected area; FIG. [Figure 10] 13 is a flowchart of a process for acquiring the depth of an affected area in a curvature portion. [Figure 11] 13A and 13B are explanatory diagrams of the extraction result of the affected area of ​​the curvature part and the depth measurement standard setting method. [Figure 12] 13 is a flowchart of an analysis process of five states of an affected area. [Figure 13] FIG. 1 is a schematic diagram showing granulation tissue, eschar, slough, exudate, and exposed bone. [Figure 14] 11 is a flowchart of a process for determining the presence or absence of a pocket and measuring the depth of a pocket. [Figure 15] FIG. 13 is an explanatory diagram of an example in which a pocket is present in the affected area. [Figure 16] FIG. 11 is an explanatory diagram of a method for extracting calculated values ​​of a cross section. [Figure 17] FIG. 11 is an explanatory diagram of a method for calculating a slope. [Figure 18] FIG. 13 is an explanatory diagram of an example of a method for measuring the depth of a pocket in a cross section. [Figure 19] 11 is an explanatory diagram of an example of a pocket depth measuring method by rotating an imaging device. FIG. [Figure 20] FIG. 13 is a diagram illustrating an example of a user assistance function. [Figure 21] FIG. 13 is an explanatory diagram of a method for allowing a user to input affected area information. [Figure 22] 13 is a flowchart of a process for displaying a comparison result before and after the healing process. [Diagram 23] FIG. 13 is an explanatory diagram of comparison points. [Figure 24] FIG. 13 is a diagram showing an example of a distance map in an affected area. [Diagram 25] FIG. 13 is a diagram showing an example of a superimposed display of improved and deteriorated parts in an affected area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments will be described with reference to the drawings. Note that the present invention is not limited to the following embodiments. In the following description, the same configurations, processes, etc. will be described with the same reference numerals.

[0011] Before describing the configuration and processing of the image processing device of this embodiment, evaluation items for evaluating the state and healing process of wounds such as bedsores will be described. DESIGN-R (registered trademark), an evaluation index developed by the Academic Education Committee of the Japanese Society of Bedsores, is known as an evaluation index for wounds such as bedsores. DESIGN-R is named after the initials of each evaluation item: Depth, Exudate, Size, Inflammation / Infection, Granulation, and Necrotic tissue. In the following, the following items will be described: Depth, Exudate, Size, Inflammation / Infection, Granulation, and Necrotic tissue. The evaluation items of DESIGN-R also include pockets.

[0012] Depth is evaluated at the wound base, which is the deepest part of the wound in relation to the wound edges, and the deeper the wound base reaches into the epidermis, dermis, subcutaneous tissue, and bone, the more severe the injury is judged to be. Size is evaluated by measuring the long and short diameters of the skin damage area and multiplying them together; the larger the value, the greater the severity. A pocket is evaluated by measuring the long and short diameters of the area of ​​skin damage that contains the pocket, multiplying them together, and subtracting the size of the area of ​​skin damage from the result.The larger the value, the more severe the condition is. Typically, the depth, size, and pocket of a pressure ulcer are assessed based on manual measurements taken by placing a tape measure on the affected area. Exudate is a yellow body fluid that seeps out from the wound surface and is evaluated based on the frequency with which the dressing is changed to remove it. The more frequently the dressing is changed, the more severe the condition is judged to be. Inflammation / infection is evaluated based on the presence or absence of signs of inflammation or infection, such as redness, swelling, and heat around the wound. If there are clear local signs of infection or systemic effects such as fever, the severity is judged to be high. There are two types of granulation: benign granulation, which is bright red and moderately moist, and poor granulation, which is pink and overly moist. Granulation is evaluated based on the proportion of the wound that is benign granulation, and the lower the proportion of benign granulation, the more severe the condition is. There are two types of necrotic tissue: slough, which is a soft, yellowish-white necrotic tissue, and eschar, which is a hard, cohesive, black necrotic tissue. Necrotic tissue is evaluated based on the presence or absence of these, with the severity being judged to be increasing in the order of eschar, slough, and no necrotic tissue. Currently, the assessment of exudate, inflammation, infection, granulation tissue, and necrotic tissue in a bedsore is often determined by visually inspecting the affected area and checking whether the state of each assessment indicator is confirmed.

[0013] However, as described above, bedsores can be in various states such as exudate, granulation tissue, and necrotic tissue, so it is often difficult to accurately evaluate the wound. For example, when benign granulation tissue is present at two locations, the wound edge and the wound bottom, the benign granulation tissue is bright red regardless of the depth, so there is a risk that the color and depth area correspondence table disclosed in Patent Document 1 may erroneously determine that the two are in the same depth area. In addition, when both black eschar and yellow exudate are present at the same depth in the wound, there is a risk that the conventional technology disclosed in Patent Document 1 may erroneously determine that the two are in different depth areas.

[0014] Therefore, in the image processing device of this embodiment, as described below, images of the stem region are used, and analysis processing is performed based on color information and distance information, thereby enabling quantitative evaluation of wounds which can take on a variety of conditions, such as bedsores. An image processing system as an application example of the image processing device according to this embodiment will be described below. FIG. 1 is a diagram showing an overview of an image processing system 1 according to an embodiment. The imaging system 1 is composed of an imaging device 200, which is a handheld portable device, and an image processing device 300. In this embodiment, an example is given in which the imaging device 200 and the image processing device 300 are configured separately, but they may be a single device in which the functions of the imaging device and the image processing device are integrated. Examples of a single device in which the functions of the imaging device and the image processing device are integrated include mobile information terminals such as smartphones and tablet terminals that are equipped with a camera with an autofocus function and a high-performance CPU, etc.

[0015] In the image processing system 1 according to this embodiment, the imaging device 200 captures an image of an affected area 102 of a patient, which is a subject 101, to obtain image data. The imaging device 200 also obtains subject distance information, which is the distance from the imaging device 200 to the subject 101 and the affected area 102 when the stem area 102 is captured. The imaging device 200 then transmits the image data and subject distance information to the image processing device 300. The image processing device 300 extracts the affected area 102 from the received image data, and analyzes the wound surface condition of the affected area (the state of the bedsore in this embodiment) based on the color information of the stem area and the subject distance. In this embodiment, the state of the bedsore analyzed by the image processing device 300 is assumed to be size, depth, granulation tissue, necrotic tissue, exudate, and pocket, following the DESIGN-R, which is an evaluation index for bedsores. In this embodiment, the affected area 102 is described as a bedsore, but the present invention is not limited to this and may be a wound that opens, such as a laceration or incision. In this embodiment, the affected area 102 of the subject 101 is described as a bedsore on the buttocks.

[0016] FIG. 2 is a diagram showing an example of a hardware configuration of the imaging device 200 included in the image processing system 1. As shown in FIG. The imaging device 200 is assumed to be a typical single-lens camera or a compact digital camera, or may be a smartphone or tablet terminal equipped with a camera with an autofocus function. The imaging unit 211 has a lens group 212, a shutter 213, and an image sensor 214. The imaging unit 211 can change the focus position and zoom magnification by changing the positions of multiple lenses included in the lens group 212. The lens group 212 also includes an aperture for adjusting the amount of exposure.

[0017] The image sensor 214 is composed of a charge-accumulation type solid-state image sensor such as a CCD or CMOS sensor that converts an optical image into image data. Reflected light from a subject that has passed through the lens group 212 and the shutter 213 forms an image on the imaging surface of the image sensor 214, and the image sensor 214 generates an electrical signal corresponding to the subject image and outputs image data based on the electrical signal.

[0018] The shutter 213 exposes or blocks light to the image sensor 214 by opening and closing the shutter blade members, thereby controlling the exposure time of the image sensor 214. Instead of the shutter 213, an electronic shutter that controls the exposure time by driving the image sensor 214 may be used. When the electronic shutter is performed with a CMOS sensor, the image sensor 214 performs a reset scan for each pixel or for each area consisting of multiple pixels (for example, for each line) to set the amount of accumulated charge of the pixel to zero. After that, the image sensor 214 performs a scan to read out a signal according to the amount of accumulated charge after a predetermined time has elapsed for each pixel or area where the reset scan was performed.

[0019] The zoom control circuit 215 controls the optical magnification of the lens group 212 by controlling a motor (not shown) for driving a zoom lens included in the lens group 212. Note that the lens group 212 may be a fixed-focus lens group without a zoom function, in which case the zoom control circuit 215 is not necessary.

[0020] The distance measuring system 216 calculates distance information to the subject. As the distance measuring system 216, a general phase difference type distance measuring sensor mounted on a single-lens camera or a system using a TOF (Time Of Flight) sensor may be used. The TOF sensor is a sensor that measures the distance to the object based on the time difference (or phase difference) between the transmission timing of an irradiated wave and the reception timing of a reflected wave of the irradiated wave reflected by the object. Furthermore, the distance measuring system 216 may use a PSD type using a PSD (Position Sensitive Device) as a light receiving element.

[0021] The image sensor 214 may have a plurality of photoelectric conversion regions for each pixel, and may be configured to have different pupil positions corresponding to the plurality of photoelectric conversion regions included in a common pixel. With such a configuration, the distance measurement system 216 can obtain distance information for each pixel or region position from the phase difference between images obtained from the photoelectric conversion regions corresponding to the respective pupil regions output from the image sensor 214.

[0022] In addition, the ranging system 216 may be configured to obtain distance information in a predetermined ranging area or multiple ranging areas within the image, or may be configured to obtain a distance map showing the distribution of distance information for a large number of pixels or areas within the image. Alternatively, the distance measurement system 216 may perform TV-AF or contrast AF by extracting and integrating high frequency components of image data, determining the position of the focus lens where the integral value is maximized, and obtaining distance information from the position of the focus lens. In this embodiment, the distance measurement system 216 will be described as being configured to obtain a distance map that indicates the distribution of distance information for a large number of pixels or regions within an image.

[0023] The image processing circuit 217 performs predetermined image processing on the image data output from the image sensor 214. The image processing circuit 217 performs various image processing such as white balance adjustment, gamma correction, color interpolation or demosaicing, and filtering on the image data output from the imaging unit 211 or the image data recorded in the internal memory 221. In addition, the image processing circuit 217 performs compression processing on the image data that has been subjected to image processing in accordance with a standard such as JPEG.

[0024] An AF control circuit 218 determines the position of a focus lens included in the lens group 202 based on distance information obtained by the distance measurement system 216, and controls a motor that drives the focus lens. The communication device 219 is a wireless communication module for communicating with external devices such as the image processing device 300 via a wireless communication network (not shown). A specific example of the network is a network based on the Wi-Fi (registered trademark) standard. Note that communication using Wi-Fi may be realized via a router. The communication device 219 may also realize communication through a wired communication interface such as a USB or a LAN. The communication device 219 is assumed to be built in the imaging device 200, but may be an external device connected to the imaging device 200.

[0025] The system control circuit 220 has a CPU (Central Processing Unit) and performs overall control by controlling each part of the imaging device 200 in accordance with a program stored in an internal memory 221. The system control circuit 220 also performs overall control of the imaging unit 211, zoom control circuit 215, distance measurement system 216, image processing circuit 217, AF control circuit 218, etc. The system control circuit 220 may use an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit) instead of a CPU.

[0026] The internal memory 221 is composed of a rewritable memory such as a flash memory, SDRAM, etc. The internal memory 221 temporarily stores various setting information such as information on the focus position at the time of imaging and zoom magnification required for the operation of the imaging device 200, image data captured by the imaging unit 211, image data processed by the image processing circuit 217, etc. The internal memory 211 may also temporarily store analysis data such as image data and information indicating the size of a subject received by the communication device 219 through communication with the image processing device 300.

[0027] The external memory I / F (Interface) 222 is an interface with a non-volatile storage medium such as an SD card or a CF card (Compact Flash is a registered trademark) that can be loaded into the imaging device 200. The external memory I / F 222 records image data processed by the image processing circuit 217, image data and analysis data received by the communication device 219 through communication with the image processing device 300, and the like, in the loadable storage medium. The external memory I / F 222 can also read out image data recorded in the loadable storage medium and output it to the outside of the imaging device 200.

[0028] The display device 223 is a device configured with a TFT (Thin Film Transistor) liquid crystal, an organic EL display, an EVF (Electronic Viewfinder), etc. The display device 223 displays an image based on image data temporarily stored in the internal memory 221, an image based on image data stored in a loadable storage medium, a setting screen of the imaging device 200, etc.

[0029] The operation unit 224 is composed of, for example, buttons, switches, keys, and a mode dial provided on the imaging device 200, or a touch panel provided on the display device 223. Commands such as mode settings and shooting instructions by the user are sent to the system control circuit 220 via the operation unit 224.

[0030] The common bus 225 is connected to the imaging unit 211, the zoom control circuit 215, the distance measurement system 216, the image processing circuit 217, the AF control circuit 218, the communication device 219, the system control circuit 220, the internal memory 221, the external memory I / F 222, the display device 223, and the operation unit 224. The common bus 225 is a signal line for transmitting and receiving signals between each of these connected blocks.

[0031] FIG. 3 is a diagram showing an example of a hardware configuration of the image processing device 300 included in the image processing system 1. As shown in FIG. The image processing device 300 includes a calculation unit 311 , a storage unit 312 , a communication unit 313 , an output unit 314 , and an auxiliary calculation unit 317 . The storage device 312 is composed of a main storage device 315 (such as a ROM or a RAM) and an auxiliary storage device 316 (such as a magnetic disk device or an SSD (Solid State Drive)). The communication device 313 is configured as a wireless communication module for communicating with external devices via a communication network. The output device 314 outputs data processed or generated by the arithmetic device 311 and data stored in the storage device 312 to a display or printer connected to the image processing device 300, or to an external network.

[0032] The auxiliary arithmetic device 317 is an auxiliary arithmetic IC used under the control of the arithmetic device 311. The auxiliary arithmetic device 317 is configured by a GPU (Graphic Processing Unit) as an example. The GPU is originally a processor for image processing, but since it has multiple multiply-accumulators and excels in matrix calculations, it is often used as a processor for signal learning processing. And, GPUs are generally used in deep learning processing as well. For example, the auxiliary arithmetic device 317 may be a Jetson TX2 module by NVIDIA. Note that the auxiliary arithmetic device 317 may be an FPGA, an ASIC, or the like. In the case of the image processing device 300 of this embodiment, the auxiliary calculation device 317 performs processing such as extracting the affected area 102 of the subject 101 and color information of the affected area from the image data.

[0033] The arithmetic unit 311 is configured with a CPU, and by executing a program stored in the storage device 312, realizes various functions including arithmetic processing for determining the size and length of the affected area 102 extracted by the auxiliary arithmetic unit 317. The arithmetic unit 311 also generates data to be displayed on the display device 223 of the imaging device 200, generates data to be output via the output device 314, controls the order in which each function is executed, and so on.

[0034] The image processing device 300 may have one or more arithmetic device 311 and storage device 312. That is, at least one or more processing devices (CPUs) and at least one storage device are connected, and when at least one or more processing devices execute a program stored in at least one or more storage devices, the image processing device 300 executes each function described below. The arithmetic device 311 may be realized by an FPGA, an ASIC, or the like, instead of a CPU.

[0035] 4 is a flowchart showing the operations of the imaging device 200 and the image processing device 300 of the image processing system 1 according to the embodiment. In the following description, each processing step in the flowchart will be represented by adding a reference number to S. That is, for example, processing step 401 will be represented as S401. The same will be true for other flowcharts described later.

[0036] In FIG. 4, steps S401 to S420 are processes performed by the imaging device 200, and steps S431, S441 to S445, and S451 to S456 are processes performed by the image processing device 300. First, the imaging device 200 and the image processing device 300 are each connected to a network (not shown) conforming to the Wi-Fi standard, which is a wireless LAN standard. In S431, the image processing device 300 searches for the imaging device 200 connected via the network, and in S401, the imaging device 200 responds to the image processing device 300. For example, UPnP (Universal Plug and Play) is used as a technology for searching for devices via a network. In UPnP, individual devices are identified by UUID (Universally Unique IDentifier).

[0037] When the imaging device 200 is connected to the image processing device 300, in S402, the live view process is started. That is, the imaging unit 211 generates image data by capturing an image, and the image processing circuit 217 applies development processing to this image data required to generate image data for live view display. The imaging device 200 repeatedly performs these processes for each frame, thereby displaying a live view image at a predetermined frame rate on the screen of the display device 223.

[0038] In S403, the distance measuring system 216 obtains distance information of the subject by any of the above-mentioned methods, and the AF control circuit 218 starts AF processing to drive and control the lens group 212 so that the subject is in focus. When the focus position is adjusted by the above-mentioned TV-AF or contrast AF, the distance measuring system 216 obtains distance information from the position of the focus lens in a focused state to the subject 101 on which the focus is adjusted. The position to be focused may be the subject located in the center of the screen, or the subject located at the closest position to the imaging device 200. In addition, if the above-mentioned distance map of the subject is obtained, the distance measuring system 216 may estimate a region to be noted from the distance map, and the AF control circuit 218 may focus on that position. Alternatively, if the position of the bedsore 102 has already been identified in the live view image by the image processing device 300, the distance measuring system 216 may obtain distance information for that position, and the AF control circuit 218 may focus on that position. The imaging device 200 repeatedly performs AF processing while displaying a live view image until it detects in S410 (described later) that the release button has been pressed.

[0039] In S404, the image processing circuit 217 performs development and compression processing on any image data of the images captured for live view, to generate image data in, for example, JPEG format. Then, the image processing circuit 217 performs resizing processing on the compressed image data to reduce the size of the image data.

[0040] In S405, the communication device 219 acquires the image data generated and resized in S404 and the distance information calculated in S403. The communication device 219 also acquires information on the zoom magnification and information on the size (number of pixels) of the resized image data. Note that if the imaging unit 211 has a single focus and does not have a zoom function, the information on the zoom magnification is not necessary.

[0041] In S406, the communication device 219 transmits the image data and distance information acquired in S405, and one or more pieces of information such as zoom magnification and size information, to the image processing device 300 by wireless communication. Since the larger the size of the image data to be transmitted, the longer the wireless communication time, the size of the image data after resizing in S405 is determined taking into consideration the allowable communication time. However, if the image data is made too small, it affects the accuracy of the affected area extraction process performed by the image processing device 300 in S442 described later, so in addition to the communication time, the accuracy of the affected area extraction process must also be considered. Furthermore, the processes from S404 to S406 may be performed for each frame, or may be performed once every few frames.

[0042] Now, the processing by the image processing device 300 will be described. In S441, the communication device 313 of the image processing device 300 receives the image data output from the communication device 219 of the imaging device 200 and one or more pieces of information including the distance information. Next, in S442, the auxiliary arithmetic unit 317 extracts the affected area 102 of the subject 101 from the image data received in S441. In this embodiment, a method of performing semantic area segmentation by deep learning is used as a method of extracting the affected area. That is, a high-performance computer for machine learning (not shown) learns a neural network model using a plurality of actual images of bedsores as teacher data in advance to generate a trained model. The auxiliary arithmetic unit 317 receives the trained model from the high-performance computer and estimates the area of ​​the bedsore, which is the affected area, from the image data based on the trained model. In this embodiment, a fully convolutional network (FCN (Fully Convolutional Network)), which is a segmentation model using deep learning, is applied as an example of a neural network model. Here, the inference of deep learning is processed by the auxiliary arithmetic unit 317, which is good at parallel execution of product-sum operations. Note that the inference process may be executed by an FPGA, an ASIC, or the like. In addition, the area segmentation may be realized using other deep learning models. In addition, the segmentation method is not limited to deep learning, and may be, for example, graph cut, region growing, edge detection, rule-and-conquer, etc. Furthermore, the auxiliary calculation device 317 may perform learning of a neural network model using images of bedsores as training data.

[0043] In S443, the arithmetic unit 311 calculates the area of ​​the affected area as information indicating the size of the affected area 102 extracted by the auxiliary arithmetic unit 317. FIG. 5 is a diagram for explaining a method for calculating the area of ​​the affected region. The imaging device 200, which is a general camera, can be treated as a pinhole model as shown in Fig. 5. Incident light 501 passes through the principal point of the lens 212a and is received on the imaging surface of the image sensor 214. When the lens group 212 is approximated to a single lens 212a with no thickness, the two principal points, the front principal point and the rear principal point, can be considered to be coincident. By adjusting the focal position of the lens 212a so that the subject image is formed on the imaging surface of the image sensor 214, the imaging device 200 can focus on the subject 504.

[0044] Moreover, by changing the focal length 502, which is the distance from the imaging plane to the lens principal point, the angle of view θ503 is changed, and the zoom magnification changes. At this time, the width 506 of the subject on the in-focus plane is geometrically determined from the relationship between the angle of view θ503 of the imaging device 200 and the subject distance 505. This width 506 of the subject is calculated using a trigonometric function. That is, the width 506 of the subject can be determined from the relationship between the angle of view θ503, which changes according to the focal length 502, and the subject distance 505. Then, the value of the width 506 of the subject is divided by the number of pixels on a line of the image, to obtain the length on the in-focus plane corresponding to one pixel on the image.

[0045] Therefore, the arithmetic unit 311 calculates the area of ​​the affected area as the product of the number of pixels in the extracted area obtained from the result of extraction of the affected area by S442 and the area of ​​one pixel obtained from the length on the focal plane corresponding to one pixel on the image. The length on the focal plane corresponding to one pixel on the image corresponding to the combination of the focal length 502 and the subject distance 505 may be obtained in advance and prepared in advance as table data. In this case, the image processing device 300 may also store table data corresponding to the imaging device 200 in advance.

[0046] In order to correctly calculate the area of ​​the affected region using the above method, it is assumed that the subject 504 is flat and that this plane is perpendicular to the optical axis. If the distance information acquired in S441 is distance information or a distance map at multiple positions in the image data, the calculation device 311 may detect the inclination or change in the depth direction of the subject and calculate the area based on the detected inclination or change.

[0047] In S444, the arithmetic unit 311 generates image data by superimposing information indicating the extraction result of the affected area and information indicating the size of the affected area on the image data used for extracting the affected area. 6 is a diagram for explaining a method of superimposing information indicating the extraction result of the affected area and information indicating the size of the affected area on image data. Fig. 6 shows an example in which the affected area 102 is located near the buttocks.

[0048] An image 601 in Fig. 6(a) is an image displayed using image data before the superimposition process, and includes a subject 101 and an affected area 102. A superimposed image 602 in Fig. 6(b) is an image based on image data after the superimposition process. 6(b), the arithmetic device 311 superimposes a label 611 in the upper left corner of the superimposed image 602 to display a character string 612 indicating the area value of the affected area as information indicating the size of the affected area. Note that the background color and the character string color of the label 611 do not have to be limited to the illustrated example as long as they are easy to see. The arithmetic device 311 may also set a transparency amount and perform alpha blending so that the image of the part where the label is superimposed can be confirmed.

[0049] Furthermore, the arithmetic unit 311 superimposes an index 613 indicating the estimated area of ​​the affected area extracted in S442 on the superimposed image 602. That is, the arithmetic unit 311 generates data for superimposing and displaying the index 613 indicating the estimated area and the image data on which the image 601 is based by alpha blending at the position where the estimated area exists. This display allows the user to confirm whether the estimated area on which the area of ​​the affected area is based is appropriate. It is desirable that the color of the index 613 indicating the estimated area does not overlap with the color of the subject. The transmittance of the alpha blend is set to a range in which the estimated area can be recognized and the original affected area 102 can also be confirmed. Note that if the index 613 indicating the estimated area of ​​the affected area is superimposed and displayed, the user can confirm whether the estimated area is appropriate even without the display of the label 611, so S443 may be omitted.

[0050] Next, in S445, the communication device 313 of the image processing device 300 transmits information indicating the extraction result of the affected area extracted as described above and information indicating the size of the affected area to the imaging device 200 as data to be displayed on the display device 223 of the imaging device 200. That is, in this embodiment, the communication device 313 transmits the image data generated in S444, including the information indicating the size of the affected area, to the imaging device 200 via wireless communication.

[0051] Now, the description returns to the process performed by the imaging device 200. In S407, if image data including information indicating the size of the affected area has been newly generated by the image processing device 300, the communication device 219 of the imaging device 200 receives this data. In S408, if the system control circuit 220 has received image data including information indicating the size of the affected area in S407, the process proceeds to S409, and if not, the process proceeds to S410.

[0052] When the process proceeds to S409, the display device 223 displays an image based on the image data including the information indicating the size of the affected area received in S407 for a predetermined period of time. Here, the display device 223 displays the superimposed image 602 shown in FIG. 6(b). In this way, by superimposing information showing the extraction result of the affected area on the live view image, the user can confirm whether the area and estimated area of ​​the affected area are appropriate before taking the actual photograph. Note that, in the present embodiment, an example has been described in which both the indicator 613 showing the estimated area of ​​the affected area and the information on the size of the affected area are displayed, but only one of them may be displayed.

[0053] In S410, the system control circuit 220 detects whether or not a release button included in the operation unit 224 has been pressed. If the release button has not been pressed, the processing of the imaging apparatus returns to S404, and if the release button has been pressed, the processing of the imaging apparatus proceeds to S411. In S411, the distance measuring system 216 obtains distance information of the subject in the same manner as in S403, and the AF control circuit 218 performs AF processing to control the driving of the lens group 212 so as to focus on the subject. If the affected area can be extracted from the live view image, the distance measuring system 216 may obtain distance information of the subject at the position where the affected area exists.

[0054] Next, in S412, the imaging device 200 captures a still image. In S413, the image processing circuit 217 performs development and compression processing on the image data generated by the imaging in S412 to generate image data in, for example, the JPEG standard. Furthermore, the image processing circuit 217 performs resizing processing on the compressed image data to reduce the size of the image data.

[0055] Here, the size of the image data resized in S413 is the same as or larger than the image data resized in S404. This is to prioritize accuracy when measuring the affected area. In this embodiment, the image data is resized to 1440 pixels x 1080 pixels and 4-bit RGB color, so that the size of the image data is approximately 4.45 megabytes. Note that the resized size is not limited to this. Alternatively, the image processing circuit 217 may proceed to the next process of S414 using the image data of the JPEG standard that has been captured and generated without resizing.

[0056] When the process proceeds to S414, the communication device 219 acquires the image data generated in S413 and resized (or not resized), and the distance information calculated in S411. Furthermore, the communication device 219 acquires information on the zoom magnification and information on the size (number of pixels) of the resized image data. If the imaging unit 211 has only a fixed focal length lens and does not have a zoom function, the information on the zoom magnification is not necessary. Furthermore, if the image processing device 300 has information on the size of the image data in advance, the size information of the image data is not necessary. Next, in S415, the communication device 219 transmits the image data acquired in S414 and one or more pieces of information including the distance information to the image processing device 300 by wireless communication.

[0057] Here, the processing by the image processing device 300 will be explained again. In S451, the communication device 313 of the image processing device 300 receives the image data output from the communication device 219 of the imaging device 200 and one or more pieces of information including distance information. In S452, the auxiliary calculation unit 317 of the image processing device 300 extracts, from the image data received in S441, the affected area 102 of the subject 101. Details of the process are the same as in S442, and therefore will not be described again.

[0058] In S453, the arithmetic unit 311 calculates the area of ​​the affected area as an example of information indicating the size of the affected area extracted by the auxiliary arithmetic unit 317. The details of the process are the same as those of S443, and therefore will not be described. In S454, the computing device 311 performs image analysis to evaluate the state of the pressure sore in the affected area. The image analysis method will be described in detail later.

[0059] Next, in S455, the calculation device 311 generates image data for the image data from which the affected area was extracted, in which information indicating the extraction result of the affected area and information indicating the size of the affected area are superimposed, i.e., data to be displayed on the display device 223 of the imaging device 200. 7 is a diagram for explaining a method of superimposing information showing the extraction result of the affected area and information showing the size of the affected area including the major axis and minor axis of the affected area on image data. Since it is considered that a plurality of pieces of information showing the size of the affected area can be obtained, the superimposed image 701 in FIG. 7(a), the superimposed image 702 in FIG. 7(b), and the superimposed image 703 in FIG. 7(c) will be explained separately. These images are sent from the image processing device 300 to the imaging device 200 and displayed on the display device 223.

[0060] The superimposed image 701 in FIG. 7(a) shows an example in which the minimum bounding rectangle is used as a method for calculating the major axis and minor axis of the affected area. The arithmetic device 311 generates an image in which a label 611 for displaying a character string 612 of the area value of the affected area is superimposed on the upper left corner of the superimposed image 701 as information indicating the size of the affected area, as in the example of FIG. 6. Furthermore, the arithmetic device 311 generates an image in which a label 712 for displaying the major axis and minor axis calculated based on the minimum bounding rectangle is superimposed on the upper right corner of the superimposed image 701 as information indicating the size of the affected area. A character string 713 indicates the length of the major axis (unit: cm), and a character string 714 indicates the length of the minor axis (unit: cm). Furthermore, the arithmetic device 311 superimposes a rectangular frame 715 representing the minimum bounding rectangle on the superimposed image 701 so as to surround the affected area 102. This causes the rectangular frame 715 to be displayed superimposed with the lengths of the major axis and minor axis, allowing the user to confirm which point in the image is being measured.

[0061] The arithmetic unit 311 also superimposes a scale bar 716 on the lower right corner of the superimposed image 701. The scale bar 716 is for measuring the size of the affected area, and the size of the scale bar for the image data is changed according to the distance information. Specifically, the scale bar 716 is a bar with graduations up to 5 cm in 1 cm increments based on the length on the focal plane corresponding to one pixel on the image obtained in S453, and corresponds to the size on the focal plane of the imaging device, i.e., on the subject. The user can roughly grasp the size of the subject or the affected area by referring to the scale bar.

[0062] Furthermore, the arithmetic unit 311 superimposes a label 717 representing the evaluation value of the above-mentioned DESIGN-R size (Size) on the lower left corner of the superimposed image 701. In the evaluation value of DESIGN-R Size, the major axis and minor axis (maximum axis perpendicular to the major axis) of the skin damage range are measured (unit: cm), and classified into the above-mentioned seven stages based on the numerical value obtained by multiplying the major axis and minor axis. In this embodiment, the evaluation value obtained by replacing the major axis and minor axis with the values ​​output by the respective calculation methods is superimposed.

[0063] The superimposed image 702 in FIG. 7(b) uses the maximum Feret's diameter as the major axis and the minimum Feret's diameter as the minor axis. The arithmetic device 311 superimposes a label 722 representing a character string 723 of the major axis and a character string 724 of the minor axis in the upper right corner of the superimposed image 702. Furthermore, the arithmetic device 311 superimposes an auxiliary line 725 corresponding to the measurement position of the maximum Feret's diameter and an auxiliary line 726 corresponding to the minimum Feret's diameter on the affected area 102 of the superimposed image 702. By superimposing the auxiliary line together with the lengths of the major axis and the minor axis, the user can confirm which part in the image is being measured.

[0064] 7(c), the long diameter of the superimposed image 703 is the same as that of the superimposed image 702, but the short diameter is measured as a length measured in a direction perpendicular to the axis of the maximum Feret diameter, not the minimum Feret diameter. For this reason, a label 732 for displaying a character string 723 of the long diameter and a character string 734 of the short diameter is superimposed in the upper right corner of the superimposed image 702. Furthermore, in the superimposed image 703, an auxiliary line 725 corresponding to the measurement position of the maximum Feret diameter and an auxiliary line 736 corresponding to the length measured in a direction perpendicular to the axis of the maximum Feret diameter are superimposed on the affected area 102.

[0065] The various information superimposed on the image data shown in each of Figures 7(a) to 7(c) may be any one or a combination of multiple pieces, and the user may be able to select the information to be displayed. The superimposed images shown in Figures 6 and 7 are just examples, and the display form, display position, font type, font size, font color, or positional relationship of the affected area 102 and the information indicating the size of the affected area can be changed according to various conditions.

[0066] Next, in S456, the communication device 313 of the image processing device 300 outputs information indicating the results of image analysis of the state of the bedsore in the affected area to the imaging device 200. In this embodiment, the communication device 313 outputs the image data including each piece of information indicating the size of the affected area, etc., generated in S455 to the imaging device 200 via wireless communication.

[0067] Here, the processing by the imaging device 200 will be described again. In S416, the communication device 219 of the imaging device 200 receives the image data generated by the image processing device 300, which includes various information indicating the size of the affected area, etc. In S417, the display device 223 displays the image data including the information indicating the size of the affected area, etc., received in S416 for a predetermined time. In this embodiment, the display device 223 displays any one of the superimposed images 701 to 703 shown in Figures 7(a) to 7(c) for a predetermined time, and after the predetermined time has elapsed, the processing of the imaging device 200 proceeds to S418.

[0068] In S418, the system control circuit 220 determines whether there is any information on the affected area that has not been input. Here, the information on the affected area refers to information indicating the location of the affected area and the evaluation values ​​of each evaluation item in DESIGN-R described above. The evaluation value for the evaluation item related to size is automatically input based on the information indicating the size received in S416.

[0069] Next, if it is determined in S418 that there is affected area information for which no value has been input, the system control circuit 220 proceeds to S419, and if all affected area information has been input, the system control circuit 220 returns to S402 and resumes live view. In S419, the system control circuit 220 causes the display device 223 to display a user interface that prompts the user to input information about the affected area. Thereafter, when the user inputs affected area information via the operation unit 224 in S420, the processing of the imaging device 200 returns to S418.

[0070] Next, a method for image-analyzing the wound surface condition of an affected area by the image processing device 300 according to this embodiment will be described. As mentioned above, FIGS. 6 and 7 are diagrams for explaining a method for analyzing the size of the affected area. DESIGN-R, an assessment index for bedsores, stipulates that the size of a bedsore should be measured as the product of the major axis and the minor axis. The image processing device 300 of this embodiment performs analysis of the major axis and the minor axis, thereby ensuring compatibility with data measured using the previous DESIGN-R.

[0071] Here, since DESIGN-R does not have a strict definition, several mathematical methods for calculating the major axis and minor axis are possible. As an example of a method for calculating the major axis and minor axis, first, the arithmetic device 311 calculates a rectangle with a minimum area (minimum bounding rectangle) among rectangles circumscribing the affected area 102. Then, the arithmetic device 311 calculates the lengths of the long and short sides of this rectangle, and calculates the length of the long side as the major axis and the length of the short side as the minor axis. Then, the arithmetic device 311 calculates the area of ​​this rectangle based on the length on the focal plane corresponding to one pixel on the image obtained in S453.

[0072] As another example of a method for calculating the major axis and the minor axis, the maximum Feret diameter, which is the maximum caliper length, may be selected as the major axis, and the minimum Feret diameter may be selected as the minor axis. Alternatively, the maximum Feret diameter, which is the maximum caliper length, may be selected as the major axis, and the length measured in a direction perpendicular to the axis of the maximum Feret diameter may be selected as the minor axis. The method for calculating the major axis and the minor axis can be selected based on compatibility with the measurement results when the measurement was performed previously.

[0073] Note that calculation of the length of the major axis and the minor axis and the rectangular area of ​​the affected area is not performed on the image data received in S441. Since the purpose of the live view is to enable the user to check the extraction result of the affected area, the processing time is reduced by omitting the image analysis processing step of S454. If the purpose is to obtain information on the actual area of ​​the pressure ulcer without evaluating the size based on DESIGN-R, the process of S454 can be omitted. In that case, the subsequent processes will be performed assuming that there is no size information as an evaluation item in DESIGN-R.

[0074] Next, a method for measuring the depth of the affected area will be described. FIG. 8 is a flowchart of a process for acquiring depth information of an affected area in the image processing device 300. As shown in FIG. 9(a) and 9(b) are diagrams for explaining a method for setting a measurement standard for acquiring information showing an extraction result of an affected area and depth information of the affected area. FIG. 9(b) is a diagram showing a cross section A-A' of FIG. 9(a). First, in S901, the auxiliary calculation unit 317 performs processing for extracting the non-affected area 921 of the subject 101 from the image data. 9(a), the non-affected area region 921 is a region that is set as a substantially annular region excluding the affected area region 102, that is, a predetermined region that is in the vicinity of the affected area region 102 but does not include the affected area region 102. Note that, in this embodiment, the non-affected area region 921 is described as having an annular shape as an example, but is not limited thereto, and may have a rectangular shape as long as it surrounds the affected area region 102.

[0075] Next, the process proceeds to S902, where the auxiliary calculation device 317 sets a reference point 922 within the non-affected area region 921. As shown in Fig. 9(a), at least three reference points 922 are set within the non-affected area region 921. Then, the calculation device 311 derives a plane including these three reference points 922a, 922b, and 922c. The plane including the reference points 922a, 922b, and 922c can be derived from an equation of a plane passing through the reference points 922a, 922b, and 922c using distance information L1, L2, and L3 corresponding to the reference points 922a, 922b, and 922c shown in Fig. 9(b) as parameters.

[0076] Next, the process proceeds to S903, where the calculation device 311 sets a plane including the reference points 922a, 922b, and 922c as a reference plane 923 for the depth information of the affected area 102, and thereafter uses this plane as a reference for calculating the depth information of the affected area 102. Note that these reference points, reference plane, depth information, position information, and the like are appropriately stored in the storage device 312. Next, the process proceeds to S904, where the calculation device 311 calculates depth information d1 of the affected area 102 from the reference plane 923. The depth information d1 is derived by taking the reference plane 923 as zero and calculating the difference between distance information to the reference plane 923 and distance information to the wound surface in the affected area 102.

[0077] Next, the process proceeds to S905, where the arithmetic device 311 determines whether or not the reference surface 923 is inclined with respect to the imaging surface 924 of the image sensor 214. As described above, in DESIGN-R, the depth of a bedsore is evaluated based on the degree of invasion into body tissues such as the epidermis, dermis, subcutaneous tissue, and bone, with the wound edge as the reference. Therefore, it is desirable to set the depth value to a value in the same direction as the layering direction of the body tissue, which is equal to the direction of the normal vector v2 of the reference plane 923 shown in Figure 9(b). The presence or absence of inclination of the normal vector v2 can be determined by whether or not it is inclined with respect to the normal vector v1 of the imaging plane 924, that is, whether or not the reference plane 923 is directly facing the imaging plane 924.

[0078] Therefore, in S905, the arithmetic device 311 determines whether or not the reference surface 923 is inclined with respect to the imaging surface 924 of the image sensor 214. If the reference surface 923 is inclined with respect to the imaging surface 942, the arithmetic device 311 proceeds to S906 and corrects the depth information d1 to depth information d1' facing the imaging surface 924. The correction process can be calculated by deriving the inclination angle θ of the normal vector v2 of the reference surface 923 with respect to the normal vector v1 of the imaging surface 924, and multiplying the depth information d1 by the trigonometric function cos θ.

[0079] Next, proceeding to S907, the calculation device 311 calculates depth information d1' for the entire affected area based on a distance map showing the distribution of distance information in the affected area 102 acquired by the distance measurement system 216, and generates a depth map of the affected area 102. By processing the flowchart described above, the image processing device 300 can obtain a depth map of the affected area 102, and by using this depth map, the depth of the affected area 102 can be quantitatively evaluated regardless of the definition of DESIGN-R.

[0080] In this embodiment, the auxiliary calculation device 317 sets the reference point 922 at an arbitrary point in the non-affected area 921, but a physical feature such as a mole may also be set as the reference point. If an arbitrary point within the non-affected area region 921 is set as the reference point, it is difficult to set the same point for each image capture if the patient's posture during image capture is different, and the calculation result of the tilt amount θ may vary, which may reduce the calculation accuracy of the depth information. In contrast, a physical feature such as a mole is located at the same position on the subject 101 regardless of the patient's posture, so the same point can be set as the reference point for each image capture, and the decrease in the calculation accuracy of the depth information can be eliminated. Therefore, if a physical feature is located near the affected area region 102, the auxiliary calculation device 317 sets that physical feature as the reference point 922.

[0081] Alternatively, the arithmetic unit 311 may display the reference point 922 set by the auxiliary arithmetic unit 317 in real time via the display unit 223. In this case, the photographer may mark the subject 101 with a water-based felt pen or the like, as if tracing the marking, while watching the display on the display unit 223. Since the marking remains in the same position regardless of the patient's posture as long as it does not disappear, it can be used as the same reference point for each imaging, like a physical feature.

[0082] In this embodiment, a method for analyzing the typical states of bedsores shown in Figs. 9(a) and 9(b) has been described, but actual bedsores can be in complex and diverse states. For example, in the case of a bedsore that has developed in a part with curvature, such as the heel shown in the example of Fig. 11(a) described later, the wound surface protrudes beyond the edge of the affected area 102, as shown in Fig. 11(b) described later. In this case, in the process of the flowchart shown in Fig. 8, it is possible that the edge of the affected area 102 is determined to be shallower than the wound surface within the affected area.

[0083] However, as mentioned above, in DESIGN-R, the depth of a pressure ulcer is evaluated based on the wound margin and the degree of invasion into body tissues such as the epidermis, dermis, subcutaneous tissue, and bone. Therefore, a determination that the wound margin is shallower than the wound surface is an incorrect determination when compared with the definition of DESIGN-R. Therefore, when generating a depth map of a bedsore that has developed in a part of the body with curvature such as the heel, it is necessary to devise a method for calculating the depth information.

[0084] Hereinafter, a method for measuring the depth of a bedsore that has developed in a part with curvature will be described, taking the heel as an example of the affected area. FIG. 10 is a flowchart of a process performed by the image processing device 300 to obtain depth information of an affected area of ​​a bedsore that has developed in a part of the body with curvature, such as the heel. 11(a) and 11(b) are diagrams for explaining a method for setting a measurement standard for acquiring information showing an extraction result of an affected area and depth information of an affected area. FIG. 11(b) is a diagram showing a cross section B-B' of FIG. 11(a).

[0085] The process of the flowchart shown in Fig. 10 is started when a part having a curve is selected by the user in S420 and the information is received by the image processing device 300. A method for inputting the part of the affected area by the user will be described in detail in Fig. 11. In this embodiment, the selection of the HEEL as the part of the affected area corresponds to the selection of a part having a curve. From S911 to S912, the same process as from S901 to S902 in Fig. 8 described above is executed, and therefore the description thereof will be omitted.

[0086] In S913, the arithmetic device 311 calculates a virtual sphere 933 for the affected area that approximates a normal skin surface shape before the onset of a bedsore as shown in Fig. 11(a). The virtual sphere 933 can be derived by deriving an equation of a sphere that passes through four reference points 932a, 932b, 932c, and 932d in the non-affected area 931. That is, the virtual sphere 933 can be derived from an equation of a sphere that passes through the reference points 932a, 932b, 932c, and 932d, using the distance information L1', L2', L3', and L4' corresponding to the reference points 932a, 932b, 932c, and 932d shown in Fig. 11(b) as parameters. The arithmetic device 311 then uses this virtual spherical surface 933 as a reference surface for the depth of the affected area, and thereafter uses this as a reference when calculating depth information of the affected area.

[0087] Next, in S914, the arithmetic device 311 calculates depth information d2 of the affected area from the virtual sphere 933, and then proceeds to S915. The depth information d2 is derived by taking the virtual sphere 933 as zero and calculating the difference between the distance information to the virtual sphere 933 and the distance information to the wound surface of the affected area. Next, in S915, the calculation device 311 corrects the depth information d2 of the affected area derived in S914 to depth information d2′ directly facing the virtual spherical surface 933. As described above, in DESIGN-R, the depth of a bedsore is evaluated based on the depth of invasion into body tissues such as the epidermis, dermis, subcutaneous tissue, and bone, with the wound edge as the reference point. Therefore, it is desirable to set the depth value to a value in the same direction as the layering direction of the body tissue, and the layering direction of the body tissue is equal to the direction of the normal vector v2' of the virtual spherical surface 933 shown in FIG. 11(b).

[0088] Therefore, the calculation device 311 derives depth information d2' by correcting the affected area depth information d2 derived in S914 in the direction of the normal vector v2'. This depth information correction calculation is a calculation to derive the inclination angle θ of the normal vector v2' of the virtual spherical surface 933 with respect to the normal vector v1' of the imaging surface 924, and multiply the depth information d2 by the trigonometric function cosθ. Next, the process proceeds to S916, where the calculation device 311 calculates depth information d2' for the entire affected area based on a distance map showing the distribution of distance information in the affected area acquired by the distance measurement system 216, and generates a depth map of the affected area. By performing the process of the flowchart described above, the image processing device 300 can obtain depth information even for a bedsore that has developed in a part with curvature such as the heel.

[0089] Next, a method for analyzing the five states of an affected area, namely, granulation tissue, eschar, slough, exudate, and exposed bone, will be described. Bone exposure is the most severe condition in the DESIGN-R depth assessment. FIG. 12 is a flowchart of a process for analyzing five states of an affected area, namely, granulation tissue, eschar, slough, exudate, and exposed bone, in the image processing device 300. 13(a) and 13(b) are schematic diagrams of granulation tissue, eschar, slough, exudate, and exposed bone that have developed in the affected area 102. Fig. 13(b) is a diagram showing cross section CC' of Fig. 13(a).

[0090] In the flowchart of Fig. 12, first, in S1001, the calculation device 311 determines whether or not there is a black area in the affected area 102. If there is a black area, the area 1021 matches the characteristics of an eschar, and so the calculation device 311 proceeds to S1002 and determines that the area 1021 is an eschar.

[0091] Next, the process proceeds to S1003, where the calculation device 311 determines whether or not there is an unconfirmed region, i.e., an undetermined region, within the affected area 102. If there is an undetermined region, the calculation device 311 returns to S1001 and repeats the analysis of the undetermined region to determine the state of the entire affected area. On the other hand, if there is no undetermined region, the calculation device 311 determines that the analysis of the entire affected area has been completed, and the image processing device 300 ends the process of the flowchart in FIG.

[0092] Returning now to S1001, the flow of processing when it is determined that there is no black area will be described. If there is no black region, the arithmetic device 311 proceeds to S1004, where it determines whether or not there is a yellow region in the affected area 102. If there is a yellow region, the arithmetic device 311 proceeds to S1005. Since the yellow region 1022 in the affected area 102 matches the characteristics of exudate, the arithmetic device 311 determines in S1005 that the region 1022 is exudate. After S1005, the arithmetic device 311 proceeds to S1003. On the other hand, if the calculation device 311 determines in S1004 that there is no yellow region, the process proceeds to S1006, where it determines whether or not there is a white region in the affected area 102. If it determines that there is a white region, the calculation device 311 proceeds to S1007, where it acquires a depth map of the affected area generated by executing the flowchart of FIG. 8, and then proceeds to S1008.

[0093] Here, bones and sloughs both have the characteristic of being white in color, and when the bedsore penetrates deeply and bones are exposed within the affected area 102, it is not possible to distinguish between the two based on color information alone. For this reason, the image processing device 300 of this embodiment analyzes the wound condition of the affected area 102 using both color information and distance information. In S1008, the arithmetic device 311 determines whether or not the region 1023 from which the white color is extracted is the deepest region in the affected area 102, that is, the wound bed.

[0094] In the example of FIG. 13(b), the depth information d3 corresponding to the region 1023 is the deepest region within the affected region 102. In this case, the area 1023 coincides with the above-mentioned state in which the bedsore has deepened and the bone is exposed, and the calculation device 311 proceeds to S1009 and determines that the bone is exposed in the area 1023. On the other hand, since the region 1024, which is not a wound bed and is white, matches the characteristics of a slough, the calculation device 311 proceeds to S1010 and determines that the region 1024 is a slough. After S1009 or S1010, the calculation device 311 proceeds to S1003.

[0095] Furthermore, in S1006, if there is no white area, the arithmetic device 311 proceeds to S1011. Here, among the states analyzed by the image processing device 300 of this embodiment, eschar, slough, exudate, and exposed bone have been determined in the regions 1021 to 1024, respectively, except for pockets, which will be described later. Then, when pockets, which will be described later, are excluded, the remaining state that has not been determined is granulation tissue.

[0096] Therefore, the arithmetic device 311 presumes that the region 1025, which is an undetermined region within the affected area 102, is granulation tissue, and determines in S1011 that the region 1025 is granulation tissue. After S1011, the arithmetic device 311 proceeds to S1003. That is, the calculation device 311 of this embodiment judges eschar or exudate as a first state for black or yellow areas based on color information, and after determining this first state, judges bone, slough, or granulation tissue as a second state based on the color information and distance information. By the processing of the above flowchart, the image processing device 300 of this embodiment can determine the wound condition in the affected area.

[0097] As described above, in steps S1001 to S1005, the calculation device 311 determines the state of the bedsore only from the color information, without using distance information for the affected area. Here, the phase difference type distance measuring sensor mounted on a single-lens camera exemplified as the distance measuring system 216 of this embodiment, or a system using a TOF sensor, are both distance measuring devices that use reflected light from a subject. Therefore, the reflectance of the subject may affect the accuracy of distance measurement, resulting in an erroneous determination. Therefore, the image processing device 300 of this embodiment is configured to first analyze eschar and exudate, which can be determined only from color information. Then, in steps S1006 to S1010, the image processing device 300 analyzes the wound condition using distance information in addition to color information for areas excluding areas 1021 and 1022 that have been determined to be eschar and exudate, thereby improving the accuracy of the determination.

[0098] Next, a process of determining the presence or absence of a pocket and a method of measuring the depth of a pocket by the image processing device 300 will be described. FIG. 14 is a flowchart showing the process from determining the presence or absence of a pocket to measuring the pocket depth. First, in S1201, the arithmetic device 311 uses the distance information acquired in the processing of the flowchart in Fig. 8 to determine whether or not a pocket exists in the affected area 102. If no pocket exists in the affected area, the arithmetic device 311 ends the processing of the flowchart for measuring the pocket depth.

[0099] FIG. 15 shows an example where a pocket exists in the affected area. FIG. 15(a) is a diagram showing the position of cross section D-D' in the affected area 102. Cross section D-D' is set at an arbitrary position by the calculation device 311. The calculation device 311 performs a pocket presence / absence determination for the entire affected area by repeatedly translating cross section D-D' at an arbitrary step and determining the presence / absence of a pocket. In FIG. 15(a), cross section D-D' is shown passing through the center of the affected area 102, but it does not necessarily have to pass through the center of the affected area 102. The setting of cross section D-D' will be described in detail later.

[0100] Fig. 15(b) is a diagram showing the shape of the affected area 102 in cross section D-D' of Fig. 15(a). Here, the positional relationship with the imaging device 200 is shown in a simplified manner. Fig. 15(b) shows an example in which a pocket 1100 exists on the right side. Although an example in which a pocket 1100 exists at one end of the affected area 102 is shown, there may be multiple pockets 1100.

[0101] Fig. 15(c) is a diagram showing the relationship between cross section D-D' of affected area 102 shown in Fig. 15(b) and depth information. The horizontal axis indicates the position on cross section D-D' of affected area 102, and the vertical axis indicates the depth information of affected area 102. The depth information of affected area 102 is written so that reference plane 923 is set to zero, and becomes larger values ​​toward the center of affected area 102. 15(c), as an example, a slope 1101 without a pocket is shown on the left side, and a slope 1102 with a pocket is shown on the right side. The slope 1101 without a pocket is characterized in that the slope from the reference plane 923 to the wound base 1023 is small, while the slope 1102 with a pocket is characterized in that the slope from the reference plane 923 to the wound base 1023 is large.

[0102] Next, the method of calculating the above-mentioned inclination will be described. The arithmetic unit 311 calculates the inclination based on the position of the affected area 102 on the cross section D-D' and the measured depth information. The calculated inclination may be the root mean square value of an arbitrary range or the maximum value. Here, the arithmetic device 311 determines whether or not there is a pocket by calculating whether the calculated inclination is equal to or greater than a threshold value. The inclination threshold is an absolute value, and is set as an initial setting value for the imaging device 200. The inclination threshold can be arbitrarily set by the user, and once set, there is no need to set it again. In addition, since the affected area may have a large depth, the inclination threshold can also be made variable with respect to the depth information. For example, when the affected area has a large depth, the inclination 1101 indicating no pocket will be a large value, and it is therefore necessary to increase the inclination threshold and determine that no pocket exists.

[0103] The result of the determination as to whether or not there is a pocket is displayed by the display device 223. The presence or absence of a pocket may be determined automatically as described above, but the calculated inclination may be displayed by the display device 223 so that the user can determine it at his / her own discretion.

[0104] In this embodiment, deep learning is performed as a method for determining the presence or absence of a pocket. That is, a high-performance computer for machine learning (not shown) is made to learn a neural network model using the judgment of the presence or absence of a pocket for the calculated inclination as teacher data, and a learned model is generated. The auxiliary calculation device 317 receives the learned model from the high-performance computer, and estimates the threshold value of the inclination required for the presence or absence of a pocket from the image data based on this learned model. In addition, other methods may be used in addition to deep learning. Furthermore, inside the auxiliary calculation device 317, learning of a neural network model using the judgment of the presence or absence of a pocket for the calculated inclination as teacher data may be performed. Note that the judgment of the presence or absence of a pocket may be performed based on the intensity of the received light signal rather than the depth information, that is, based on the inclination of the received light signal intensity at an arbitrary cross section.

[0105] Next, the setting of the cross section DD' will be described. FIG. 16 is a diagram for explaining a method for extracting (hereinafter referred to as reading) the calculated values ​​of the cross section DD'. 16(a) shows a method of reading out 360° radially from the center of the affected area 102. The center 1110 of the affected area 102 is calculated by the calculation device 311 of the image processing device 300. A cross section passing through the center 1110 of the affected area 102 is set as a central cross section 1111. The central cross section 1111 is rotated 360° around the center 1110 at an arbitrarily set read interval, so that the entire affected area is read out. The center 1110 may be set by using a plurality of reference points 922 for setting a reference plane 923.

[0106] 16(b) shows a method of reading out the affected area 102 in the horizontal direction and in the vertical direction perpendicular to the horizontal direction by 90°. In the following description, a horizontal section on the screen of the display device 223 is referred to as a horizontal section 1112, and a vertical section on the screen of the display device 223 is referred to as a vertical section 1113. The horizontal section 1112 and the vertical section 1113 are translated at an arbitrarily set read interval to read out the entire affected area. Here, an example is given in which the horizontal direction and the vertical direction are set on the screen of the display device 223, but the present invention is not limited to this example as long as the sections in the two directions set are shifted by 90° from each other.

[0107] By reading out the entire affected area as described above, it becomes possible to identify the direction (position) in which the pocket exists. Furthermore, if there are multiple pockets, the image processing device 300 causes the display device 223 to display the number and positions of the pockets.

[0108] FIG. 17 is a diagram for explaining a method of calculating the inclination while omitting the central area of ​​the affected area 102. In FIG. The affected area 102 may have a complex shape, such as an uneven shape or a shape with sharp angles, including the wound bed 1023. When determining the presence or absence of a pocket, the central area of ​​the affected area 102 is omitted when calculating the inclination, thereby preventing erroneous detection.

[0109] 17(a) is a diagram showing an example in which the central area of ​​the affected area 102 is omitted. The omitted area is shown as a central area 1120, and the cross section D-D' is set to pass through the central area 1120. The calculation device 311 calculates an area obtained by subtracting the central area 1120 from the entire affected area 102, and calculates the slope of this area. Here, the central area 1120 has a reduced shape while maintaining a similar relationship to the affected area 102. For example, the larger the reduction rate, the more likely the false detection rate is to increase, so the central area 1120 is set to 2 / 3 the size of the affected area 102.

[0110] FIG. 17(b) is a diagram showing the shape of the affected area 102 in cross section DD' of FIG. 17(a). FIG. 17(c) is a diagram showing the relationship between the cross section DD' of the affected area 102 shown in FIG. 17(b) and depth information. In this way, by omitting the central cutout range 1120, it is possible to extract only the inclined portion from the reference plane 923 to the wound bottom 1023. Therefore, even if the central cutout range 1120 has an uneven shape and has an inclination that is judged to indicate the presence of a pocket, it is possible to make a judgment without erroneous detection.

[0111] Next, a method for measuring pocket depth will be described. FIG. 18 is a diagram showing an example of a method for measuring the depth of the pocket 1100 in the cross section DD' of FIG. 15(a). 14, the arithmetic device 311 requests the user to tilt the imaging device 200 from a position 1212 facing the reference plane 922 to a position 1213 where the deepest part of the pocket 1100 in the cross section D-D' can be measured. The request to the user is realized, for example, by the arithmetic device 311 generating display information requesting the user to perform a tilt operation on the imaging device 200 in S1202 and displaying the display information on the display device 223 of the imaging device 200. Then, when the user tilts the imaging device 200 as requested, the imaging device 200 performs distance measurement by the distance measurement system 216 in S1203, and the arithmetic device 311 thereby acquires distance information of the deepest part 1222 of the pocket.

[0112] At this time, the calculation device 311 may predict the position 1213 by performing calculations using the positional relationship and distance information between the affected area 102 and the wound edge 1211, and may limit the tilt direction and tilt angle of the imaging device 200 based on the predicted position. An example of a method for predicting the position 1213 based on the calculation by the calculation device 311 and acquiring distance information of the deepest part 1222 of the pocket will be described below.

[0113] First, the arithmetic device 311 calculates the position of the midpoint of the line connecting the wound edge 1211 and the wound bottom 1220 near the wound edge, and sets the position of this midpoint as the tilt center 1218 of the imaging device 200. Any position on the line connecting the tilt center 1218 and the wound edge 1211 is equal to the position 1212 directly facing the reference plane 922. The arithmetic device 311 also calculates the position of the wound margin 1219, which is different from the wound margin 1211, on the cross section D-D' of the affected area 102, and sets any position on the line connecting the tilt center 1218 and the wound margin 1219 as the measurement limit position. Furthermore, the arithmetic device 311 predicts that the position 1213 is included within the angle 1215 at which the imaging device 200 is tilted from the position 1212 to the measurement limit position, and limits the tilt angle of the imaging device 200 when measuring the depth of the pocket to the angle 1215. Then, the imaging device 200 is tilted within the angle 1215, and the distance measuring system 216 performs distance measurement multiple times, thereby acquiring distance information within the pocket 1100. Furthermore, the arithmetic device 311 calculates the point farthest from the tilt center 1218 within the cross section D-D' from the distance information within the pocket 1100, and sets the point as the deepest part 1222 of the pocket.

[0114] At this time, the rotation direction of the imaging device 200 does not have to be limited to one axial direction. FIG. 19 is a diagram showing an example of a method for measuring the depth of the pocket 1100 by rotating the imaging device 200 about an axis of a straight line that passes through the wound edge 1211 and is perpendicular to the reference plane. At this time, the initial position of the imaging device 200 is set to position 1213. Then, the arithmetic device 311 requests the user via the imaging device 200 to rotate the imaging device from position 1213 around an axis that passes through the wound edge 1211 and is perpendicular to the reference plane, to position 1224 where the deepest part of the pocket 1100 can be measured. After the user rotates the imaging device 200 as requested and the distance measurement is performed by the distance measurement system 216, the arithmetic device 311 obtains distance information of the deepest part of the pocket 1225 from the distance measurement information.

[0115] At this time, the calculation device 311 may predict the position 1224 by calculation using the positional relationship and distance information between the affected area 102 and the wound edge 1211, and limit the rotation direction and rotation angle of the imaging device 200. An example of a method in which the calculation device 311 predicts the position 1224 by calculation and obtains distance information of the deepest part of the pocket 1225 will be described below.

[0116] First, the arithmetic device 311 calculates the smallest rectangular shape 1226 that can surround the affected area 102, includes a straight line horizontal to the cross section D-D', and is created within the reference plane 922. The arithmetic device 311 also extracts straight lines that connect the wound edge 1211 and the vertices of the rectangular shape 1226 and pass through the projection plane of the affected area 102 relative to the reference plane 922. Furthermore, the arithmetic device 311 predicts that the position 1224 exists within the range of the maximum angle 1223 among the angles that include the position 1213 and are created by selecting two straight lines, for example, the straight lines 1227 and 1228, from among the extracted straight lines. Then, the arithmetic device 311 limits the rotation angle of the imaging device 200 when measuring the pocket depth to the angle 1223.

[0117] Next, when the imaging device 200 is rotated within the angle 1223 and the distance measurement system 216 performs distance measurement multiple times, the calculation device 311 acquires distance information within the pocket 1100 from the multiple distance measurement information. Then, the calculation device 311 calculates the point within the reference plane 922 that is farthest from the wound edge 1211 from the acquired distance information within the pocket 1100, and sets this point as the deepest part 1225 of the pocket.

[0118] Furthermore, when performing distance measurement using the distance measuring system 216 while tilting or rotating the camera, a user assistance function that enables the user to recognize the optimal tilt or rotation direction can be realized by the imaging device 200 and the arithmetic device 311. The user assistance function may be provided in the imaging device 200, or may be realized by the imaging device 200 in response to an instruction from the arithmetic device 311. Here, an example will be described in which the arithmetic device 311 causes the imaging device 200 to realize the user assistance function.

[0119] FIG. 20 is a diagram showing an example of a user assistance function. For example, when the user is to recognize that the position of the pocket 1100 is around the wound edge 1211, the arithmetic device 311 displays a rectangular pocket determination mark 1216 indicated by a dotted line on the display portion of the wound edge 1211 on the live view image of the imaging device 200. The arithmetic device 311 may also display a direction instruction 1217 on the live view image to instruct the user in the direction to tilt or rotate the imaging device 200 within the above-mentioned angle 1215 or angle 1223. At this time, the imaging device 200 may be equipped with an acceleration sensor and an angular velocity sensor to obtain tilt angle amount information and translation movement amount information of the imaging device 200. In this case, the image processing device 300 has a position and orientation acquisition function that acquires position and orientation information of the imaging device 200 based on acceleration information obtained by the acceleration sensor of the imaging device 200 and angular velocity information obtained by the angular velocity sensor. The calculation device 311 then calculates the amount of change in the position and orientation of the image capture device 200 based on the acceleration information and angular velocity information, and calculates tilt angle amount information and translational movement amount information based on the amount of change.

[0120] In S1204, the calculation device 311 determines whether the tilting and rotating of the imaging device 200 and the associated distance measurement as described above have been completed, and if not, the process returns to S1202, whereas if completed, the process proceeds to S1205. When the process proceeds to S1205, the calculation device 311 performs pocket depth calculation. That is, in S1205, the calculation device 311 calculates the projection distance of the wound edge 1211 and the deepest part of the pocket 1222 or the deepest part of the pocket 1225 relative to the reference plane 922 from the acquired distance information of the deepest part of the pocket 1222 or the deepest part of the pocket 1225. Then, the calculation device 311 sets the calculated projection distance as the pocket depth 1214. Thereafter, in S1205, the arithmetic device 311 ends the process of this flowchart upon confirming that the calculation of the pocket depth has been completed.

[0121] 21A shows an example of a display that prompts the user to input the affected area information from the affected area information. The display information may be generated by the arithmetic unit 311 of the image processing device 300 and displayed on the display device 223 of the imaging device 200, or may be generated by the system control circuit 220 of the imaging device 200 and displayed on the display device 223.

[0122] The display device 223 displays a region selection item 801 for specifying the region of the affected area, Head, Shoulder, Arm, Back, Waist, Hip, and Leg. At the bottom of the region selection item 801, there is an item for completing the input of the affected area information, and the user can select this item through the operation of the operation unit 224 to end the input work of the affected area information even if some of the affected area information has not been input. The user can also specify the region where the imaged affected area exists by operating the operation unit 224. The display device 223 displays the item selected by the user as an item surrounded by a frame 802. FIG. 21(a) shows the state where Hip is selected. Since one region indicated by the region selection item 801 may have two or more affected areas, it may be possible to select a plurality of items such as Hip1, Hip2, and Hip3.

[0123] Also, one part may be divided, for example, the leg may be further divided, so that a plurality of items such as the heel and the knee can be selected. Fig. 21(b) shows an example of the display of item 803 for confirming with the user whether the site where the affected area is present is acceptable after the site is selected in Fig. 21(a). When the user operates item 803 to confirm that the selected site is acceptable, the display device 223 displays the screen shown in Fig. 21(c).

[0124] FIG. 21(c) is a diagram showing an example of a display prompting input of evaluation values ​​for each evaluation item of DESIGN-R from among the affected area information. As shown in Fig. 21(c), an evaluation item selection section 804 is displayed on the left side of the screen. In Fig. 21(c), D represents the items of Depth, E represents Exudate, S represents Size, I represents Inflammation / Infection, and G represents Granulation. Furthermore, N represents Necrotic tissue, and P represents Pocket. Each of these items and information indicating whether each item has been input are displayed on the display device 223 together with an image of the affected area.

[0125] In Figure 21(c), the evaluation value "s9" is displayed for S (size), which has already been analyzed from the image, and "non" is displayed for the other evaluation items, indicating that they have not been confirmed. The S (size) item is displayed with a shaded area, indicating that this item has already been entered.

[0126] The user can specify an evaluation item by operating the operation unit 224, and the display device 223 displays the selected evaluation item (here, D (depth)) surrounded by a frame 805. At this time, the display device 223 also superimposes the evaluation value of the severity of the evaluation item selected on the left side of the screen on the lower part of the screen as a severity selection section 806. In Fig. 21(c), evaluation values ​​d0, d1, d2, D3, D4, D5, and DU indicating the severity of D (depth) are displayed. Furthermore, the user can select one of the evaluation values ​​by operating the operation unit 224. At this time, the display device 223 displays the selected evaluation value surrounded by a frame 807, and also displays an explanation 808 of the evaluation value (explanation of evaluation item Depth, severity d2: damage down to the dermis). Note that the evaluation value may be input by the user inputting a character string.

[0127] In this case, the severity may be further subdivided and displayed by adding an auxiliary evaluation value in addition to the above-mentioned evaluation value. As the auxiliary evaluation value of the depth, distance information from the wound edge to the affected area calculated by the calculation device 311 and the auxiliary calculation device 317, or any symbol defined based on the distance information, can be used.

[0128] For example, when distance information from the wound edge to the inside of the affected area is added as an auxiliary evaluation value, the display device 223 adds distance information from the wound edge to the inside of the affected area (e.g., 3 mm) to the evaluation value, and displays it as d2 (3 mm). When an arbitrary symbol defined based on distance information from the wound edge to the inside of the affected area is added as an auxiliary evaluation value, for example, when the evaluation value is d2 closer to d1, the display device 223 adds an auxiliary evaluation value + to the evaluation value d2 and displays it as d2+. Conversely, when the evaluation value is d2 closer to D3, the display device 223 adds an auxiliary evaluation value - to the evaluation value d2 and displays it as d2-. Note that the method of adding an auxiliary evaluation value to the evaluation value and displaying the severity is not limited to the above example, and can be set arbitrarily.

[0129] By dividing the severity into smaller parts and displaying it in this way, it becomes possible to evaluate the affected area in more detail, for example, when the affected area is photographed multiple times during the healing process from the initial treatment to complete recovery. In other words, even if the DESIGN-R result has the same evaluation value, it becomes possible to evaluate whether the affected area is improving or worsening. In addition, by indicating the severity of multiple points within the affected area using the evaluation value and auxiliary evaluation value, it becomes possible to evaluate the parts within the affected area that are improving and those that are worsening. This makes it possible to evaluate the degree of improvement of the affected area according to the actual condition of the affected area, rather than just the definition of DESIGN-R.

[0130] It is desirable to set the auxiliary evaluation value appropriately according to the location of the affected area, the tissue reached by the affected area, and the attributes of the subject 101. For example, if the location of the affected area is the buttocks, the tissue reached by the affected area is subcutaneous tissue, and the attribute of the subject 101 is an obese body type, the thickness of the skin tends to be thick, so it is advisable to indicate the auxiliary evaluation value in more detail. It is noted that the selection of the evaluation value and the assignment of the auxiliary evaluation value described above may be input by the user, or may be determined and input by automatic analysis.

[0131] Fig. 21(d) shows an example of a display of a confirmation notice 809 for asking the user whether the selected evaluation value is acceptable after the evaluation value is selected in Fig. 21(c). When the user selects via the operation unit 224 that the selected evaluation value is acceptable, the display device 223 transitions the screen to the display shown in Fig. 21(e). FIG. 21(e) shows an example in which, as a result of an input of an evaluation value, the display of evaluation item 810 for D (depth) changes from "non" to "d2" and evaluation item 810 is displayed in a shaded manner. Similarly, a message is displayed prompting the user to enter evaluation values ​​for E (exudate), I (inflammation / infection), G (granulation tissue), N (necrotic tissue), and P (pocket) until there are no more evaluation items left to be entered. Then, when the evaluation values ​​for all evaluation items have been input, a display is displayed on the display device 223 to notify the user that the input of the affected area information has been completed, and then the imaging device 200 returns to S402 and starts live view processing.

[0132] In this way, in the system of the embodiment, in the process from S418 to S420, after imaging of the affected area, a display is displayed to prompt the user to input evaluation values ​​of evaluation items that were not automatically analyzed and information on the part of the affected area, and the user is provided with a function to input the affected area information. In this way, it is possible to input the affected area information, which has conventionally been input using a separate medium, only using the imaging device. In addition, the system of this embodiment determines whether all affected area information has been entered before photographing the next affected area, and sequentially prompts the user to enter evaluation items that have not been entered, thereby making it possible to prevent omissions in inputting affected area information.

[0133] The operation unit 224 according to this embodiment may be an input device using voice recognition. 21(a), when inputting the affected area, the area is displayed as an item with characters such as "Head" and "Shoulder" and can be selected. In contrast, as shown in FIG. 21(f), the display device 223 may be configured to display a human body model 811 and allow the user to specify the area of ​​the affected region using a touch sensor or the like provided on the display device 223. At that time, as shown in FIG. 21(g), the human body model 811 may be configured to be enlarged, reduced, or rotated to make it easier to select the affected area.

[0134] In addition, in Figure 21(e), a shaded display is used as a means of indicating that the evaluation item has been completed for input of the evaluation value, but the brightness of the characters may be reduced or the display may be inverted, and the method of display is not limited to these as long as it can be clearly indicated to the user. In addition, in this embodiment, DESIGN-R is used as an evaluation index for pressure ulcers that can be input, but this is not limited to this. For example, other evaluation indexes such as the Bates-Jensen Wound Assessment Tool (BWAT), Pressure Ulcer Scale for Healing (PUSH), and Pressure Sore Status Tool (PSST) may be used. In other words, a user interface for inputting evaluation items in BWAT, PUSH, PSST, etc. may be displayed in response to the extraction result of the pressure ulcer area and the information on the size of the extracted area being obtained.

[0135] In addition, although the present embodiment has been described with an example of a configuration for inputting evaluation values ​​of evaluation items for bedsores, evaluation values ​​of evaluation items for other skin diseases may be used as long as they are visual evaluation items, such as Severity Scoring of Atopic Dermatitis (SCORAD) for atopic dermatitis, and Body Surface Area, Psoriasis Area and Severity Index (PASI) for psoriasis.

[0136] Next, a method for comparatively displaying the severity of the depth of the affected area 102 during the healing process when affected area information of the affected area 102 of the subject 101 is acquired multiple times during the healing process of the affected area will be described. Fig. 22 is a flowchart of a process for comparing the results before and after the healing process for the severity of depth, which is one of the evaluation items for the affected area 102, and displaying the comparison result on the display device 223 of the imaging device 200. That is, in this flowchart, a past evaluation value previously obtained in a predetermined part of the affected area is compared with a newly obtained new evaluation value, and if the past evaluation value and the new evaluation value are the same, difference information based on the distance information acquired corresponding to the predetermined part is displayed. Note that the process of the flowchart of Fig. 22 may be included in any of the processes from S419 to S420 in the flowchart shown in Fig. 4, or the process of comparing the depth before and after S1300 may be incorporated after S420.

[0137] In S1301, the arithmetic device 311 of the image processing device 300 compares how far the affected area 102 has reached to what tissue. The arithmetic device 311 compares how far the affected area has reached from the affected area information of the evaluation value and auxiliary evaluation value input in S419, and the evaluation value and auxiliary evaluation value calculated based on the distance information from the wound edge to the inside of the affected area obtained by imaging and analyzing the affected area 102. That is, here, the reached tissue that a predetermined part of the affected area has reached is determined based on the distance information, and the past evaluation value and the new evaluation value are evaluation values ​​based on the reached tissue.

[0138] As a result of comparing the tissues reached by the diseased area in S1301, if the tissues reached by the previous result, which is the past evaluation value, and the current result, which is the new evaluation value, are different, the calculation device 311 proceeds to S1302. Then, the calculation device 311 displays the evaluation result according to the definition of DESIGN-R on the display device 223 of the imaging device 200. For example, as shown in the comparison point P1 in Fig. 23, if the previous result is D4 (damage beyond the subcutaneous tissue) and the current result is D3 (damage to the subcutaneous tissue), the calculation device 311 displays the result of only the evaluation value without giving an auxiliary evaluation value as the severity. On the other hand, if the tissues reached by the diseased area are the same, the calculation device 311 proceeds to S1303.

[0139] In S1303, the arithmetic device 311 compares the depths based on the distance information from the wound edge to the inside of the affected area calculated as described above. If the result of comparing the depths in S1303 shows that the distance information from the wound edge to the inside of the affected area differs between the previous result and the current result, the arithmetic device 311 proceeds to S1304. Here, the threshold value used as the criterion for comparing the depths can be set arbitrarily.

[0140] When the process proceeds to S1304, the arithmetic device 311 causes the severity of the affected area to be displayed in detail using the evaluation value and auxiliary evaluation value. For example, as shown at the comparison point P2 in FIG. 23, the evaluation value for both the previous result and the current result is d2 (damage down to the dermis), but the auxiliary evaluation value is further added and the result is displayed. Here, the current result is displayed with a + indicating d2 closer to d1. This allows the user to distinguish whether the affected area is improving or worsening, even though it has reached the same tissue in both the previous and current cases. In this case, the + indicating d2 closer to d1 is added, so it can be seen that there is a tendency towards improvement.

[0141] As shown in the comparison point P3 in FIG. 23, the evaluation value for both the previous and current results is D3 (damage to the subcutaneous tissue), but the calculation device 311 displays the results by adding an auxiliary evaluation value. Here, the distance information from the wound edge to the affected area is added and displayed. As a result, even though the affected area reaches the same tissue in the previous and current results, it is possible to determine whether the tissue is improving or worsening. In this case, the distance information from the wound edge to the affected area in the previous result is 5 mm, while the distance information from the wound edge to the affected area in the current result is 4 mm, indicating a tendency to improve.

[0142] On the other hand, if the result of comparing the depths in S1303 shows that the distance information from the wound edge to the affected area is the same between the previous and current results, the calculation device 311 proceeds to S1305 and displays the same evaluation result as the previous result, for example, as shown at comparison point P4 in Figure 23.

[0143] In addition, as shown in Figure 23, depending on the size of the affected area, it is possible to evaluate the affected area in more detail than just the depth from the wound edge to the wound bottom defined in DESIGN-R. In other words, by indicating the severity of multiple points within the affected area (for example, comparison points P1 to P4) using evaluation values ​​and auxiliary evaluation values, it becomes possible to evaluate the affected area in more detail. Furthermore, by displaying distance information from the wound edge to the inside of the affected area at any cross section (e.g., cross section A-A') in the affected area in Figure 23, such as the distance map shown in Figure 24, the healing process of the affected area can be visually evaluated.

[0144] Here, the distance map of FIG. 24 shows the relationship between the cross section A-A' of the affected area 102 and the distance information. The horizontal axis shows the position of the cross section A-A' of the affected area 102, and the vertical axis shows the depth of the affected area 102, that is, the distance information. The distance information is written so that the reference plane is zero (0) and the value increases toward the vertical axis direction. As shown in FIG. 24, the vertical axis may display evaluation values ​​d0 to DU indicating the severity of D (depth) instead of distance information. Here, d1 to D4 are displayed. It is preferable that the cross section A-A' be selected by the user by operating the operation unit 224 or a touch panel or the like while the affected area 102 is displayed on the display device 223. In addition, since the severity of multiple points in the affected area is input using the evaluation value or auxiliary evaluation value, it is also possible to display the improved and deteriorated parts in the affected area as, for example, image layers, as shown in FIG. 25.

[0145] Thus, according to this embodiment, an image processing system can be provided in which a user captures an image of an affected area with the imaging device 200, and information indicating the size of the affected area is displayed on the display device 223 of the imaging device 200. This reduces the burden on medical personnel in evaluating the size of the affected area of ​​a bedsore, and the burden on the patient being evaluated. Furthermore, by calculating the size of the affected area based on a program, individual differences can be reduced compared to when medical personnel manually measure, and the accuracy of the evaluation of the size of the bedsore can be improved. Furthermore, the affected area can be calculated and displayed as an evaluation value to more accurately represent the scale of the bedsore. Note that since the function for the user to check whether the estimated area of ​​the affected area is valid during live view display is not essential, the processing steps of S406, S407, and S441 to S445 may be omitted.

[0146] The image processing device 300 may store information indicating the extraction result of the affected area, information indicating the size of the affected area, and image data of a superimposed image in which these pieces of information are superimposed in the storage device 312. The output device 314 can output one or more pieces of information or image data stored in the storage device 312 to an output device such as a connected display. By displaying the superimposed image on the display, a user other than the user who photographs the affected area can obtain information indicating the image and size of the affected area in real time or as a past evaluation value obtained in the past.

[0147] The arithmetic device 311 of the image processing device 300 may also have a function of displaying a scale bar, the position and angle of which can be arbitrarily changed, for image data transmitted from the output device 314 to the display. By displaying such a scale bar, a user viewing the display can measure the length of any point in the affected area. It is desirable that the width of the memory of this scale bar is automatically adjusted based on the distance information, zoom magnification information, and size (number of pixels) information of the resized image data received in S451.

[0148] Furthermore, if the image processing device 300 is used in a stationary state with a constant power supply, there is no need to worry about the battery running out, and information indicating the image and size of the affected area can be obtained at any time. Furthermore, the image processing device 300, which is generally a stationary device, has a large memory capacity, so a large amount of image data can be stored.

[0149] Furthermore, according to this embodiment, when a user images an affected area using the imaging device 200, the user can input and record information about the affected area that is different from the information obtained by image analysis of the image. This eliminates the need to later input evaluations of the affected area into an electronic medical record or paper medium while looking at the captured image data. Furthermore, by presenting the user with information about items that have not been input, it is possible to prevent the user from forgetting to input the information when the user images the affected area.

[0150] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0151] 200: imaging device, 216: distance measurement system, 217: image processing circuit, 219: communication device, 220: system control circuit, 221: imaging unit, 300: image processing device, 311: arithmetic device, 314: output device, 317: auxiliary arithmetic device

Claims

1. An acquisition means for acquiring distance information relating to a distance from an imaging device to a subject; an extraction means for extracting an affected area of ​​the subject from image data obtained by capturing an image of the subject by the imaging device, and further extracting color information of the affected area; A calculation means for generating depth information corresponding to the affected area based on the distance information; A determination means for determining a state of the affected area based on the depth information and the color information; having The extraction means extracts a predetermined area from the image data that is set in the vicinity of the affected area and does not include the affected area, and sets at least three reference points in the predetermined area; The image processing device according to the present invention is characterized in that the calculation means sets a reference plane including the reference point, and generates depth information corresponding to the affected area based on the reference plane.

2. 2. The image processing device according to claim 1, wherein the determination means determines a first state existing in the affected area based on the color information, and after determining the first state, determines a second state existing in the affected area based on the color information and the depth information.

3. 3. The image processing device according to claim 1, wherein, when the reference surface is not directly opposed to the imaging surface of the imaging device, the calculation means corrects the depth information so that the reference surface is directly opposed to the imaging surface of the imaging device.

4. 4. The image processing apparatus according to claim 1, wherein the calculation means sets the reference surface to a plane or a sphere so as to correspond to a shape of the subject.

5. a storage means for storing the distance information at the reference point and information on the position of the reference point on the reference plane; A generating unit that generates data for displaying an image based on the image data, 5. The image processing device according to claim 1, wherein the generating means generates data for displaying the distance information and the position of the reference point stored in the memory means together with data for displaying the affected area.

6. A setting means for setting an arbitrary cross section in the predetermined region; and a tilt acquisition means for acquiring a tilt of the distance information in the cross section, 6. The image processing apparatus according to claim 1, wherein the determining unit determines whether the inclination is equal to or greater than a predetermined threshold value.

7. A setting means for setting an arbitrary cross section in the predetermined region; and a gradient acquisition means for acquiring a gradient of the received light signal intensity in the cross section, 6. The image processing apparatus according to claim 1, wherein the determining unit determines whether the inclination is equal to or greater than a predetermined threshold value.

8. 8. The image processing apparatus according to claim 6, wherein the determining means varies the predetermined threshold value in accordance with the distance information in the cross section.

9. The determination means acquires a position where the inclination is equal to or greater than the predetermined threshold value with respect to the predetermined region, 9. The image processing apparatus according to claim 6, further comprising a generating unit that generates data for displaying the position where the inclination is equal to or greater than the predetermined threshold value.

10. 10. The image processing device according to claim 6, further comprising: a unit that stores the gradient in the predetermined region and performs machine learning using the gradient.

11. The acquisition means includes: First distance information on a distance from the imaging device to the subject in a state where the imaging device is located at a first position directly facing a measurement standard with respect to the subject; acquire second distance information relating to a distance from the imaging device to the subject in a state in which the imaging device is located at a second position with respect to the subject that is different from the first position; The extraction means includes: extracting a predetermined first region from first image data captured with the imaging device located at the first position; A second area including the first area is extracted based on the first image data, second image data captured in a state where the imaging device is located at the second position, the first distance information, and the second distance information.

2. The image processing device according to claim 1,

12. The image processing apparatus according to claim 11 , wherein the second position is set based on the first image data and the first distance information.

13. 13. An image processing apparatus according to claim 11 or 12, comprising user assistance means for transferring the imaging device from the first position to the second position.

14. 14. The image processing device according to claim 11, further comprising a position and orientation acquisition means for acquiring position and orientation information of the imaging device based on acceleration information from an acceleration sensor included in the imaging device and angular velocity information from an angular velocity sensor included in the imaging device.

15. The image processing device described in claim 5 or claim 9, characterized in that the generation means compares a past evaluation value obtained in the past for a specified portion of the affected area with a newly obtained new evaluation value, and when it is determined that the past evaluation value and the new evaluation value are identical, generates data for displaying difference information based on the distance information obtained corresponding to the specified portion.

16. 16. The image processing apparatus according to claim 15, wherein the generating means generates data for displaying the predetermined portion from which the difference information has been obtained in a manner superimposed on the affected area.

17. An image processing method executed by an image processing device, comprising: An acquisition step of acquiring distance information relating to a distance from an imaging device to a subject; an extraction step of extracting an affected area of ​​the subject from image data of the subject captured by the imaging device, and further extracting color information of the affected area; a calculation step of generating depth information corresponding to the diseased area based on the distance information; a determining step of determining a state of the affected area based on the depth information and the color information; having In the extraction step, a predetermined area is extracted from the image data, the predetermined area being set in the vicinity of the affected area and not including the affected area, and at least three reference points are set in the predetermined area; The image processing method according to the present invention, characterized in that in the calculation step, a reference plane including the reference point is set, and depth information corresponding to the affected area is generated based on the reference plane.

18. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 16.

Images