Image Processing Apparatus, Image Processing Method, and Storage Medium

Abstract

The present invention relates to an image processing apparatus, an image processing method, and a storage medium. At least one processor of the apparatus functions as: a generation unit configured to at least identify an outer edge of a specific region in a surface layer of a subject and generate an outer edge candidate; and a control unit configured to select an outer edge candidate from the generated outer edge candidates based on an instruction from a user.

Description

Technical Field

[0001] Aspects of the embodiments relate to techniques for processing images obtained by imaging. Background Art

[0002] A pressure ulcer (so-called bedsore) is an example of an injury that occurs in the surface layer of human or animal skin. One of the indices for evaluating the severity of a pressure ulcer is the size of the pressure ulcer. Currently, the size of a pressure ulcer is obtained based on values measured by manual operations such as contacting, with a measuring tool, the area where the pressure ulcer has occurred (referred to as the affected part). When obtaining the size of a pressure ulcer, medical staff measure the direct distance between two points presumed to be the farthest on the outer edge of the affected part where the pressure ulcer has occurred as the major axis length. The distance between two points on the outer edge in a direction perpendicular to its major axis is measured as the minor axis length. However, since the shape of a pressure ulcer is usually complex, when measuring the affected part of the pressure ulcer, medical staff should adjust how to bring the measuring tool into contact with the pressure ulcer. This operation is performed at least twice for the measurement of the major axis length and the minor axis length. Medical staff obtain the size of the pressure ulcer by multiplying the manually measured major axis length and minor axis length.

[0003] Medical staff perform operations periodically to obtain the size of the pressure ulcer as described above, for example, to evaluate the progress of healing of the pressure ulcer. However, the obtained size varies depending on where on the affected part of the pressure ulcer the major axis length and the minor axis length are measured. Therefore, medical staff should appropriately adjust, each time, how to bring the measuring tool into contact with the affected part of the pressure ulcer, for example, to appropriately evaluate the progress of healing of the pressure ulcer. In the existing situation, the size is obtained by multiplying the major axis length and the minor axis length of the pressure ulcer. However, there is a possibility that the severity of the pressure ulcer can be more appropriately evaluated by using the area of the affected part of the pressure ulcer as the size of the pressure ulcer. Although the affected part where the pressure ulcer has occurred is used as an example here, the above matters are common for cases where measurements and evaluations are performed for affected parts other than the affected part of the pressure ulcer, such as burns and lacerations, and for affected parts related to other medical cares.

[0004] Medical staff bear the heavy burden of the above-described measurements and evaluations performed by manual operations. Patients also bear a heavy burden, for example, when they are required to maintain the same posture during the measurement.

[0005] In view of the above, it can be considered that, instead of measurement and evaluation performed by manual work, image processing of an image captured by an imaging device is performed for automatic identification and size measurement of an affected part or, for example, automatic calculation of an affected part area, and this makes it possible to reduce the burden on medical staff and patients. Techniques that can be used for automatically identifying a specific area (such as an affected part of a pressure ulcer in the surface layer of the skin, etc.) in the surface layer of an object by image processing are disclosed in Japanese Unexamined Patent Application Publication No. 2006-146775 and Japanese Unexamined Patent Application Publication No. 2004-185555. Japanese Unexamined Patent Application Publication No. 2006-146775 discloses a technique for automatically identifying the shape of a specific area in such a manner that an image is scanned spirally from the center to the outside and features are calculated based on color changes. Japanese Unexamined Patent Application Publication No. 2004-185555 discloses a technique for providing, as candidates, images of specific areas of a person's face with the nose as the center, and examples of these images include images of areas including the lips, images of areas including the chin, and images of areas including the neck.

[0006] Automatically identifying a specific area such as an injury occurring in the surface layer of an object by image processing may cause the problems described below. Here, the affected part of the above pressure ulcer is taken as an example of the specific area. The imaging device determines which area is regarded as the affected part during shooting. However, in many cases, the range of the image of the affected part required for evaluation changes according to the purpose of the evaluation performed by medical staff. For example, in some cases, medical staff only evaluate the part with the largest injury in the affected part, and in other cases, they evaluate a large area of the normal skin part including the outer edge starting from the center of the affected part. For example, in the case where the affected part is shrinking, in some cases, medical staff evaluate the healing progress of the affected part within a range including the same range as the range of the affected part in the previous shooting.

[0007] According to the technique disclosed in Japanese Unexamined Patent Application Publication No. 2006-146775, the identification of features makes it possible to roughly identify the boundary between the affected part and its surrounding area. However, it is difficult to strictly identify the affected part and its surrounding area. Similarly, for the technique disclosed in Japanese Unexamined Patent Application Publication No. 2004-185555, it is difficult to strictly identify the affected part and its surrounding area. The techniques disclosed in Japanese Unexamined Patent Application Publication No. 2006-146775 and Japanese Unexamined Patent Application Publication No. 2004-185555 cannot accurately identify where the affected part extends from the center of the affected part to the periphery. For this reason, there is a possibility that the image of the affected part obtained through the identification process does not include the entire affected part or a part thereof that medical staff need. Even in the case of performing image processing and automatic identification, for example, the necessity of shooting imposes a heavy burden on medical staff and patients. Therefore, situations that require reshooting should be avoided as much as possible. In the case of automatically identifying a specific area (such as an injury occurring in the surface layer of an object) by image processing as described above, it is desired to identify the specific area with high accuracy. Summary of the Invention

[0008] One aspect of the embodiments provides an apparatus, including: at least one processor configured to operate as the following units: a generation unit configured to at least identify an outer edge of a specific region in a surface layer of a subject from an image of the subject and generate an outer edge candidate; and a control unit configured to select an outer edge candidate from the generated outer edge candidates based on an instruction from a user.

[0009] A method includes: generating an outer edge candidate by at least identifying an outer edge of a specific region in a surface layer of a subject from an image of the subject; and selecting an outer edge candidate from the generated outer edge candidates based on an instruction from a user.

[0010] A non-transitory computer-readable storage medium storing an instruction program for causing a computer to perform a method, the method including: generating an outer edge candidate by at least identifying an outer edge of a specific region in a surface layer of a subject; and selecting an outer edge candidate from the generated outer edge candidates based on an instruction from a user.

[0011] More features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0012] Figure 1 Schematically shows a lesion photographing device.

[0013] Figure 2 Shows an example of the structure of an imaging device.

[0014] Figure 3 Shows an example of the structure of an image processing device.

[0015] Figure 4A and Figure 4B Shows a flowchart of the operation of the lesion photographing device according to the first embodiment.

[0016] Figure 5 Shows an image of a lesion to be photographed and evaluated.

[0017] Figure 6A and Figure 6B Shows a flowchart of the operation of the lesion photographing device according to the second embodiment.

[0018] Figure 7A and Figure 7B Shows a flowchart of the operation of the lesion photographing device according to the third embodiment.

[0019] Figure 8 Shows a method for calculating the area of a lesion.

[0020] Figure 9A and Figure 9B shows an example in which the extraction result and size information of the affected area are superimposed on an image.

[0021] Figure 10 shows an example of a release operation in the imaging device.

[0022] Figure 11 shows another example of a release operation in the imaging device.

[0023] Figure 12A 、 Figure 12B and Figure 12C shows an example in which the affected area, the major axis length, and the minor axis length are superimposed on an image.

[0024] Figure 13A and Figure 13B shows a flowchart of the operation of the affected area photographing device according to the fourth embodiment.

[0025] Figure 14 schematically shows the affected area photographing device according to the fifth embodiment.

[0026] Figure 15A and Figure 15B shows a flowchart of the operation of the affected area photographing device according to the fifth embodiment.

[0027] Figure 16 shows an example of a selection screen displayed on the terminal device.

[0028] Figure 17 shows an example of a list screen displayed on the terminal device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Hereinafter, embodiments for carrying out the present invention will be described in detail by way of examples with reference to the accompanying drawings. According to an embodiment of the present invention, as an example of identifying a specific area such as damage occurring in the surface layer of a subject by image processing, the estimation of the area of a pressure ulcer (so-called bedsore) occurring in the surface layer of human or animal skin will be described. Before describing the structure and processing of the image processing device according to each embodiment in detail, an example of a pressure ulcer occurring in the surface layer of the skin and the evaluation of the medical condition of the pressure ulcer will be described.

[0030] In a state where a human or an animal lies down, the part where the body contacts the floor or the mattress under the body is squeezed due to the body weight. Continuing the same posture causes blood circulation failure and necrosis of the surrounding tissues in the part of the body in contact with the floor. The necrotic state of this tissue is called a pressure ulcer or bedsore. Patients with pressure ulcers receive pressure ulcer care such as body pressure redistribution care and skin care provided by medical staff, and thus the pressure ulcers should be evaluated and managed periodically.

[0031] As a tool for evaluating the healing process of injuries including pressure ulcers, the Pressure Ulcer Status Judgment Scale DESIGN-R (registered trademark), developed by the Academic Education Committee of the Japanese Pressure Ulcer Society, is proposed.

[0032] Reference: "Pressure Ulcer Guide" by Shorinsha, Second Edition, based on the Pressure Ulcer Prevention and Management Guidelines (Fourth Edition), edited by the Japanese Pressure Ulcer Society, page 23.

[0033] DESIGN-R is a tool for evaluating the healing process of injuries including pressure ulcers. In DESIGN-R, each observation item of depth, exudate, size, inflammation and infection, granulation tissue, and necrotic tissue is evaluated. DESIGN-R is defined by the following two parts, one for the purpose of severity classification for daily simple evaluation, and the other for the purpose of process evaluation for detailed evaluation of the healing process. In DESIGN-R for severity classification, the six evaluation items are classified into two items of mild severity and severe severity. At the first visit, DESIGN-R for severity classification is used for evaluation, and this enables a rough grasp of the state of the pressure ulcer. It is known which item is the problem, and it is easy to decide the treatment plan. DESIGN-R defined for process evaluation enables comparison of severity between patients in addition to process evaluation. The evaluation items are weighted, and the total score (score from 0 to 66) of the six items other than depth represents the severity of the pressure ulcer. As a result, after the start of treatment, the treatment process can be objectively evaluated in detail. Not only can individual process evaluation be performed, but also comparison of severity between patients can be performed.

[0034] For example, the size of a pressure ulcer (hereinafter referred to as size) is evaluated by measuring the length of the long axis and the length of the short axis (axis perpendicular to the long axis) of the skin injury area, and the numerical values of the product are classified into seven stages. For DESIGN-R, it is recommended to measure and grade the skin injury area once a week to every two weeks to evaluate the healing process of the pressure ulcer and select appropriate care. Therefore, the medical condition of the pressure ulcer should be evaluated and managed regularly. Accurate evaluation of the pressure ulcer is required to understand the changes in the medical condition of the pressure ulcer. In the existing situation, the affected part of the pressure ulcer is measured manually. However, medical staff (hereinafter referred to as users) bear the heavy burden of the above-mentioned measurement and evaluation performed manually. Patients also bear a heavy burden.

[0035] In view of this, the image processing device according to the aspect of the embodiment can accurately identify the affected part with high precision and reduce the burden on users, for example, by estimating candidates for a specific area and the outer edge in the surface layer of the subject from the image of the subject and obtaining information used for evaluation of the affected part.

[0036] First Embodiment

[0037] Figure 1 Schematically shows an affected part photographing device 1 which is an example of an image processing device as an aspect according to an embodiment. The affected part photographing device 1 according to the first embodiment includes an imaging device 200 which is a portable device that can be carried and an image processing device 300 such as a personal computer. In Figure 1 the example, the imaging device 200 and the image processing device 300 are different components. However, the imaging device 200 may have the function of the image processing device 300, or the image processing device 300 may have the function of the imaging device 200. According to the present embodiment, the condition of the affected part 102 of the subject 101 where a pressure ulcer occurs on the buttocks is taken as an example. However, this is not a limitation. The affected part 102 may be an affected part such as a burn or a laceration, or an affected part related to other medical care.

[0038] The imaging device 200 of the affected part photographing device 1 according to the present embodiment photographs the affected part 102 of the subject 101, obtains the subject distance, and sends its data to the image processing device 300. The image processing device 300 identifies the affected part from the received image data, identifies a closed curve that can correspond to the outer edge of the affected part 102, and determines that the closed curve corresponds to the outer shape of the affected part.

[0039] Figure 2 Shows an example of the hardware structure of the imaging device 200 included in the affected part photographing device 1. The imaging device 200 may be a typical single-lens digital camera, a compact digital camera, or a smart phone or a tablet including a camera with an autofocus function.

[0040] The imaging unit 211 includes a lens group 212, a shutter 213, and an image sensor 214. The lens group 212 includes a focusing lens and a zoom lens, and the focusing position and the zoom ratio can be changed by changing the position of the lens. The lens group 212 also includes an aperture for adjusting the exposure amount.

[0041] The image sensor 214 includes, for example, a charge accumulation solid-state image sensor such as a CCD or a CMOS sensor that converts an optical image formed by the lens group 212 into image data. The reflected light from the subject passes through the lens group 212 and the shutter 213, and converges on the imaging surface of the image sensor 214 as a subject image. The image sensor 214 generates an analog electrical signal based on the subject image and outputs the image data obtained by converting the electrical signal into a digital form.

[0042] The shutter 213 enables the image sensor 214 to be exposed or blocked from light by opening or closing shutter blades, thereby controlling the exposure time of the image sensor 214. Instead of the shutter 213, an electronic shutter that controls the exposure time by driving the image sensor 214 can be used. In the case of the CMOS sensor operating the electronic shutter, a reset scan is performed to reduce the accumulated charge amount of each pixel to zero for each pixel or each pixel region (e.g., each row). Subsequently, a scan is performed such that, for the pixels or regions for which the reset scan has been performed, a signal depending on the accumulated charge amount is read after a predetermined time.

[0043] The distance measurement system 216 calculates information related to the distance to the subject. Examples of the distance measurement system 216 can include a typical distance measurement sensor using the phase difference method equipped in a single-lens reflex camera, and a system using a TOF (Time of Flight) sensor. The TOF sensor measures the distance to the subject based on the time difference (or phase difference) between the transmission timing of the transmitted wave and the reception timing of the reflected wave corresponding to the transmitted wave reflected from the subject. The distance measurement system 216 can use the PSD (Position Sensitive Device) method using a PSD as a light receiving element.

[0044] In the case where the image sensor 214 is, for example, an imaging surface phase difference sensor, the distance measurement system 216 can obtain distance information based on the phase difference signals for each pixel output from the image sensor 214. The image sensor 214 adopting the imaging surface phase difference method obtains a pupil split image by using an imaging element including a photoelectric conversion element arranged relative to a microlens and having pixels, and can detect focus by obtaining the phase difference of the pupil split image. In this case, the distance measurement system 216 obtains distance information for each position in each pixel or region by using the signals of the phase difference between the images obtained from the photoelectric conversion elements corresponding to the respective pupil regions output from the image sensor 214.

[0045] The distance measurement system 216 can obtain distance information in one or more predetermined distance measurement regions within each image, or can obtain a distance map representing the distribution of the distance information of the pixels or regions in the image. In addition to this, the distance measurement system 216 can use TV-AF or contrast AF to determine the position of the focusing lens where the integrated value of the high-frequency components of the extracted image data is the largest, and obtain distance information from this position of the focusing lens.

[0046] The image processing circuit 217 performs predetermined image processing on the image data output from the image sensor 214 of the imaging unit 211 or the image data recorded in the internal memory 221. The image processing performed by the image processing circuit 217 includes various image processes such as white balance adjustment, gamma correction, color interpolation, demosaicing, and filtering. The image processing circuit 217 may perform compression processing on the image data on which the image processing has been performed based on a standard such as JPEG.

[0047] The AF control circuit 218 determines the position of the focus lens included in the lens unit 212 based on the distance information obtained by the distance measurement system 216, and controls a motor (not shown) for driving the focus lens based on this position to control the focus position.

[0048] The zoom control circuit 215 controls a motor (not shown) for driving the zoom lens included in the lens unit 212 to control the optical magnification of the lens unit 212.

[0049] The communication device 219 is a communication interface used by the imaging device 200 to communicate with an external device such as an image processing device 300 via a wireless network (not shown). A specific example of the network is a network based on the Wi-Fi (registered trademark) standard. Communication with Wi-Fi can be achieved via a router. The communication device 219 can communicate with an external device such as an image processing device 300 by using a wired communication interface such as USB or LAN.

[0050] The system control circuit 220 includes a CPU (Central Processing Unit), which controls the components of the imaging device 200 according to the program stored in the internal memory 221. With Figure 2 this structure, the system control circuit 220 controls, for example, the imaging unit 211, the zoom control circuit 215, the distance measurement system 216, the image processing circuit 217, and the AF control circuit 218. Instead of or in addition to the CPU, the system control circuit 220 may also include an FPGA or an ASIC.

[0051] The internal memory 221 is a rewritable memory such as a flash memory or an SDRAM. The internal memory 221 temporarily stores information related to various settings such as information related to the focus position used during imaging operation of the imaging device 200, data of the image captured by the imaging unit 211, and image data on which the image processing circuit 217 performs image processing. The internal memory 221 temporarily stores analysis data such as image data and information related to the size of the pressure ulcer described later received through communication with the image processing device 300 by the communication device 219.

[0052] The external memory I / F 222 is an interface for non-volatile storage media such as an SD card or a CF card (CF: CompactFlash is a registered trademark) that can be mounted on the imaging device 200. The external memory I / F 222 records the image data processed by the image processing circuit 217, the image data received through communication with the image processing device 300 by the communication device 219, or the analysis data in the non-volatile storage media. The external memory I / F 222 can read the image data stored in the non-volatile storage media and output the image data to the outside of the imaging device 200.

[0053] The display device 223 is a TFT (Thin Film Transistor) liquid crystal display or an organic EL display. The display device 223 may include an EVF (Electronic Viewfinder). The display device 223 displays an image based on the image data temporarily stored in the internal memory 221 or the image data stored in the non-volatile storage media described above, or displays the setting screen of the imaging device 200.

[0054] The operation unit 224 includes, for example, buttons, switches, keys, and mode dials included in the imaging device 200, and a touch screen shared with the display device 223. Instructions from the user such as mode setting instructions and shooting instructions are sent to the system control circuit 220 via the operation unit 224.

[0055] The imaging unit 211, the zoom control circuit 215, the distance measurement system 216, the image processing circuit 217, the AF control circuit 218, and the communication device 219 are connected to the common bus 225. The system control circuit 220, the internal memory 221, the external memory I / F 222, the display device 223, and the operation unit 224 are connected to the common bus 225. The common bus 225 corresponds to a communication line for transmitting and receiving signals between blocks.

[0056] Figure 3 An example of the hardware configuration of the image processing device 300 included in the affected part imaging device 1 is shown. The image processing device 300 includes a CPU (Central Processing Unit) 310, a storage device 312, a communication device 313, an output device 314, and an auxiliary arithmetic unit 317. The CPU 310 includes an arithmetic unit 311. The storage device 312 includes a main storage device 315 (such as a ROM or a RAM) and an auxiliary storage device 316 (such as a disk device or an SSD (Solid State Drive)).

[0057] The communication device 313 functions as a wireless communication module.

[0058] The output device 314 outputs the data processed by the arithmetic unit 311 or the data stored in the storage device 312 to a display, a printer, or an external network connected to the image processing device 300.

[0059] The auxiliary arithmetic unit 317 is an auxiliary processing IC used under the control of the CPU 310. An example thereof is a GPU (Graphics Processing Unit). The GPU, which is basically used as a processor for image processing, includes a multiply-accumulate unit and is excellent in matrix calculations. Therefore, in many cases, the GPU is used as a processor for processing including signal learning. The GPU is generally used for processing including deep learning. An example of the auxiliary arithmetic unit 317 is the Jetson TX2 module (Jetson is a registered trademark) manufactured by NVIDIA Corporation. The auxiliary arithmetic unit 317 may be an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The auxiliary arithmetic unit 317 identifies, recognizes, and extracts the affected part 102 of the subject 101 from the image data as described later.

[0060] The arithmetic unit 311 included in the CPU 310 runs the program stored in the storage device 312 to perform various functions as described later, including the calculation of the actual size or area of the affected part 102 extracted by the auxiliary arithmetic unit 317. The arithmetic unit 311 also controls the order of performing these functions.

[0061] The image processing device 300 may include a single CPU 310 and a single storage device 312, or may include a plurality of CPUs 310 and a plurality of storage devices 312. That is, in the case where at least one or more processing units (CPUs) and at least one storage device are connected, and the at least one or more processing units run the program stored in the at least one or more storage devices, the image processing device 300 performs the functions described later. Instead of or in addition to the CPU 310, an FPGA or an ASIC may be used.

[0062] The following reference Figures 1 - 5 The description, for example, includes a series of processes according to the present embodiment, in which a user such as a medical staff takes a picture of an affected part such as a pressure ulcer using the imaging device 200, generates information for use in the evaluation of the affected part from the captured image, and saves the information.

[0063] Figure 4A and Figure 4B The flowchart showing the operation of the affected part photographing device 1 according to the first embodiment.

[0064] In Figure 4A and Figure 4BIn this process, the processing of steps S401 to S420 on the left side of the process is performed by the imaging device 200, and the processing of steps S431, steps S441 to S445, and steps S451 to S455 on the right side of the process is performed by the image processing device 300.

[0065] The imaging device 200 and the image processing device 300 are connected to a network (not shown) conforming to the Wi-Fi standard, which is a wireless LAN standard. In step S431, the image processing device 300 performs a search process for searching for the imaging device 200 to be wirelessly connected. In step S401, the imaging device 200 performs a response process for the search process. The technology for searching for devices via the network is UPnP (Universal Plug and Play). According to UPnP, each device is identified by a UUID (Universal Unique Identifier).

[0066] In step S402, when the imaging device 200 is connected to the image processing device 300, the live view process is started. In the live view process, the imaging unit 211 converts the analog signal of the image captured at a predetermined frame rate into a digital signal to generate image data, and the image processing circuit 217 performs an imaging process for generating image data for live view display on the image data. These processes are repeatedly performed for each frame to display the live view image at the predetermined frame rate on the display device 223.

[0067] In step S403, the distance measurement system 216 obtains information related to the distance from the imaging device 200 to the subject 101 by any one of the above-described distance measurement methods. The AF control circuit 218 starts AF processing for driving control of the lens unit 212 based on this distance information so that the subject 101 is in focus. For example, in the case of adjusting the focus position by the above-described TV-AF or contrast AF, the distance measurement system 216 obtains information related to the distance to the focused subject 101 from the position of the focusing lens focused on the subject 101. It is possible to focus on the subject located at the center of the image or the subject located at the position closest to the imaging device 200. For example, when the AF control circuit 218 has a distance map of the subject, the AF control circuit 218 can estimate the region of interest from the distance map and can adjust the focus to the position of that region. When the position of the affected part 102 of the pressure ulcer has been identified in the live view image by the image processing device 300, the AF control circuit 218 can adjust the focus to that position. The imaging device 200 repeatedly performs AF processing and display of the live view image until it is detected in step S410 described later that the release button is pressed.

[0068] In step S404, the image processing circuit 217 then performs image display processing on the data of the image captured for real-time viewfinder image display, and performs compression processing to generate image data in, for example, the JPEG standard. The image processing circuit 217 performs resizing processing on the compressed image data to reduce the size of the image data. In step S406 described later, the image data after the resizing processing is transmitted to the image processing device 300 via wireless communication. At this time, as the size of the transmitted image data increases, the time for wireless communication increases. Therefore, the size of the image data reduced by the resizing processing in step S404 is determined in consideration of the allowable communication time.

[0069] In step S405, the system control circuit 220 then obtains the image data after the resizing processing generated in step S404 and the distance information obtained by the distance measurement system 216 in step S403. The system control circuit 220 obtains information related to the zoom ratio and information related to the size (number of pixels) of the image data after the resizing processing as needed.

[0070] In step S406, the system control circuit 220 causes the communication device 219 to transmit one or more pieces of information including at least the image data and the distance information obtained in step S405 to the image processing device 300 via wireless communication.

[0071] The processing in steps S404 to S406 can be performed for each frame, or can be performed once for every several frames.

[0072] Now, the processing performed by the image processing device 300 will be described.

[0073] In step S441, the communication device 313 of the image processing device 300 receives the image data and one or more pieces of information including distance information transmitted from the communication device 219 in step S406 of the imaging device 200.

[0074] In step S442, the CPU 310 and the auxiliary arithmetic unit 317 of the image processing device 300 identify the affected part 102 of the subject 101 from the image data received in step S441. According to the present embodiment, for example, the processing for identifying the affected part 102 is performed by semantic segmentation using deep learning. For this reason, according to the present embodiment, a high-performance computer (not shown) for learning generates a learned model by learning a neural network model using an image of an actual pressure ulcer affected part as teacher data. The data of the learned model is stored in the main storage device 315 or the auxiliary storage device 316. An example of the neural network model is a fully convolutional network (FCN) as a segmentation model of deep learning.

[0075] According to this embodiment, the auxiliary arithmetic unit 317 excellent in parallel execution of multiplication-accumulation operations performs inference processing of deep learning. That is, the auxiliary arithmetic unit 317 receives the learned model stored in the main storage device 315 or the auxiliary storage device 316 via the arithmetic unit 311, and estimates the affected part 102 of the pressure ulcer from the image data based on the learned model. The inference processing can be performed by an FPGA or an ASIC. Semantic segmentation can be achieved by using other deep learning models. The segmentation method is not limited to deep learning, and for example, it can be graph cut, region growing, edge detection, or divide-and-conquer method. In addition, the auxiliary arithmetic unit 317 can perform internal learning processing of the neural network model by using the image of the affected part of the pressure ulcer as teacher data.

[0076] In step S443, the arithmetic unit 311 of the CPU 310 then identifies the boundary lines 501 to 505 corresponding to the outer edge candidates of the affected part 102 based on the outer edge of the affected part 102 identified by the auxiliary arithmetic unit 317 in step S442. Figure 5 as shown.

[0077] Figure 5 Shows an image of the affected part to be photographed and evaluated according to this embodiment. In many cases where the pressure ulcer becomes particularly severe, exudate leaks from the center of the affected part 102 of the pressure ulcer, the color of the skin outside it changes due to another exudate leak, and the color of the skin outside it changes. The outer edges of the pressure ulcer in each state seem to define a closed curve. In Figure 5 the example, the closed curves defined by the outer edges of the pressure ulcer in each state are represented as boundary line 501, boundary line 502, boundary line 503, boundary line 504, and boundary line 505 in order from the center to the periphery of the pressure ulcer. That is, the boundary line 501, the boundary line 502, the boundary line 503, the boundary line 504, and the boundary line 505 correspond to the lines representing the outer shape (shape of the outer edge) of the pressure ulcer in each state.

[0078] According to this embodiment, when the auxiliary arithmetic unit 317 identifies the affected part 102 of the pressure ulcer based on the above-mentioned learned model in step S442, it also identifies the area of the pressure ulcer in each state and the outer edge of the pressure ulcer in each state. In step S443, the arithmetic unit 311 identifies the closed curve defined by the outer edge of the pressure ulcer in each state identified by the auxiliary arithmetic unit 317 as the boundary lines 501 to 505. At this time, the auxiliary arithmetic unit 317 estimates which of the boundaries 501 to 505 is most likely to be selected by the user as the outer edge of the affected part 102 of the pressure ulcer based on the result of the inference processing.

[0079] In step S444, the arithmetic unit 311 then determines, for the affected part 102, that the area surrounded by the boundary lines 501 to 505 corresponding to the outer edge candidates of the affected part 102 is the affected part, and the area including the boundary lines is classified into a temporary category. The arithmetic unit 311 stores information related to the temporary category of the boundary lines in the main storage device 315 or the auxiliary storage device 316. At this time, the arithmetic unit 311 also saves information indicating which category corresponds to the boundary line most likely to be selected by the user in step S443. In the case where the user designates an area in step S454 described later, classification is performed again, and the processing of classification in step S444 can be simple.

[0080] In step S445, the communication device 313 transmits information related to the recognition result of the affected part 102, information related to the category corresponding to the boundary lines 501 to 505, and information related to the category corresponding to the boundary line most likely to be selected to the imaging device 200. The communication device 313 also transmits the image data in which the affected part 102 whose outer shape corresponds to the boundary lines 501 to 505 is classified, generated in step S444, to the imaging device 200.

[0081] The processing performed by the imaging device 200 will be described below.

[0082] In step S407, the communication device 219 of the imaging device 200 receives the image data newly generated when the image data in which the affected part is classified by the image processing device 300.

[0083] In step S408, the system control circuit 220 then determines whether the image data of the categories of the above-mentioned boundary lines 501 to 505 and the image data of the category corresponding to the boundary line most likely to be selected by the user are received in step S407. If it is determined that the image data is received, the system control circuit 220 causes the process to proceed to step S409. If it is determined that the image data is not received, the process proceeds to step S410.

[0084] In step S409, the display device 223 displays, under the control of the system control circuit 220, the image data including the information related to the category of the affected part received in step S407 for a predetermined time. At this time, Figure 5 The images of the boundary lines 501 to 505 shown are displayed in a manner superimposed on the image of the affected part 102. At this time, regarding the category corresponding to the boundary line most likely to be selected by the user, the display of the boundary line can be emphasized, or the area of the category can be displayed in shadow, etc., to achieve obvious display. In the case where there are multiple boundary lines, the system control circuit 220 of the imaging device 200 can omit the display of the boundary line least likely to be selected by the user.

[0085] In step S410, the system control circuit 220 then detects whether the release button included in the operation unit 224 is pressed while the image of the display boundary line is superimposed on the image of the affected part 102 for a predetermined time. If the release button is not pressed while the image is displayed for the predetermined time, the system control circuit 220 causes the process to return to the process of step S404 performed by the imaging device 200. If the release button is pressed, the imaging device 200 performs the process of step S411.

[0086] In step S411, the distance measurement system 216 obtains information related to the distance to the subject in the same manner as in step S403, and the AF control circuit 218 performs AF processing for driving control of the lens unit 212 to focus on the subject.

[0087] In step S412, the imaging device 200 then captures a still image.

[0088] In step S413, the image processing circuit 217 then performs development and compression processing on the image data obtained by capturing the still image in step S412 to generate image data in, for example, the JPEG standard. The image processing circuit 217 performs resizing processing on the compressed image data to reduce the size of the image data. The size of the image data after the resizing processing in step S413 is equal to or larger than the size of the image data for which the resizing processing is performed in step S404. The reason is that accuracy has priority when obtaining the size or area of the affected part 102 from this image. For example, resizing processing is performed to obtain a 4-bit RGB color image of 1440 pixels × 1080 pixels, so that the size of the image data is approximately 4.45 megabytes. However, the size of the image after the resizing processing is not limited to this.

[0089] In step S414, the system control circuit 220 then obtains the image data after the resizing processing in step S413 and the distance information obtained in step S411. The system control circuit 220 obtains information related to the zoom ratio and information related to the size (number of pixels) of the image data after the resizing processing as needed.

[0090] In step S415, the communication device 219 then wirelessly transmits one or more pieces of information including at least the image data and the distance information obtained by the system control circuit 220 in step S414 to the image processing device 300.

[0091] The following description returns again to the processing performed by the image processing device 300.

[0092] In step S451 , the communication device 313 of the image processing apparatus 300 receives one or more information including the image data and the distance information transmitted from the communication device 219 of the camera 200 .

[0093] In step S452, the CPU 310 and the auxiliary arithmetic unit 317 of the image processing apparatus 300 then recognize the affected part 102 of the subject 101 from the image data received in step S451. Details of this process are the same as those of step S442, and description of the process is omitted.

[0094] In step S453, the operation unit 311 of the CPU 310 then identifies a region corresponding to the outer edge candidate of the affected part 102 based on the outer edge of the affected part 102 identified by the auxiliary operation unit 317 in step S452. Figure 5 The boundary lines 501 to 505 are shown. The details of this process are the same as the process of step S443, and the description of this process is omitted.

[0095] In step S454, the operation unit 311 then determines that the area surrounded by the boundary lines 501 to 505 obtained in step S453 and corresponding to the outer edge candidates of the affected part 102 is the affected part, and classifies the area including the boundary lines into categories. The operation unit 311 stores information on the categories of the boundary lines in the main storage device 315 or the auxiliary storage device 316. At this time, similarly to step S444, the operation unit 311 also stores information indicating which category corresponds to the boundary line that is estimated to be most likely to be selected by the user.

[0096] In step S455, the communication device 313 transmits information on the recognition result of the affected part 102, information on the categories corresponding to the boundary lines 501 to 505, and information on the categories corresponding to the boundary line most likely to be selected to the imaging device 200. The communication device 313 also transmits the image data generated in step S454, into which the affected part 102 whose outer shape corresponds to the boundary lines 501 to 505 is classified, to the imaging device 200.

[0097] The following description returns to the processing performed by the imaging device 200 .

[0098] In step S416 , the communication device 219 of the camera 200 receives the image data into which the image processing apparatus 300 classifies the affected part.

[0099] In step S417, the display device 223 then displays, under the control of the system control circuit 220, the image data received in step S416 that includes information related to the category of the affected part for a predetermined period of time. At this time, similar to step S409, the images of the boundary lines 501 to 505 are displayed in a manner superimposed on the image of the affected part 102. At this time, regarding the category corresponding to the boundary line most likely to be selected by the user, the display of the boundary line can be emphasized, or the area of the category can be displayed by shading, etc., for obvious display. In the case where there are multiple boundary lines, as described above, also in step S417, the system control circuit 220 can omit the display of the boundary line least likely to be selected by the user.

[0100] In step S418, the system control circuit 220 then receives the indication information in the case where the user inputs via the operation unit 224 to specify the category to be saved as the image of the affected part 102. The system control circuit 220 determines that the category specified by the user is saved as the image of the affected part 102. At this time, in the case where the user specifies the category corresponding to the boundary line among the boundary lines 501 to 505, the system control circuit 220 determines that this category is the category of the pressure ulcer to be photographed and evaluated at this time. The input of the image displayed on the display device 223 from the user can be received via the touch screen of the display device 223, or the specification from the user can be received via other pointing devices of the operation unit 224 to specify this category. This category can be specified in another way. For example, a boundary line can be specified, or the outermost boundary line closest to the position indicated by the user can be specified. The system control circuit 220 generates the image data of any one of the boundary lines 501 to 505 regarding the affected part 102 recognized by the image processing device 300, and this boundary line is specified as corresponding to the outer shape of the affected part 102 by the user's specification of the category.

[0101] In step S419, the system control circuit 220 then saves the image data obtained in step S412 and the category of the image of the affected part specified by the user, that is, the information related to the boundary line among the boundary lines 501 to 505 specified by the user as the best outer shape. This information is saved in the internal memory 221 or the external storage medium via the external memory I / F 222. The system control circuit 220 can save this information in an external storage device other than the internal memory 221 and the external storage medium via the communication device 219. In this case, it is important to save the image data of the affected part and the information related to the boundary line, and there is no restriction on where these two are saved.

[0102] In step S420, the display device 223 then displays the image data received in step S416 and to which the affected part 102 is classified for a predetermined period of time. Here, the display device 223 displaysFigure 5 the image shown, and the process returns to the process of step S402 after a predetermined time.

[0103] According to the first embodiment, the image processing apparatus 300 thus estimates candidates for the outer shape that can be selected as the outer edge of the affected part 102 during preview display such as the display of the affected part 102 on the imaging device 200. On the preview screen of the imaging device 200, the boundary lines 501 to 505 indicating the candidates for the outer edge of the affected part 102 are displayed in a manner superimposed on the image of the affected part 102. Therefore, a user such as a medical staff can easily know whether an image of the affected part 102 can be captured within a desired range. In this state, the imaging device 200 captures a still image within the range desired by the user in response to the release operation of the user. The image processing apparatus 300 estimates outer edge candidates that can be selected as the outer edge of the affected part 102 in the still image. The boundary lines 501 to 505 corresponding to these candidates are displayed on the display device 223 of the imaging device 200. In this state, when the user designates the category corresponding to the desired boundary line, the image of the affected part of the designated category and the information related to the boundary line corresponding to the category are displayed and saved. That is, the user evaluates the affected part 102 of, for example, a pressure ulcer based on the displayed and saved image of the affected part. In this way, the affected part photographing apparatus 1 according to the first embodiment reduces the burden on the medical staff and the burden on the patient to be evaluated, and can accurately evaluate the size of the affected part of, for example, a pressure ulcer. According to the present embodiment, the size of the affected part 102 is calculated based on a program, and the difference between individuals can be smaller than the difference between individuals in the case of measurement by, for example, manual work of the user, and the evaluation of the size of the pressure ulcer can be more accurate. In addition, according to the present embodiment, the affected part area, which is an index used to more accurately represent the size of the pressure ulcer, can be calculated by image processing, and the affected part area can be displayed. A method for calculating the affected part area will be described according to the embodiments described later.

[0104] When displaying the live view, the user does not have to confirm whether the region corresponding to the affected part is appropriate. Therefore, the processes of steps S441 to S445 can be omitted. Figure 4A

[0105] The image processing apparatus 300 can store the information related to the recognition result of the affected part 102, the information related to the size of the affected part 102, and the image data of the superimposed image on which this information is superimposed in the storage device 312. The output device 314 can output one or more pieces of information or image data stored in the storage device 312 to an output device such as a display connected thereto. Displaying the superimposed image on the display enables other users other than the user who photographed the affected part 102 to obtain the acquired or real-time image of the affected part 102 and the information related to the size.

[0106] The arithmetic unit 311 of the image processing device 300 may have a function of displaying a scale for freely changing the position and angle of the image data transmitted from the output device 314 to the display. Displaying the scale enables a user viewing the display to measure the length of a freely selected part of the affected area 102. The width of the scale ratio can be automatically adjusted based on, for example, information related to the distance information and zoom ratio received in step S451 and information related to the size (number of pixels) of the image data that has undergone resizing processing. When the image processing device 300 is used in a stationary state while being supplied with power, there is no concern about battery power depletion, and image and size information of the affected area 102 can be obtained at any time. The image processing device 300 is generally a fixed device, has a large storage capacity, and can store a large amount of image data. This also applies to other embodiments described later.

[0107] Second Embodiment

[0108] In the following description by way of example, the affected area photographing device 1 according to the second embodiment learns based on the result of an instruction from the user, and reflects an inference derived from the result of learning during continuous photographing and evaluation. Structures and steps that are the same as those according to the first embodiment described above in the second embodiment are designated by the same reference numerals as in the description of the first embodiment, and repeated descriptions of the same structures and steps are appropriately omitted.

[0109] The following references Figures 1 - 3 、 Figure 5 、 Figure 6A and Figure 6B The description includes a series of processes according to the second embodiment, such as from photographing the affected area until generation and storage of information used in the evaluation of the affected area. Figure 6A and Figure 6B show a flowchart of the operation of the affected area photographing device 1 according to the second embodiment. In Figure 6A and Figure 6B the processing of the steps on the left side of the flow is performed by the imaging device 200, and the processing of the steps on the right side of the flow is performed by the image processing device 300. The processing of steps S401 to S419, steps S431 to S444, and steps S451 to S454 is the same as the processing of the corresponding steps in Figure 4A and Figure 4B above, and the description of these steps is omitted.

[0110] According to the second embodiment, the processing of steps S401 to S406 is performed using the imaging device 200, and then the processing of step S441 is performed using the image processing device 300. After the processing of step S444, the processing of step S645 is performed.

[0111] In step S645, the arithmetic unit 311 of the image processing apparatus 300 searches for information related to the result of the inference that the boundary line selected by the user in the past is any one of the boundary lines 501 to 505 in the main storage device 315 or the auxiliary storage device 316.

[0112] In step S646, the arithmetic unit 311 then determines whether the learning data indicating that the boundary line selected by the user in the past is any one of the boundary lines 501 to 505 is in the main storage device 315 or the auxiliary storage device 316. When there is learning data and information related to the result of the inference based on the learning data, the arithmetic unit 311 causes the process to proceed to step S647. When there is no learning data, the process proceeds to step S648.

[0113] In step S647, the arithmetic unit 311 infers the boundary line among the boundary lines 501 to 505 that the user is most likely to select based on the learning data of the boundary line selected by the user in the past, and reflects the inference of the above category. The arithmetic unit 311 may attach a flag to the category corresponding to the inferred boundary line, or may determine the order of each category by using metadata. At this time, the image processing apparatus 300 may cause the output device 314 to display, for example, the inferred boundary line emphasized by using a thick line, or the category corresponding to the boundary line by using shading for the user to select. The image processing apparatus 300 may gray out the display of other categories other than the category corresponding to the inferred boundary line. The method of displaying the category is not limited to these examples.

[0114] In step S648, the arithmetic unit 311 then transmits information related to the recognition result of the affected part 102 extracted, information related to the category corresponding to the boundary lines 501 to 505, and information related to the category corresponding to the boundary line that the user is most likely to select to the communication device 313. The communication device 313 transmits this information to the imaging device 200.

[0115] After step S648, the imaging device 200 performs the process of step S407. The imaging device 200 performs the above processes of steps S407 to S415, and then the image processing apparatus 300 performs the process of step S451. After the processes of steps S451 to S454, the image processing apparatus 300 performs the process of step S655.

[0116] In step S655, the arithmetic unit 311 of the image processing apparatus 300 searches for information related to the result of the inference that the boundary line selected by the user in the past is any one of the boundary lines 501 to 505 in the main storage device 315 or the auxiliary storage device 316.

[0117] In step S656, the arithmetic unit 311 then determines whether learning data indicating that the boundary line selected by the user in the past is any one of boundary lines 501 to 505 is in the main storage device 315 or the auxiliary storage device 316. In the case where there is learning data and information related to the result of the inference based on the learning data, the arithmetic unit 311 causes the process to proceed to step S657. In the case where there is no learning data, the process proceeds to step S658.

[0118] In step S657, similar to step S647, the arithmetic unit 311 infers the boundary line among boundary lines 501 to 505 that the user is most likely to select based on the learning data of the boundary line selected by the user in the past, and reflects the inference on the above-mentioned category. In step S657, as described above, a flag can be attached to the category, the order can be determined by using metadata, the display of the boundary line can be emphasized, the category can be displayed by shading, or the display of the category that does not correspond to the candidate can be grayed out.

[0119] In step S658, the arithmetic unit 311 sends information related to the recognition result of the affected part 102 extracted, information related to the category corresponding to boundary lines 501 to 505, and information related to the category corresponding to the boundary line that the user is most likely to select to the communication device 313. Similar to step S648, the communication device 313 sends this information to the imaging device 200.

[0120] After step S658, the imaging device 200 performs the process of step S416. After the processes of steps S416 to S419, the imaging device 200 performs the process of step S620.

[0121] In step S620, the system control circuit 220 of the imaging device 200 sends information related to the boundary line specified by the user among boundary lines 501 to 505 to the communication device 219. The communication device 219 sends this information to the image processing device 300 via wireless communication. After step S620, the image processing device 300 performs the process of step S659.

[0122] In step S659, the arithmetic unit 311 of the image processing device 300 designates the category surrounded by the boundary line specified by the user among the recognized boundary lines 501 to 505 as the affected part 102. At the same time, the arithmetic unit 311 saves the learning data including information related to the category corresponding to the boundary line specified by the user as learning data for use in the inferences in steps S645 and S655 in the main storage device 315 or the auxiliary storage device 316.

[0123] After step S659, the flow returns to the process of step S620 performed by the imaging device 200. The imaging device 200 performs the process of step S621.

[0124] In step S621, the display device 223 of the camera device 200 displays the image data into which the affected part 102 is classified received in step S416 for a predetermined time. Figure 5 The image shown is displayed, and the process returns to step S402 after a predetermined time.

[0125] Thus, the affected part photographing device 1 according to the second embodiment learns based on the selection result of the user, and reflects the inference derived from the learning result in continuous photographing and evaluation. Therefore, the second embodiment can reduce the burden of medical staff and the burden of the patient to be evaluated to a greater extent than the first embodiment, and can more accurately evaluate the size of the affected part such as a pressure sore.

[0126] Third Embodiment

[0127] An affected part photographing apparatus 1 according to a third embodiment will now be described.

[0128] According to the third embodiment, the camera 200 captures the affected part 102 of the subject 101, obtains information at the time of release (for example, information about the touched position in the case of a release operation of a touch on a touch screen), and transmits the data thereof to the image processing device 300. The image processing device 300 extracts the affected part from the received image data, and identifies the most likely correct affected part and its shape by using the received information at the time of release.

[0129] The same structures and steps according to the third embodiment as those according to the above-described embodiments are designated by, for example, the same reference numerals as those in the description according to the above-described embodiments, and repeated description of the same structures and steps is appropriately omitted.

[0130] The following references Figure 1 , Figure 2 , Figure 3 , Figure 7A and Figure 7B The description includes a series of processes according to the third embodiment, for example, from imaging an affected part to generating and saving information used in evaluating the affected part.

[0131] The affected - part photographing device 1 according to the third embodiment obtains information related to the position or area indicated on the image by a user (i.e., the photographer) such as a medical staff member as information at the time of release in the imaging device 200, although details will be described later. The affected - part photographing device 1 identifies the affected part 102 and its outline based on the information related to the position or area. According to the third embodiment, the CPU 310 of the image - processing device 300 runs a program stored in the storage device 312 to perform various functions including processing for identifying a potentially correct area from candidates of the affected part 102 extracted from the auxiliary arithmetic unit 317. According to the third embodiment, the display device 223 of the imaging device 200 is a touch screen including a touch sensor that can obtain the touch position to obtain information related to the position or area indicated on the image by the user (photographer). Of course, the structure for obtaining information related to the indicated position or area is not limited to the touch screen, and other structures are acceptable as long as they can obtain information related to the position or area indicated by the user at the time of release.

[0132] Figure 7A and Figure 7B A flowchart showing the operation of the affected - part photographing device 1 according to the third embodiment. In Figure 7A and Figure 7B the processing of the steps on the left - hand side of the flowchart is performed by the imaging device 200, and the processing of the steps on the right - hand side of the flowchart is performed by the image - processing device 300. The processing of steps S401 - S406, steps S411 - S413, step S431, and step S441 is the same as the corresponding step processing in the above - mentioned Figure 4A and Figure 4B and the description of these processes is omitted.

[0133] According to the third embodiment, the imaging device 200 performs the processing of steps S401 - S406, and then the image - processing device 300 performs the processing of step S441.

[0134] In step S441, the communication device 313 of the image - processing device 300 receives the image data and one or more pieces of information including distance information sent from the communication device 219 of the imaging device 200.

[0135] In step S742, the CPU 310 and the auxiliary arithmetic unit 317 of the image - processing device 300 then extract the affected part 102 of the subject 101 from the image data received in step S441. Similar to the above - mentioned embodiment, the auxiliary arithmetic unit 317 extracts the affected part 102 by using semantic segmentation of deep learning. The semantic segmentation of deep learning is the same as the semantic segmentation described in the first embodiment, and the description of the semantic segmentation is omitted.

[0136] In many cases of semantic segmentation using deep learning, a combination of a region presumed to correspond to the affected part 102 and its reliability is extracted. The size of the displayed live view image is small, and thus the live view image is the "standard" at the time of shooting. In step S742, the auxiliary arithmetic unit 317 automatically selects a region candidate that is presumed to be most likely correct (i.e., has the highest reliability).

[0137] In step S743, the arithmetic unit 311 of the CPU 310 then calculates the area of the affected part 102, which is information related to the size of the affected part 102 extracted by the auxiliary arithmetic unit 317.

[0138] Figure 8 A method for calculating the area of the affected part 102 is shown. The imaging device 200 is a typical camera and can be treated as Figure 8 the pinhole model shown. Incident light 801 passes through the principal point of the lens 212a and is received by the imaging surface of the image sensor 214. It can be considered that in the case where the lens group 212 approximates a single lens 212a with no thickness, the two principal points, the front principal point and the rear principal point, are flush with each other. Adjusting the focal position of the lens 212a so that the subject image is formed on the plane of the image sensor 214 enables the imaging device 200 to focus on the subject 804. When the focal length F from the imaging surface to the principal point of the lens changes, the viewing angle θ changes and the zoom ratio changes. At this time, the width 806 of the subject 804 on the focal plane is geometrically determined based on the relationship between the viewing angle θ of the imaging device 200 and the subject distance 805. The width 806 of the subject 804 can be calculated using trigonometric functions. That is, the width 806 of the subject 804 is determined based on the relationship between the viewing angle θ that changes according to the focal length F and the subject distance 805. The value of the width 806 of the subject 804 is divided by the number of pixels in a row of the image data to obtain the length of the focal plane corresponding to one pixel of the image data.

[0139] Therefore, the arithmetic unit 311 calculates the area of the affected part 102 by multiplying the number of pixels in the extraction region obtained from the extraction result of the affected part in step S742 by the area of one pixel obtained from the length of the focal plane corresponding to one pixel of the image. The length of the focal plane corresponding to one pixel of the image can be obtained in advance for each combination of the focal length F and the subject distance 805 and can be prepared as table data. In this case, the storage device 312 of the image processing device 300 stores the table data according to the imaging device 200 in advance.

[0140] The area of the affected part 102 is correctly obtained in the above manner on the assumption that the subject 804 is planar and the plane is perpendicular to the optical axis of the lens. When the distance information obtained from the imaging device 200 in step S441 is the distance information or distance map at the position in the image data, the arithmetic unit 311 detects the change or inclination of the subject in the depth direction and can calculate the area based on the detected change or inclination.

[0141] In step S744, the arithmetic unit 311 then generates image data obtained by superimposing the information representing the result of extracting the affected part 102 and the information related to the size of the affected part 102 on the image data from which the affected part 102 is to be extracted.

[0142] Figure 9A and Figure 9B A method of superimposing the information related to the extraction result of the affected part 102 and the information related to the size of the affected part 102 on the image data is shown. Figure 9A The image 901 is shown by way of example using the image data before the superimposition process and includes the subject 101 and the affected part 102. Figure 9B The superimposed image 902 is based on the image data after the superimposition process. In this example, there is one candidate for the affected part 102. However, when there are candidate regions that may correspond to the affected part, the arithmetic unit 311 can perform the superimposition process on the candidate regions or can perform the superimposition process only on the candidate region that is most likely to correspond to the affected part.

[0143] As Figure 9B shown, in the upper left corner of the superimposed image 902, a marker 911 is superimposed in which a string 912 of the area value of the affected part 102 is displayed in white on a black background as the information related to the size of the affected part 102. The colors of the background and the string in the marker 911 are not limited to black and white as long as these colors are easily visible. The transparency of the colors of the background and the string in the marker 911 can be set to α blending so that the part where the marker is superimposed can be seen.

[0144] As Figure 9BAs shown, an index 913 representing the estimated area of the affected part 102 extracted in step S742 is superimposed on the superimposed image 902. That is to say, the superimposed image 902 is obtained by the following method: at the position of the area estimated to correspond to the affected part, the index 913 representing the estimated area and the image data used to create the image 901 are superimposed by using alpha blending. This enables the user to confirm whether the estimated area for obtaining the area of the affected part 102 is appropriate. In one embodiment, the color of the index 913 representing the estimated area does not overlap with the color of the subject. The range of the transparency value of alpha blending is such that the estimated area can be recognized and the original affected part 102 can also be seen. The superimposed display of the index 913 representing the estimated area of the affected part 102 enables the user to confirm whether the estimated area is appropriate without displaying the marker 911, and the process of step S743 can be omitted.

[0145] In step S745, the communication device 313 of the image processing device 300 then sends the information representing the extraction result of the affected part 102 extracted as described above and the information related to the size of the affected part 102 to the imaging device 200. That is to say, the communication device 313 sends the image data generated in step S744 and including the information related to the size of the affected part 102 to the imaging device 200 via wireless communication. After step S745, the imaging device 200 performs the process of step S707.

[0146] In step S707, when the image processing device 300 newly generates image data including the information related to the size of the affected part 102, the communication device 219 of the imaging device 200 receives the image data.

[0147] In step S708, the system control circuit 220 then determines whether image data including the information related to the size of the affected part 102 is received in step S707. If the image data is received, the system control circuit 220 performs the process of step S709. If the image data is not received, the process of step S710 is performed.

[0148] In step S709, the display device 223 displays the image data received in step S707 and including the information related to the size of the affected part 102 for a predetermined time. Here, the display device 223 displays Figure 9BThe superimposed image 902 shown. According to the third embodiment, information representing the extraction result of the affected part 102 is superimposed on the live view image and displayed. This enables the user to confirm whether the area and the estimated region of the affected part are appropriate by using the preview image before shooting. In the example according to the present embodiment, an index 913 representing the estimated region of the affected part 102 and information related to the size of the affected part 102 are displayed. However, either one of the two may be displayed. After step S709, the imaging device 200 performs the process of step S710.

[0149] In step S710, the system control circuit 220 determines whether a shooter such as a user has input a release operation by using the operation unit 224. When the system control circuit 220 determines that no release operation has been input, the process returns to the process of step S404 performed by the imaging device 200. When a release operation is input, the imaging device 200 performs the process of step S411.

[0150] According to the third embodiment, in Figure 10 and Figure 11 a release operation is shown. In the example according to the present embodiment, for simplicity of explanation, the operation unit 224 is a touch screen shared with the display device 223. However, the structure of the operation unit 224 is not limited thereto.

[0151] Figure 10 An example of the live view image displayed on the display device 223 including the operation unit 224 is shown, and an example is shown in which a shooter such as a user touches the touch screen at a desired position with, for example, a finger 1002 to perform a release. At this time, for example, the imaging device 200 displays a message or outputs a sound to request the shooter to touch any position in the area determined by itself to correspond to the affected part, thereby performing a release operation. In response to the touch operation of the shooter, the system control circuit 220 obtains information related to the touched position as touch position information 1001 at the time of release.

[0152] Figure 11 An example of the same live view image as Figure 10 is shown, and another example of a release operation different from the release operation in Figure 10 is shown. Figure 11 The difference from Figure 10 is that the release is performed when touching the touch screen. In Figure 11In the example, the photographer does not release finger 1002 during touching and moves the finger along the edge of the area determined by himself / herself to correspond to the affected part 102 to perform a sliding operation. At this time, for example, the imaging device 200 displays a message or outputs a sound to request the photographer to move finger 1002 along the edge of the area determined by himself / herself to correspond to the affected part 102, so as to perform a sliding operation. As Figure 11 As shown in the example of, when a closed shape is formed along the trajectory of the sliding operation on the outer edge of the area that may correspond to the affected part 102 by the photographer touching with finger 1002, the system control circuit 220 determines that an instruction to release is input and captures an image. The system control circuit 220 obtains information related to the position of the area of the closed shape surrounded by the trajectory of the sliding operation in which finger 102 moves while in the touched state as the touch area information 1101 at the time of release.

[0153] Return to Figure 7A and Figure 7B For the description of, the processing of steps S411 to S413 after step S710 is the same as that described above with reference to Figure 4A and Figure 4B and the detailed description of these processes is omitted. After step S413, the imaging device 200 performs the process of step S714.

[0154] In step S714, the system control circuit 220 obtains the image data after the resizing process generated in step S413, the distance information obtained in step S411, and the touch position information 1001 or touch area information 1101 obtained in step S710. The system control circuit 220 obtains information related to the zoom ratio and information related to the size (number of pixels) of the image data that has undergone the resizing process.

[0155] In step S715, the communication device 219 then wirelessly communicates the image data, distance information, and the information including the touch position information or touch area information obtained in step S714 to the image processing device 300.

[0156] After step S715, the image processing device 300 performs the processes of steps S751 to S756.

[0157] In the process of step S751, the communication device 313 of the image processing device 300 receives the image data, distance information, touch position information, or touch area information sent from the communication device 219 of the imaging device 200.

[0158] In step S752, the CPU 310 and the auxiliary arithmetic unit 317 of the image processing apparatus 300 then extract the affected part 102 of the subject 101 from the image data received in step S751. The affected part 102 is extracted by the auxiliary arithmetic unit 317 in the same manner as in the above-described step S742 (semantic segmentation by deep learning). In step S742, a region candidate that is automatically selected as being most likely to correspond to the affected part (with the highest reliability) is selected. However, here, by using a larger amount of image data and touch position information or touch area information, the region desired by the photographer is selected.

[0159] For example, in the case where touch position information is received, as described above, the photographer such as a user is requested to touch any position of the affected part 102. For this reason, it can be said that the probability that the touch position is within the affected part is high, and for a region having features that are substantially similar to the image features at the touch position, that is, a region where the features that are the same as or approximate to the image features at the touch position are continuous, the confidence that this region corresponds to the affected part is high. It can be said that for a region having features different from the image features at the touch position, that is, a region having features that are discontinuous from the image features at the touch position, the confidence that this region corresponds to the affected part is low. Therefore, the auxiliary arithmetic unit 317 excludes from the candidates for the affected part a region whose features are discontinuous and different from the image features at the touch position. Alternatively, the auxiliary arithmetic unit 317 decreases the reliability value indicating the confidence of a region whose features are discontinuous and different from the image features at the touch position. The auxiliary arithmetic unit 317 can increase the reliability value indicating the confidence of a region where the features that are the same as or approximate to the image features at the touch position are continuous. Of course, these examples do not limit the processing in a region where the touch position has a high priority in the candidate regions. After determining that the processing of excluding from the candidate regions or the processing of increasing or decreasing the reliability value, the auxiliary arithmetic unit 317 shows the region with the highest reliability in the candidate regions to the photographer.

[0160] For example, in the case where touch area information is received, as described above, the photographing requester is requested to move a finger along the edge of the area determined by himself / herself to correspond to the affected part. For this reason, it can be said that for the following area, the confidence level (high reliability) that the area corresponds to the affected part is high, the area has characteristics substantially similar to the characteristics of the area surrounded by the trajectory of the sliding operation in response to the request, and the overlapping part of the area with the area surrounded by the trajectory of the sliding operation is large. It can be said that for the following area, the confidence level that the area corresponds to the affected part is low, the area has characteristics substantially similar to the characteristics of the area surrounded by the trajectory of the sliding operation, and the overlapping part of the area with the area surrounded by the trajectory of the sliding operation is small. For this reason, the auxiliary arithmetic unit 317 obtains the degree of overlap between the area having characteristics substantially similar to the area surrounded by the trajectory of the sliding operation and the area surrounded by the trajectory of the sliding operation. The auxiliary arithmetic unit 317 excludes from the candidate areas of the affected part the areas that overlap with the area surrounded by the trajectory of the sliding operation by less than a threshold value. Alternatively, the auxiliary arithmetic unit 317 decreases the reliability value by a greater amount as the degree of overlap between the area and the area surrounded by the trajectory of the sliding operation decreases. The auxiliary arithmetic unit 317 may increase the reliability value by a greater amount as the degree of overlap between the area and the area surrounded by the trajectory of the sliding operation increases. As described above, the auxiliary arithmetic unit 317 determines to show the area with the highest reliability among the candidate areas after the process of excluding from the candidate areas or the process of increasing or decreasing the reliability value.

[0161] In step S753, the arithmetic unit 311 of the CPU 310 then calculates the area of the affected part 102 as an example of the information related to the size of the affected part 102 extracted in step S752. The process for calculating the area is the same as that in step S743 described above, and the description of this process is omitted.

[0162] In step S754, the arithmetic unit 311 performs image analysis based on the length of the focal plane corresponding to one pixel of the image obtained in step S753 to calculate the major axis length and minor axis length of the above-extracted affected part 102 and the area of the rectangle circumscribing the affected part 102. The size of the affected part, which is one of the indexes for evaluating the affected part in the above DESIGN-R, is defined as the product of the major axis length and the minor axis length. In the affected part photographing device 1 according to the present embodiment, the analysis of the major axis length and the minor axis length enables ensuring compatibility with the data measured by DESIGN-R. DESIGN-R is not strictly defined, and several methods for calculating the major axis length and the minor axis length can be considered mathematically.

[0163] In the example, the major axis length and the minor axis length are calculated as follows. The operation unit 311 calculates a rectangle with the smallest area (the minimum circumscribed rectangle) within the rectangle circumscribing the affected part 102. The operation unit 311 calculates the lengths of the long side and the short side of the rectangle, and determines that the major axis length is equal to the length of the long side and the minor axis length is equal to the length of the short side. The operation unit 311 calculates the area of the rectangle based on the length of the focal plane corresponding to one pixel of the image obtained in step S753.

[0164] In another example, the operation unit 311 may calculate the major axis length and the minor axis length such that the major axis length is equal to the maximum Feret diameter (i.e., the maximum caliper length), and the minor axis length is equal to the minimum Feret diameter. The operation unit 311 may determine that the major axis length is equal to the maximum Feret diameter (i.e., the maximum caliper length), and the minor axis length is equal to the length measured in the direction perpendicular to the axis having the maximum Feret diameter. The major axis length and the minor axis length may be calculated in a freely selectable manner based on compatibility with existing measurement results.

[0165] Regarding the image data received in step S441, the major axis length and the minor axis length of the affected part 102 and the area of the rectangle described above are not calculated. That is, a live view image is displayed for the user to view the extraction result of the affected part 102, and the processing of the image analysis step in step S754 is omitted to shorten the processing time.

[0166] In step S755, the operation unit 311 generates an image by superimposing information related to the extraction result of the region determined to be most likely corresponding to the affected part 102 in step S752 and information related to the size of the image data for extracting the affected part 102.

[0167] Figures 12A - 12C Shows a method of superimposing information representing the extraction result of the affected part 102 and information related to the size of the affected part including the major axis length and the minor axis length of the affected part 102 on the image data. It can be considered that there are multiple pieces of information related to the size of the affected part 102. Therefore, for Figure 12A the superimposed image 1201, Figure 12B the superimposed image 1202, and Figure 12C the superimposed image 1203, the description is divided.

[0168] Figure 12A The superimposed image 1201 is an example in the case of using the minimum circumscribed rectangle as the method for calculating the major axis length and the minor axis length. For Figure 9BSimilar to the example, in the upper left corner of the superimposed image 1201, a marker 911 is superimposed, which shows the string of the area value of the affected part 102 in white on a black background, as information related to the size of the affected part 102. In the upper right corner of the superimposed image 1201, a marker 1212 that shows the major axis length and minor axis length calculated based on the minimum bounding rectangle is superimposed as information related to the size of the affected part 102 and is displayed. The string 1213 represents the major axis length (unit: centimeter), and the string 1214 represents the minor axis length (unit: centimeter). In the superimposed image 1201, a rectangular frame 1215 representing the minimum bounding rectangle is displayed in the affected part 102. The rectangular frame 1215 is superimposed together with the major axis length and minor axis length, and the user can view which part of the measurement image has the length.

[0169] A scale 1216 is superimposed in the lower right corner of the superimposed image 1201. The scale 1216 is used to measure the size as information related to the size of the affected part 102. The size of the scale for the image data changes according to the distance information. Specifically, the scale 1216 has graduations in units of 1 centimeter to 5 centimeters, based on the length of the focal plane corresponding to one pixel of the image obtained in step S753, and corresponds to the size of the focal plane (i.e., the subject) of the imaging device. The user can roughly grasp the size of the subject 101 or the affected part 102 by referring to the scale.

[0170] In the lower left corner of the superimposed image 1201, the above-mentioned index for evaluating size in DESIGN-R is superimposed and displayed as a marker 1217. The index for evaluating size in DESIGN-R is classified into the above seven stages according to the value (unit: centimeter) of the product of the major axis length and minor axis length (the maximum length on the axis perpendicular to the major axis) measured within the range of the skin lesion. According to this embodiment, the index obtained by converting the major axis length and minor axis length into values output by various calculation methods is superimposed and displayed as the marker 1217.

[0171] In Figure 12B In the superimposed image 1202, the major axis length is the maximum Feret diameter, and the minor axis length is the minimum Feret diameter. In the upper right corner of the superimposed image 1202, a marker 1222 that shows the string 1223 of the major axis length and the string 1224 of the minor axis length is superimposed. In the affected part 102 of the superimposed image 1202, a reference line 1225 corresponding to the position where the maximum Feret diameter is measured and a reference line 1226 corresponding to the minimum Feret diameter are displayed. As described above, the reference lines are superimposed together with the major axis length and minor axis length, which enables the user to view which part of the measurement image has the length.

[0172] In Figure 12CIn the superimposed image 1203, the minor axis length is measured in a direction perpendicular to the axis of the maximum Feret diameter rather than the minimum Feret diameter, although the major axis length is the same as that in the superimposed image 1202. In the upper right of the superimposed image 1203, a marker 1232 is superimposed, which shows a string 1223 indicating the major axis length and a string 1234 indicating the minor axis length. In the affected part 102 in the superimposed image 1203, a reference line 1225 corresponding to the position where the maximum Feret diameter is measured and a reference line 1236 corresponding to the length measured in the direction perpendicular to the axis of the maximum Feret diameter are shown.

[0173] In the information superimposed on Figures 12A - 12C the image data shown, any one or a combination of these pieces of information is sufficient. The user can select the information to be displayed. In Figure 9A , Figure 9B and Figures 12A - 12C the superimposed images are shown by way of example. Therefore, the display form, display position, size, font, font size, font color, and positional relationship, etc. of the affected part 102 and the information related to the size of the affected part 102 can be changed according to various conditions.

[0174] Returning to Figure 7A and Figure 7B the description, in step S756, the communication device 313 of the image processing device 300 sends the information indicating the extraction result of the extracted affected part 102 and the information related to the size of the affected part 102 to the imaging device 200. According to this embodiment, the communication device 313 sends the image data including the information related to the size of the affected part 102 generated in step S755 to the imaging device 200 via wireless communication.

[0175] After step S756, the imaging device 200 performs the process of step S716.

[0176] In the process of step S716, the communication device 219 of the imaging device 200 receives the image data including the information related to the size of the affected part 102 generated by the image processing device 300.

[0177] In step S717, the display device 223 then displays the image data including the information related to the size of the affected part 102 received in step S716 for a predetermined time. At this time, the display device 223 displays Figures 12A - 12C any one of the superimposed images 1201 to 1203 shown. After the predetermined time, the process returns to the process of step S402 performed by the imaging device 200.

[0178] According to the third embodiment, as described above, information related to the position indicated by the user (photographer) on the image is obtained at the time of release in the imaging device 200. Based on the information related to the indicated position, the affected part 102 and the outer edge are estimated. That is, according to the third embodiment, the affected part 102 and the outer edge are estimated based on the above-mentioned information related to the position specified by the user, and the area closest to the affected part expected by the user can be identified. The display device 223 of the imaging device 200 displays information related to the size of the affected part. As a result, according to the third embodiment, when evaluating the size of the affected part 102, the burden on the user and the burden on the patient to be evaluated are reduced. According to the third embodiment, similar to the above embodiment, the size of the affected part is calculated based on a program. Therefore, the difference between individuals can be smaller than the difference between individuals in the case of manual measurement by the user, and the evaluation of the size of the affected part 102 can be more accurate.

[0179] A function that enables the user to confirm whether the estimated area of the affected part is appropriate when displaying the live view is not necessary, and the processing of steps S441 to S745 can be omitted. The image processing device 300 can store the information indicating the extraction result of the affected part 102, the information related to the size of the affected part 102, and the image data of the superimposed image on which this information is superimposed in the storage device 312. The output device 314 can output one or more pieces of information or image data stored in the storage device 312 to an output device such as a display connected thereto.

[0180] Fourth Embodiment

[0181] In the example according to the above-mentioned third embodiment, the image processing device 300 superimposes the information indicating the extraction result of the affected part and the information related to the size of the affected part on the image for display. However, in the example according to the fourth embodiment described below, the image processing circuit 217 of the imaging device 200 performs the process of superimposing the information indicating the extraction result of the affected part 102 and the information related to the size of the affected part on the image for display.

[0182] Figure 13A and Figure 13B A flowchart showing the operation of the affected part photographing device 1 according to the fourth embodiment. In Figure 13A and Figure 13B , the processing of the steps on the left side of the flowchart is performed by the imaging device 200, and the processing of the steps on the right side of the flowchart is performed by the image processing device 300. In Figure 13A and Figure 13B of the flowchart, the superimposing process performed by the image processing device 300 in steps S744 and S755 is removed from Figure 7A and Figure 7Bis removed during the processing, and instead, the superimposing process performed by the imaging device 200 in step S1301 and step S1302 is added. Instead of the process of transmission performed by the image processing device 300 in step S745 and step S756, the process of transmission in step S1311 and step S1312 is added. In Figure 13A and Figure 13B In the steps shown, at the steps specified by the same reference numerals as those of the steps shown in Figure 7A and Figure 7B the same processing as that in the corresponding steps in Figure 7A and Figure 7B is performed.

[0183] According to the fourth embodiment, the imaging device 200 generates a superimposed image, and the data sent from the image processing device 300 to the imaging device 200 in step S1311 and step S1312 does not need to be color bar image data. According to the fourth embodiment, the image processing device 300 does not send image data, but instead sends metadata related to the size of the presumed affected part and data indicating the position of the affected part. For this reason, according to the fourth embodiment, the communication traffic is reduced and the communication speed is increased. The data indicating the position of the presumed affected part is in a reduced-size vector form. The data indicating the position of the presumed affected part may be in a binary raster form.

[0184] In step S707, the imaging device 200 receives metadata related to the size of the presumed affected part and data indicating the position of the affected part from the image processing device 300. In step S1301, the imaging device 200 generates a superimposed image. Specifically, in step S1301, the image processing circuit 217 of the imaging device 200 generates an overlapping image as described in step S744 with reference to Figure 9B The image data superimposed with the information indicating the size and position of the presumed affected part may be the image data sent from the imaging device 200 to the image processing device 300 in step S406, or may be the latest image data used as a live view image for display. Subsequently, the imaging device 200 performs the process of step S709.

[0185] In step S716, the imaging device 200 receives metadata related to the size of the presumed affected part and data indicating the position of the affected part from the image processing device 300. In step S1302, the imaging device 200 generates a superimposed image. Specifically, in step S1302, the image processing circuit 217 of the imaging device 200 generates an overlapping image as described in step S745 with reference to Figure 9BThe overlapping image is generated. The image data overlaid with the information indicating the size and position of the presumed affected part is the image data transmitted from the imaging device 200 to the image processing device 300 in step S715. Subsequently, the imaging device 200 performs the process of step S717.

[0186] According to the fourth embodiment, compared with the third embodiment, the data transmitted from the image processing device 300 to the imaging device 200 can be reduced to a greater extent, the communication traffic between the imaging device 200 and the image processing device 300 can be reduced, and the communication speed can be increased.

[0187] Fifth Embodiment

[0188] Figure 14 The affected part photographing device 11 according to the fifth embodiment is schematically shown. The affected part photographing device 11 according to the fifth embodiment includes, in addition to the imaging device 200 and the image processing device 300 that are the same as those of the imaging device 200 and the image processing device 300 according to the above embodiments, a terminal device 1400 that can access the Web. For example, the terminal device 1400 includes a tablet terminal, has the function of a Web browser, and can access a Web server and display the obtained HTML file. The terminal device 1400 is not limited to a tablet terminal, as long as the terminal device 1400 can display a Web browser and can be, for example, a smart phone or a personal computer.

[0189] In addition to performing the processes according to the third and fourth embodiments, the CPU 310 of the image processing device 300 also performs a process for identifying a subject from the image data. The CPU 310 performs a process for storing, in the storage device 312, the information related to the size and position of the presumed affected part and the image data of the affected part 102 that are associated with each other for each identified subject. The terminal device 1400 enables the use of a Web browser to view the information related to the size of the presumed affected part and the image data of the affected part stored in the storage device 312 of the image processing device 300 and associated with the subject 101.

[0190] The function of identifying a subject from the image data, the function of storing information or image data related to the affected part for each identified subject, or the function of performing Web service processing is performed by the image processing device 300 according to the present embodiment. However, this is not a limitation. At least one of these functions can be performed by hardware other than the image processing device 300.

[0191] According to the fifth embodiment, as Figure 14As shown, a barcode label 103 is attached to the subject 101 as information for identifying the subject, and the image data of the affected part 102 captured and the ID of the subject represented by the barcode label 103 are associated with each other. The label for identifying the subject is not limited to a one-dimensional barcode label, and may be a two-dimensional code such as a QR code (QR code, registered trademark) or a numerical value. The label for identifying the subject may be a label with text written on it, and the image processing device 300 may read the text of the label by using its OCR (Optical Character Recognition / Reader) function.

[0192] The CPU 310 of the image processing device 300 compares the ID obtained by analyzing the barcode label included in the data of the captured image with the subject ID pre-registered in the storage device 312 to obtain the name of the subject 101. The imaging device 200 may analyze the ID and send the obtained ID to the image processing device 300.

[0193] The CPU 310 generates a record based on the image data of the affected part 102, information related to the size of the affected part 102, the subject ID, the obtained name of the subject, and the time and date of the capture, and registers these records in the database of the storage device 312.

[0194] The CPU 310 sends the information registered in the database of the storage device 312 in response to a request from the terminal device 1400.

[0195] Figure 15A and Figure 15B FIG. shows a flowchart of the operation of the affected part photographing device 11 according to the fifth embodiment. In Figure 15A and Figure 15B , the processing of the steps on the left side of the flow is performed by the imaging device 200, and the processing of the steps on the right side of the flow is performed by the image processing device 300. In Figure 15A and Figure 15B the steps shown, at the steps designated by the same reference numerals as those in the steps shown in Figure 7A and Figure 7B , the same processing as the corresponding steps in Figure 7A and Figure 7B is performed.

[0196] In step S1501, the imaging device 200 connected to the image processing device 300 in steps S401 and S431 causes the display device 223 to display a message for instructing a photographer such as a user to photograph the barcode label 103, for example. The imaging device 200 captures the barcode label 103 in response to the release operation of the user. After step S1501, the imaging device 200 performs the processing of step S402.

[0197] The barcode label 103 includes information related to the patient ID for identifying the patient as the subject ID for identifying the subject 101 described above. According to the fifth embodiment, the imaging device 200 captures the barcode label 103 and then captures the affected part 102. The affected part imaging device 11 according to the fifth embodiment manages the imaging order by using, for example, the imaging time and date, and identifies the image of the subject located between the image of the barcode label and the image of the next barcode label by using the subject ID. Of course, the order may be to capture the barcode label 103 after capturing the affected part 102.

[0198] After performing the processes of steps S402 to S715, the communication device 219 transmits one or more pieces of information including the image data and the distance information to the image processing device 300 via wireless communication. According to the fifth embodiment, the image data transmitted in step S715 includes the image data of the captured affected part 102 and the image data of the barcode label 103 captured in step S1501.

[0199] According to the fifth embodiment, after performing the processes of steps S751 to S756 as described above, the image processing device 300 performs the process of step S1502.

[0200] In step S1502, the CPU 310 performs the following process: reads the subject ID (i.e., the patient ID) for identifying the subject 101 from the image data of the barcode label included in the image data captured by the imaging device 200 in step S1501.

[0201] In step S1503, the CPU 310 then compares the read subject ID with the ID of the subject pre-registered in the database of the storage device 312 to obtain the name of the subject.

[0202] In step S1504, the CPU 310 then creates records based on the image data of the captured affected part 102, the information related to the size of the affected part 102, the subject ID, the obtained name of the subject, and the imaging time and date, etc., and registers these records in the database of the storage device 312. In step S1505, the CPU 310 then transmits the information registered in the database of the storage device 312 to the terminal device 1400 in response to a request from the terminal device 1400.

[0203] Figure 16 and Figure 17 Shows an example of the display of the browser in the terminal device 1400. Figure 16An example of a data selection screen displayed on the terminal device 1400 is shown. The data selection screen 1601 is partitioned for each date 1602 by partition lines 1603. In the date area, icons 1605 are displayed for each shooting time 1604. For the icons 1605, the subject ID and the name of the subject are displayed. Each icon 1605 represents a data set of the same subject shot in the same time zone. The data selection screen 1601 includes a search window 1606. When the user inputs, for example, a date, a subject ID, or the name of a subject in the search window 1606, the terminal device 1400 searches for the data set based on the input information. When the user operates, for example, the scroll bar 1607, the terminal device 1400 displays enlarged data in a restricted display area in response to the operation. When the user selects and clicks any one of the icons 1605, for example, the terminal device 1400 displays a data viewing screen of a browser and allows the user to view the image of the data set and the information related to the size of the subject. The terminal device 1400 sends a request indicating, for example, the subject, time, and date specified by the user to the image processing device 300. The image processing device 300 sends the image data and the information related to the size of the subject to the terminal device 1400 in response to the request from the terminal device 1400.

[0204] Figure 17 An example of a data list screen displayed on the browser of the terminal device 1400 is shown. On the data viewing screen 1701, Figure 16 the subject ID, the name 1702 of the subject, the shooting time and date 1703 of the data set selected on the data selection screen 1601 are displayed. On the data viewing screen 1701, an image 1704 based on the image data at the time of one shooting and data 1705 including information related to the size and position of the affected part in the image 1704 are also displayed. On the data viewing screen 1701, the shooting number 1706 in the case where the affected part of the same subject is continuously shot multiple times is also displayed.

[0205] With these configurations, in addition to the effects of the affected part photographing device 1 according to the third embodiment, the affected part photographing device 11 according to the fifth embodiment can identify and store the image data of the affected part 102 and its analysis result for each subject. The user can view the analysis result of the image data of the affected part 102 in association with the subject ID and the name of the subject by using a Web browser of a terminal device 1400 such as a tablet terminal.

[0206] Other Embodiments

[0207] In an example according to the above-described embodiment, an imaging is performed on an injury area such as a pressure ulcer occurring in the surface layer of the skin on the body surface of a human or an animal. However, the imaging device may be, for example, an endoscope camera, and the object to be imaged by the endoscope camera may be, for example, an injury area in the surface layer of the gastric wall. In addition to these, the method according to the above-described embodiment can be used in a case where a damaged area such as peeling in the surface layer of the wall surface of a building is identified as a specific area.

[0208] Aspects of the embodiment can also be performed in the following manner: A program for performing one or more functions according to the above-described embodiment is provided to a system or a device via a network or a storage medium, and one or more processors of a computer of the system or the device read and execute the program. Aspects of the embodiment can also be executed by a circuit (e.g., ASIC) for performing one or more functions.

[0209] The above-described embodiment specifically illustrates, by way of example, to implement aspects of the embodiment. Due to the above-described embodiment, the technical scope of the present invention should not be interpreted within a limited range. That is to say, without departing from the technical concept or main features of the present invention, the present invention can be implemented as various embodiments.

[0210] Other Embodiments

[0211] Embodiments of the present invention can also be implemented by the following method, that is, a software (program) for executing the functions of the above-described embodiment is provided to a system or a device via a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or the device reads and executes the program.

[0212] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: at least one processor configured to operate as the following units: a generation unit configured to identify at least an outer edge of a specific area in a surface layer of the subject from an image of the subject, and generate an outer edge candidate; and a control unit configured to select an outer edge candidate from the generated outer edge candidates based on an instruction from a user, wherein the generation unit identifies the specific area with respect to a first image and generates an outer edge candidate, where the first image is an image of the subject captured based on a position specified by the user on an image showing the outer edge by using an operation unit, wherein the generation unit excludes, from candidate areas to be identified as the specific area, candidate areas having features different from those of the image at the specified position among the candidate areas presumed to correspond to the specific area based on the specified position, and wherein the control unit selects an outer edge candidate from the generated outer edge candidates based on the specified position.

2. The apparatus according to claim 1, Among them, wherein the control unit stores information related to the shape of the selected outer edge candidate and an image of the specific area according to the selected outer edge candidate in a storage medium.

3. The apparatus according to claim 1, Among them, wherein the generation unit identifies a condition area included in the specific area, and generates an outer edge candidate for each condition area.

4. The apparatus according to claim 3, Among them, wherein the control unit causes a display device to display an image obtained by superimposing the outer edge candidates for each condition area on the image of the specific area, and selects an outer edge candidate from the obtained image based on an instruction from a user.

5. The apparatus according to claim 4, Among them, wherein the control unit causes the display device to display an image obtained by superimposing at least one of the outer edge candidates on the specific area.

6. The apparatus according to claim 5, Among them, wherein the control unit causes the display device to display an image obtained by superimposing the outer edge candidate corresponding to at least one condition area among the outer edge candidates for each condition area on the specific area.

7. The apparatus according to claim 3, Among them, wherein the generation unit classifies the specific area for each condition area into categories, and wherein the control unit selects an outer edge candidate corresponding to the category indicated by the user.

8. The apparatus according to claim 1, Among them, wherein the generation unit identifies the outer edge of the specific area based on learning data when learning the selection result of the outer edge candidate.

9. The apparatus according to claim 1, Among them, wherein the generation unit identifies the specific area from an area where features identical to those of the image at the specified position are continuous.

10. The apparatus according to claim 1, Among them, wherein the generation unit identifies a first candidate area among the remaining candidate areas other than the excluded candidate areas as the specific area, where the reliability of the confidence corresponding to the specific area of the first candidate area is the highest.

11. The apparatus according to claim 1, Among them, The generating unit selects, from candidate regions presumed to correspond to the specific region based on the specified position, a candidate region having features different from those of the image at the specified position, and reduces the reliability of the confidence that the candidate region corresponds to the specific region.

12. The apparatus according to claim 1, Among them, wherein the generating unit identifies the specific region with respect to a second image and generates outer-edge candidates, the second image being an image of the subject captured based on a region specified by a user, and wherein the control unit selects an outer-edge candidate from the generated outer-edge candidates based on the specified region.

13. The apparatus according to claim 12, Among them, wherein the generating unit identifies the specific region based on a region specified by a user by defining a closed curve.

14. The apparatus according to claim 13, Among them, wherein the generating unit identifies the specific region based on the size of a portion overlapping with the region where the user defines the closed curve.

15. The apparatus according to claim 13, Among them, wherein the generating unit excludes, from candidate regions to be identified as the specific region, candidate regions having a portion that overlaps with the region where the user defines the closed curve and has a size smaller than a threshold value, from among candidate regions presumed to correspond to the specific region based on the region where the user defines the closed curve.

16. The apparatus according to claim 15, Among them, wherein the generating unit identifies, as the specific region, a candidate region having a portion that overlaps with the region where the user defines the closed curve and has the largest size, from among the remaining candidate regions other than the excluded candidate regions.

17. The apparatus according to claim 13, Among them, wherein the generating unit reduces the reliability of the confidence that a candidate region presumed to correspond to the specific region based on the region where the user defines the closed curve corresponds to the specific region, and the reduction amount increases as the size of the portion overlapping with the defined region decreases.

18. The apparatus according to claim 1, Among them, wherein the control unit causes a display device to display a preview image in which the outer-edge candidate is superimposed on the specific region, and wherein the generating unit identifies the specific region with respect to a third image and generates outer-edge candidates, the third image being a still image obtained when the preview image is displayed.

19. The apparatus according to claim 1, Among them, wherein the control unit obtains information to be used in the evaluation of the specific region based on information related to the shape of the selected outer-edge candidate.

20. The apparatus according to claim 19, Among them, wherein the specific region is a damaged region in the surface layer of the subject, and wherein the control unit obtains information to be used in the evaluation of the size of the shape of the damaged region.

21. The apparatus according to claim 20, Among them, wherein the damaged region is an affected part of a damage occurring in the surface layer of the skin.

22. An image processing method, comprising: generating outer-edge candidates by at least identifying an outer edge of a specific region in the surface layer of a subject from an image of the subject; and Select an outer edge candidate from the generated outer edge candidates based on an instruction from the user. In the generation, a specific region is identified with respect to a first image and outer edge candidates are generated, where the first image is an image of the subject captured based on a position specified by the user on an image showing the outer edge by using an operation unit. In the generation, from candidate regions to be recognized as the specific region, candidate regions having features different from those of the image at the specified position among the candidate regions presumed to correspond to the specific region based on the specified position are excluded, and In the selection, an outer edge candidate is selected from the generated outer edge candidates based on the specified position.

23. A non-transitory computer-readable storage medium storing an instruction program for causing a computer to perform a method, the method including: Generating outer edge candidates by at least identifying an outer edge of a specific region in a surface layer of a subject; And Selecting an outer edge candidate from the generated outer edge candidates based on an instruction from the user, In the generation, a specific region is identified with respect to a first image and outer edge candidates are generated, where the first image is an image of the subject captured based on a position specified by the user on an image showing the outer edge by using an operation unit. In the generation, from candidate regions to be recognized as the specific region, candidate regions having features different from those of the image at the specified position among the candidate regions presumed to correspond to the specific region based on the specified position are excluded, and In the selection, an outer edge candidate is selected from the generated outer edge candidates based on the specified position.

Images