Image processing system, camera device, image processing device, electronic device, control method thereof, and storage medium storing control method

Abstract

An image processing system is provided for improving user-friendliness in evaluation of an affected area. The image processing system includes an image pickup device that receives light from a subject to generate image data and outputs the generated image data to a communication network, and an image processing device that acquires the image data via the communication network, extracts a specific area of ​​the subject from the acquired image data, and outputs information indicating an extraction result of the extracted specific area to the communication network. A display unit in the image pickup device performs display based on the information indicating the extraction result of the specific area acquired via the communication network.

Description

Technical Field

[0001] The present invention relates to a technique for evaluating a specific area of ​​a subject from an image. Background Art

[0002] When a person or animal is lying down, the contact area between the body and the floor, mat or cushion beneath the body is compressed by the body weight.

[0003] If the same posture is continued, vascular insufficiency occurs in the contact area between the floor and the body, resulting in necrosis of the surrounding tissue. The state in which tissue necrosis occurs is called a bedsore or ulcer. It is necessary to give bedsore care, such as body pressure distribution and skin care, to patients who experience bedsores to regularly evaluate and manage bedsores.

[0004] Measurement of bedsore size is known as one method of evaluating bedsores.

[0005] For example, as described in Non-Patent Document 1, DESIGN-R (registered trademark), which is an evaluation index of bedsores developed by the Academic Education Committee of the Japan Bedsore Society, is well known as an example of using the size of bedsores in evaluation.

[0006] DESIGN-R (registered trademark) is a tool used to evaluate the healing process of wounds (such as bedsores). The tool is named by the initials of the following evaluation items: Depth, Exudate, Size, Inflammation / infection, Granulation, and Necrotic tissue. In addition to the above evaluation items, the evaluation items also include capsules, but the initials of the capsules are not used in the name.

[0007] DESIGN-R (registered trademark) is classified into two groups, one for classification of severity levels for daily and simple evaluation, and the other for process evaluation indicating the flow of the healing process in detail. In DESIGN-R (registered trademark) for classification of severity levels, six evaluation items are classified into two types: mild and severe. Mild evaluation items are indicated using lowercase letters, and severe evaluation items are indicated using uppercase letters.

[0008] In the initial treatment, evaluation using DESIGN-R (registered trademark) for classification of severity levels enables grasping the rough state of bedsores. Since problematic items are revealed, a treatment strategy can be easily determined.

[0009] DESIGN-R (registered trademark) which can compare the severity level between patients in addition to the process evaluation is also defined as DESIGN-R (registered trademark) for process evaluation. Here, R represents rating (evaluation and rating). The sum of the weights of six items (0 point to 66 points) in which different weights are added to the corresponding items and the depth is excluded represents the severity level of bedsores. With DESIGN-R (registered trademark), the course of treatment can be objectively evaluated in detail after the start of treatment, so that in addition to evaluating the individual course of treatment, the severity level between patients can be compared.

[0010] In the evaluation of the size of DESIGN-R (registered trademark), the major axis length (cm) and the minor axis length (maximum diameter orthogonal to the major axis length) (cm) of the skin damage range are measured and the size, which is a numerical value obtained by multiplying the major axis length by the minor axis length, is classified into seven levels. The seven levels include s0: no skin damage, s3: less than four, s6: not less than four and less than 16, s8: not less than 16 and less than 36, s9: not less than 36 and less than 64, s12: not less than 64 and less than 100, and s15: not less than 100.

[0011] Currently, the evaluation of the size of bedsores is often based on a value obtained by manual measurement using a measured affected area. Specifically, the maximum straight-line distance between two points within the range of skin damage is measured, and the measured distance is used as the major axis length. The length orthogonal to the major axis length is used as the minor axis length, and the value obtained by multiplying the major axis length by the minor axis length is set as the size of bedsores.

[0012] Prior art literature

[0013] Non-patent literature

[0014] Non-patent document 1: "Pressure Sore Prevention and Management Guide JSPU (4th Edition)" by Terunoshisha (edited by the Japanese Pressure Sore Society, International Standard Book Number (ISBN)-13 978-4796523608), page 23, "Pressure Sore Guide (2nd Edition)"

[0015] However, bedsores generally have a complex shape, and it is necessary to adjust the use of measurement when manually evaluating the size of bedsores. Since the above work needs to be performed at least twice to measure the major axis length and the minor axis length, it takes time and requires heavy workload. In addition, since the patient whose bedsore is being evaluated needs to maintain the same posture during the work, it is considered that manually evaluating the size of bedsores imposes a heavy burden on the patient.

[0016] It is recommended to perform rating with DESIGN-R (registered trademark) once a week or every two weeks, and it is necessary to repeat the measurement. In addition, the position of the long axis length judged as a bedsore may vary depending on the individual in manual measurement, and it is difficult to ensure the accuracy of measurement.

[0017] Although the above describes an example of bedsore evaluation based on DESIGN-R (registered trademark), the above description is not limited to the case of DESIGN-R (registered trademark), and the same problem occurs regardless of the method of measuring the size of bedsores. Manual measurements at multiple locations are required to calculate the area of ​​bedsores, which results in a lot of work.

[0018] As another problem, the evaluation items of bedsores include evaluation items that are expected to be visually judged in addition to the evaluation items of the measured size. The evaluation items that should be visually judged are then input into an electronic health record or a paper medium by an evaluator while viewing the captured image data. In this case, since the input device used for the information indicating the size is different from the input device used for the other information, the input operation is complicated and omission may occur.

[0019] These problems are not limited to bedsores, and the same problems may occur with affected areas on the body surface, such as burns or lacerations. Summary of the invention

[0020] An image processing system according to one aspect of the present invention comprises: an imaging device and an image processing device, characterized in that the imaging device comprises: an imaging component for receiving light from a subject to generate image data, a first communication component for outputting the image data to a communication network, and a display component for displaying an image based on the image data generated by the imaging component, and the image processing device comprises: a second communication component for acquiring the image data via the communication network, and an operation component for extracting a specific area of ​​the subject from the image data, the second communication component outputs information indicating an extraction result of the specific area extracted by the operation component to the communication network, the first communication component acquires information indicating an extraction result of the specific area via the communication network, and the display component performs a display based on the information indicating the extraction result of the specific area.

[0021] Another aspect of the present invention is an imaging device, comprising: an imaging component for receiving light from a subject to generate image data; a communication component for outputting the image data to an external device via a communication network; and a display component for displaying an image based on the image data generated by the imaging component, characterized in that the communication component obtains information indicating an extraction result of a specific area of ​​the subject in the image data from the external device via the communication network, and the display component performs display based on the information indicating the extraction result of the specific area.

[0022] Another aspect of the present invention is an image processing device, comprising: a communication component for acquiring image data and distance information corresponding to a subject included in the image data from a camera device via a communication network; and a calculation component for extracting a specific area of ​​the subject from the image data and calculating the size of the specific area based on the distance information, characterized in that the communication component outputs information indicating an extraction result of the specific area extracted by the calculation component and information indicating the size to the camera device via the communication network.

[0023] Another aspect of the present invention is a camera device, comprising: a camera component for receiving light from a subject of the camera device to generate image data; a control component for obtaining an extraction result of a specific area of ​​the subject in the image data; and an interface component for allowing a user to input evaluation values ​​of a predetermined plurality of evaluation items in the specific area of ​​the subject, characterized in that the control component associates the input evaluation values ​​of the plurality of evaluation items with the image data.

[0024] Another aspect of the electronic device of the present invention is characterized in that it includes: a communication component for acquiring, via a communication network, image data generated by a camera device and information indicating evaluation values ​​of multiple evaluation items for an affected area of ​​a subject in the image data input by a user using the camera device; and a control component for causing a display component to display an image based on the image data and the evaluation values ​​of the multiple evaluation items. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram schematically showing an image processing system according to the first embodiment.

[0026] Figure 2 is a diagram showing an example of a hardware configuration of an imaging apparatus included in an image processing system.

[0027] Figure 3 is a diagram showing an example of a hardware configuration of an image processing apparatus included in an image processing system.

[0028] Figure 4is a workflow chart showing the operation of the image processing system according to the first embodiment.

[0029] Figure 5 is a diagram used to describe how to calculate the area of ​​a region.

[0030] Fig. 6A A diagram for describing image data including an affected area.

[0031] Figure 6B It is a diagram for describing how to superimpose information indicating the extraction result of the affected area and information indicating the size of the affected area on image data.

[0032] Fig. 7A It is a diagram for describing a method of superimposing information indicating an extraction result of an affected area and information including a major axis length and a minor axis length of the affected area and indicating a size of the affected area on image data.

[0033] Figure 7B It is a diagram for describing another method of superimposing information indicating the extraction result of the affected area and information including the major axis length and the minor axis length of the affected area and indicating the size of the affected area on image data.

[0034] Figure 7C It is a diagram for describing another method of superimposing information indicating the extraction result of the affected area and information including the major axis length and the minor axis length of the affected area and indicating the size of the affected area on image data.

[0035] Fig. 8A This is a diagram for describing a method of causing a user to input information related to a location of an affected area.

[0036] Figure 8B This is a diagram for describing a method of causing a user to input information related to a location of an affected area.

[0037] Figure 8C This is a diagram for describing a method of causing a user to input information related to an evaluation value of an affected area.

[0038] Fig.8D This is a diagram for describing a method of causing a user to input information related to an evaluation value of an affected area.

[0039] Fig. 8E This is a diagram for describing a method of causing a user to input information related to an evaluation value of an affected area.

[0040] Fig.8F This is a diagram for describing another method of causing a user to input information related to a location of an affected area.

[0041] Figure 8GThis is a diagram for describing another method of causing a user to input information related to a location of an affected area.

[0042] Fig. 9 is a flowchart showing the operation of the image processing system according to the second embodiment.

[0043] Fig.10 is a diagram schematically showing an image processing system according to a third embodiment.

[0044] Fig.11 is a workflow chart showing the operation of the image processing system according to the third embodiment.

[0045] Fig. 12A This is a diagram for describing a method of displaying information related to a part of an affected area for which an evaluation value has been acquired.

[0046] Fig. 12B This is a diagram for describing a method of displaying information related to the acquired evaluation value of the affected area.

[0047] Fig.13 is a diagram for describing an example of a data selection window displayed in a browser of a terminal device.

[0048] Fig.14 It is a diagram for describing an example of a data browsing window displayed in a browser of a terminal device.

[0049] Fig.15 is a flowchart showing a modification of the operation of the image processing system according to the third embodiment. DETAILED DESCRIPTION

[0050] An object of an embodiment is to improve the user-friendliness of the evaluation of a specific area of ​​a subject.

[0051] Here, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0052] (First embodiment)

[0053] Now refer to Figures 1 to 3 An image processing system according to an embodiment of the present invention is described. Figure 1 1 is a diagram schematically showing an image processing system 1 according to the first embodiment. The imaging system 1 is composed of an imaging apparatus 200 as a portable handheld device, and an image processing apparatus 300. In this embodiment, an example of a clinical condition of an affected area 102 of a subject 101 is described as a bedsore on the buttocks.

[0054] In the image processing system 1 according to the embodiment of the present invention, the imaging device 200 captures the affected area 102 of the subject 101, acquires the subject distance, and transmits the data to the image processing device 300. The image processing device 300 extracts the affected area from the received image data, measures the area of ​​each pixel of the image data based on the information including the subject distance, and measures the area of ​​the affected area 102 based on the extraction result of the affected area 102 and the area of ​​each pixel. Although an example in which the affected area 102 is a bedsore is described in the present embodiment, the affected area 102 is not limited thereto and may be a burn or a laceration.

[0055] Figure 2 2 is an example showing a hardware configuration of the imaging apparatus 200 included in the image processing system 1. For example, a general single-lens camera, a compact digital camera, or a smartphone or a tablet computer provided with a camera having an auto focus function may be used as the imaging apparatus 200.

[0056] The image pickup unit 211 includes a lens group 212, a shutter 213, and an image sensor 214. Changing the positions of the plurality of lenses included in the lens group 212 enables the focus position and the zoom ratio to be varied. The lens group 212 also includes a diaphragm for adjusting the exposure amount.

[0057] The image sensor 214 is composed of a charge storage type solid-state image sensor (such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor, etc.), which converts an optical image into image data. The reflected light from the subject that has passed through the lens group 212 and the shutter 213 forms an image on the image sensor 214. The image sensor 214 generates an electrical signal corresponding to the subject image and outputs image data based on the electrical signal.

[0058] The shutter 213 performs exposure and shielding of the image sensor 214 by opening and closing the shutter barrier member to control the exposure time of the image sensor 214. Instead of the shutter 213, an electronic shutter that controls the exposure time in response to the drive of the image sensor 214 may be used. When the electronic shutter is operated using a CMOS sensor, a reset process is performed to set the accumulation of the charge of the pixel to zero for each pixel or for each area composed of a plurality of pixels (for example, for each row). Then, for each pixel or area for which the reset process is performed, a scanning process is performed after a predetermined time to read out a signal corresponding to the charge accumulation.

[0059] The zoom control circuit 215 controls a motor (not shown) for driving the zoom lens included in the lens group 212 to control the optical magnification of the lens group 212. The lens group 212 may be a single focus lens group without a zoom function. In this case, the zoom control circuit 215 does not need to be provided.

[0060] The distance measurement system 216 calculates the distance information to the object. A general phase difference type distance measurement sensor installed in a single-lens reflex camera can be used as the distance measurement system 216, or a system using a time of flight (TOF) sensor can be used as the distance measurement system 216. The TOF sensor is a sensor that measures the distance to the object based on the time difference (or phase difference) between the timing of transmitting the irradiation wave and the timing of receiving the reflected wave generated by the reflection of the irradiation wave from the object. In addition, for example, a position sensitive device (PSD) method using the PSD as a photodetector can be used for the distance measurement system.

[0061] Alternatively, the image sensor 214 may have a structure that includes a plurality of photoelectric conversion regions for each pixel and in which pupil positions corresponding to the plurality of photoelectric conversion regions included in a common pixel vary. With this structure, the distance measurement system 216 can calculate distance information for each pixel or for each region position based on the phase difference between images output from the image sensor 214 and acquired from the photoelectric conversion regions corresponding to the respective pupil regions.

[0062] The ranging system 216 may have a structure for calculating distance information in predetermined one or more ranging regions in an image, or may have a structure for acquiring a distance map indicating the distribution of distance information in a plurality of pixels or regions in an image.

[0063] Alternatively, the distance measurement system 216 may perform TV auto focus (AF) or contrast AF that extracts the radio frequency component of the image data for integration and determines the position of the focus lens having the maximum integrated value to calculate distance information according to the position of the focus lens.

[0064] The image processing circuit 217 performs predetermined image processing on the image data output from the image sensor 214. The image processing circuit 217 performs various image processing such as white balance adjustment, gamma correction, color interpolation, demosaicing, and filtering on the image data output from the imaging unit 211 or the image data recorded in the internal memory 221. In addition, the image processing circuit 217 performs compression processing on the image data subjected to the image processing according to, for example, the Joint Photographic Experts Group (JPEG) standard.

[0065] The AF control circuit 218 determines the position of the focus lens included in the lens group 202 based on the distance information calculated in the ranging system 216 to control a motor for driving the focus lens.

[0066] The communication unit 219 is a wireless communication module used by the camera 200 to communicate with an external device (such as the image processing device 300) through a wireless communication network (not shown). A specific example of a network is a network based on the Wi-Fi standard. Communication using Wi-Fi can be implemented using a router. The communication unit 219 can be implemented through a wired communication interface such as a universal serial bus (USB) or a local area network (LAN).

[0067] The system control circuit 220 includes a central processing unit (CPU), and controls the respective blocks in the image pickup apparatus 200 according to a program stored in the internal memory 221 to control the entire image pickup apparatus 200. In addition, the system control circuit 220 controls the image pickup unit 211, the zoom control circuit 215, the ranging system 216, the image processing circuit 217, and the AF control circuit 218, etc. Instead of the CPU, the system control circuit 220 may use a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.

[0068] The internal memory 221 is composed of a rewritable memory such as a flash memory or a synchronous dynamic random access memory (SDRAM). The internal memory 221 temporarily stores various setting information necessary for the operation of the imaging device 200 including information on the focus and zoom magnification in image capturing, image data captured by the imaging unit 211, and image data subjected to image processing in the image processing circuit 217. For example, the internal memory 221 can temporarily record image data received by the communication unit 219 through communication with the image processing device 300 and analysis data including information indicating the size of a subject.

[0069] The external memory interface (I / F) 222 is an interface with a nonvolatile storage medium (such as a secure digital (SD) card or a compact flash (CF) card) that can be loaded in the image pickup apparatus 200. The external memory I / F 222 records the image data processed in the image processing circuit 217 and the image data received by the communication unit 219 through communication with the image processing apparatus 300, analysis data, and the like on the storage medium that can be loaded in the image pickup apparatus 200. The external memory I / F 222 can read out the image data recorded on the storage medium that can be loaded in the image pickup apparatus 200, and can output the read-out image data to the outside of the image pickup apparatus in playback.

[0070] The display unit 223 is a display composed of, for example, a thin film transistor (TFT) liquid crystal display, an organic electroluminescent (EL) display, or an electronic viewfinder (EVF). The display unit 223 displays an image based on image data temporarily stored in the internal memory 221, an image based on image data stored in a storage medium that can be loaded in the imaging device, and a setting screen of the imaging device 10, etc.

[0071] The operation member 224 is constituted by, for example, buttons, switches, keys, and a mode dial provided on the imaging apparatus 200 or a touch panel also used as the display unit 223. Instructions from the user, for example, to set a mode or instruct shooting, are provided to the system control circuit 220 through the operation member 224.

[0072] The imaging unit 211, the zoom control circuit 215, the ranging system 216, the image processing circuit 217, the AF control circuit 218, the communication unit 219, the system control circuit 220, the internal memory 221, the external memory I / F 222, the display unit 223, and the operation member 224 are connected to a common bus 225. The common bus 225 is a signal line for transmitting and receiving signals between the respective blocks.

[0073] Figure 3 is a diagram showing an example of a hardware structure of an image processing apparatus 300 included in the image processing system 1. The image processing apparatus 300 includes an operation unit 311 composed of a CPU, a storage unit 312, a communication unit 313, an output unit 314, and an auxiliary operation unit 317. The storage unit 312 is composed of a main storage unit 315 (e.g., a read-only memory (ROM) or a random access memory (RAM)) and an auxiliary storage unit 316 (e.g., a disk drive or a solid-state drive (SSD)).

[0074] The communication unit 313 is configured as a wireless communication module for communicating with an external device via a communication network. The output unit 314 outputs data processed in the operation unit 311 and data stored in the storage unit 312 to a display, a printer, or an external network connected to the image processing apparatus 300 .

[0075] The auxiliary operation unit 317 is an integrated circuit (IC) for auxiliary operation operations used under the control of the operation unit 311. A graphics processing unit (GPU) can be used as an example of an auxiliary operation unit. Since the GPU includes multiple product-sum operators and excels in matrix calculations, although the GPU was originally a processor for image processing, the GPU is also often used as a processor for signal learning processing. GPUs are commonly used for deep learning processing. For example, the Jetson TX2 module manufactured by NVIDIA can be used as an auxiliary operation unit 317. FPGA or ASIC can be used as the auxiliary operation unit 317. The auxiliary operation unit 317 extracts the affected area 102 of the subject 101 from the image data.

[0076] The calculation unit 311 can realize various functions including calculation processing for calculating the size and length of the affected area 102 extracted by the auxiliary calculation unit 317 by executing the program stored in the storage unit 312. In addition, the calculation unit 311 controls the order in which each function is performed.

[0077] The image processing device 300 may include one operation unit 311 and one storage unit 312 or a plurality of operation units 311 and a plurality of storage units 312. In other words, when at least one processing unit (CPU) is connected to at least one storage unit and at least one processing unit executes a program stored in at least one storage unit, the image processing device 300 performs the functions described below. Instead of a CPU, an FPGA or an ASIC, etc. may be used as the operation unit 311.

[0078] Figure 4 FIG. 1 is a flowchart showing the operation of the image processing system 1 according to the first embodiment. Figure 4 , the step is represented by S. That is, step 401 is represented by S401. The same applies to the following description Fig. 9 , Fig.11 and Fig.15 .

[0079] exist Figure 4 In the workflow diagram of , steps 401 to 420 are performed by the camera device 200, and steps 431, 441 to 445, and 451 to 456 are performed by the image processing device 300.

[0080] First, the camera 200 and the image processing device 300 are connected to a network (not shown) that complies with the Wi-Fi standard, which is a wireless LAN standard. In step 431, the image processing device 300 performs a search process for the camera 200 to which the image processing device 300 is to be connected. In step 401, the camera 200 performs a response process in response to the search process. For example, Universal Plug and Play (UPnP) is used as a technology for searching for devices through a network. In UPnP, a universal unique identifier (UUID) is used to identify each device.

[0081] In response to the camera 200 being connected to the image processing device 300, the camera 200 starts live view processing in step 402. The camera unit 211 generates image data, and the image processing circuit 217 applies the development processing required to generate image data for live view display to the image data. Repeating these processes causes live view video of a specific frame rate to be displayed in the display unit 223.

[0082] In step 403, the distance measurement system 216 calculates the distance information about the subject using any of the above methods, and the AF control circuit 218 starts the AF process to drive and control the lens group 212 so that the subject is focused. When the focus is adjusted using TV-AF or contrast AF, the distance information from the position of the focus lens in the focused state to the focused subject 101 is calculated. The position to be focused may be a subject located in the center of the image data or a subject existing at a position closest to the camera 200. When a distance map of the subject is acquired, a target area may be estimated based on the distance map, and the focus lens may be focused on the position. Alternatively, when the position of the bedsore 102 on the live view image is identified by the image processing device 300, the focus lens may be focused on the position of the bedsore on the live view image. The camera 200 repeatedly performs the display of the live view video and the AF process until the pressing of the release button is detected in step 410.

[0083] In step 404, the image processing circuit 217 performs development processing and compression processing on any image data captured for live view, for example, to generate image data that conforms to the JPEG standard. Then, the image processing circuit 217 performs resizing processing on the compressed image data to reduce the size of the image data.

[0084] In step 405, the communication unit 219 acquires the image data subjected to the resizing process in step 404 and the distance information calculated in step 403. In addition, the communication unit 219 acquires information on the zoom magnification and information on the size (the number of pixels) of the image data subjected to the resizing process. When the imaging unit 211 has a single focus without a zoom function, it is not necessary to acquire information on the zoom magnification.

[0085] In step 406 , the communication unit 219 transmits the image data acquired in step 405 and at least one information including distance information to the image processing apparatus 300 through wireless communication.

[0086] Since wireless communication takes longer as the size of the image data to be transmitted increases, the size of the image data after the resizing process in step 405 is determined in consideration of the allowed communication time. However, since the accuracy of extraction of the affected area performed by the image processing apparatus 300 in step 433 described below is affected if the image data has an excessively reduced size, the accuracy of extraction of the affected area needs to be considered in addition to the communication time.

[0087] Steps 404 to 406 may be performed for each frame, or may be performed once every several frames.

[0088] The operation proceeds to the description of the steps performed by the image processing apparatus 300 .

[0089] In step 441 , the communication unit 313 in the image processing apparatus 300 receives the image data and at least one information including the distance information transmitted from the communication unit 219 in the imaging apparatus 200 .

[0090] In step 442, the operation unit 311 and the auxiliary operation unit 317 in the image processing device 300 extract the affected area 102 of the subject 101 from the image data received in step 441. As a method of extracting the affected area 102, semantic segmentation using deep learning is performed. Specifically, a high-performance computer (not shown) for learning is made to learn a neural network model using a plurality of actual bedsore images as teaching data in advance, thereby generating a learning model. The auxiliary operation unit 317 receives the generated learning model from the high-performance computer, and estimates the area of ​​the bedsore as the affected area 102 from the image data based on the learning model. A fully convolutional network (FCN) as a segmentation model using deep learning is applied as an example of a neural network model. Here, the inference of deep learning is processed by the auxiliary operation unit 317 which excels in parallel execution of product and sum operations. The inference processing can be performed using FPGA or ASIC. Other deep learning models can be used to implement region segmentation. The segmentation method is not limited to deep learning, and for example, graphic clipping, region growth, edge detection, or division and restriction, etc. can be used as the segmentation method. In addition, the auxiliary operation unit 317 may perform learning of a neural network model using an image of bedsores as teaching data.

[0091] In step 443 , the operation unit 311 calculates the area of ​​the affected region 102 as information indicating the size of the affected region 102 extracted by the auxiliary operation unit 317 .

[0092] Figure 5 is a diagram for describing how to calculate the area of ​​the affected area 102. The imaging device 200, which is a general camera, can be processed as Figure 5 The pinhole model shown. Incident light 501 passes through the principal point of lens 212a and is received on the imaging plane of image sensor 214. When lens group 212 is approximated as a thin single lens 212a, it is considered that the principal point of the front side coincides with the principal point of the rear side. The focus of lens 212 is adjusted so that an image is formed on the plane of image sensor 214, so that the imaging device can focus on subject 504. The focal length 502, which is the distance from the imaging plane to the principal point of the lens, is changed, and the angle of view 503 is changed to change the zoom magnification. At this time, the width 506 of the subject on the focal plane is geometrically determined according to the relationship between the angle of view 503 and the subject distance 505 of the imaging device. The width 506 of the subject is calculated using trigonometric functions. Specifically, the width 506 of the subject is determined based on the relationship between the angle of view 503 and the subject distance 505 that vary with the focal length 502. The value of the width 506 of the subject is divided by the number of pixels on each line of the image data to calculate the length on the focal plane corresponding to one pixel on the image data.

[0093] Therefore, the operation unit 311 calculates the area of ​​the affected area 102 as the product of the number of pixels in the extracted area obtained from the extraction result of the affected area in step 442 and the area of ​​one pixel obtained from the length on the focal plane corresponding to one pixel on the image. The length on the focal plane corresponding to one pixel on the image (which corresponds to a combination of the focal length 502 and the object distance 505) may be calculated in advance to be prepared as table data. The image processing device 300 may store table data corresponding to the imaging device 200 in advance.

[0094] In order to accurately calculate the area of ​​the affected area 102 using the above method, it is assumed that the subject 504 is a plane and the plane is perpendicular to the optical axis. If the distance information received in step 441 is distance information or a distance map at a plurality of positions in the image data, the tilt or change in the depth direction of the subject can be detected to calculate the area based on the detected tilt or change.

[0095] In step 444 , the operation unit 311 generates image data obtained by superimposing information indicating the extraction result of the affected area 102 and information indicating the size of the affected area 102 on the image data used to extract the affected area 102 .

[0096] Fig. 6A and Figure 6B 1 is a diagram showing how information indicating the extraction result of the affected area 102 and information indicating the size of the affected area 102 are superimposed on the image data. Fig. 6A An image 601 in FIG. 1 is an image displayed using image data before superimposition processing, and includes a subject 101 and an affected area 102 . Figure 6B The superimposed image 602 in FIG. 1 is an image based on the image data after the superimposition processing. Fig. 6A and Figure 6B The affected area 102 is indicated to be proximal to the buttocks.

[0097] The calculation unit 311 superimposes a mark 611 at the upper left corner of the superimposed image 602. A character string 612 indicating the area value of the affected area 102 is displayed on the mark 611 with white characters on a black background as information indicating the size of the affected area 102.

[0098] The background color and the color of the character string on the mark 611 are not limited to black and white, respectively, as long as the background and the character string are easily visible. The transmittance can be set, and the set transmittance can be alpha-blended to enable confirmation of the portion where the mark is superimposed.

[0099] In addition, the indicator 613 indicating the estimated area of ​​the affected area 102 extracted in step 442 is superimposed on the superimposed image 602. The indicator 613 indicating the estimated area and the image data on which the image 601 is based are alpha blended at the position where the estimated area exists so that the user can confirm whether the estimated area on which the area of ​​the affected area is based is appropriate. The color of the indicator 613 indicating the estimated area is not expected to be equal to the color of the subject. It is expected that the transmittance of the alpha blending is within a range that can identify the estimated area and also can confirm the original affected area 102. Since the user can confirm whether the estimated area is appropriate without displaying the mark 611 when the indicator 613 indicating the estimated area of ​​the affected area 102 is superimposed, step 443 can be omitted.

[0100] In step 445, the communication unit 313 in the image processing apparatus 300 transmits information indicating the extraction result of the extracted affected area 102 and information indicating the size of the affected area 102 to the imaging apparatus 200. In the present embodiment, the communication unit 313 transmits the image data including the information indicating the size of the affected area 102 generated in step 444 to the imaging apparatus 200 through wireless communication.

[0101] The operation returns to the description of the steps performed by the imaging apparatus 200 .

[0102] In step 407 , the communication unit 219 in the imaging apparatus 200 receives any image data that includes information indicating the size of the affected area 102 and is newly generated in the image processing apparatus 300 .

[0103] In step 408 , if image data including information indicating the size of the affected area 102 is received in step 407 , the system control circuit 220 proceeds to step 409 , and otherwise proceeds to step 410 .

[0104] In step 409, the display unit 223 displays the image data including the information indicating the size of the affected area 102 received in step 407 for a specific period of time. Here, the display unit 223 displays Figure 6B The superimposed image 602 shown. Superimposing the information indicating the extraction result of the affected area 102 on the live view image in the above manner enables the user to shoot after the user confirms whether the area of ​​the affected area and the estimated area are appropriate. Although the example of displaying the index 613 indicating the estimated area of ​​the affected area 102 and the information about the size of the affected area 102 is described in the present embodiment, either the index 613 indicating the estimated area of ​​the affected area 102 and the information about the size of the affected area 102 may be displayed.

[0105] In step 410, the system control circuit 220 determines whether a release button included in the operation member 224 is pressed. If the release button is not pressed, the imaging apparatus 200 returns to step 404. If the release button is pressed, the imaging apparatus proceeds to step 411.

[0106] In step 411, the distance measuring system 216 calculates the distance information about the subject, and the AF control circuit 218 performs AF processing to drive and control the lens group 212 using the same method as in step 403 so that the subject is focused. If the affected area 102 has been extracted from the live view image, the distance measuring system 216 calculates the distance information about the subject at the position where the affected area 102 exists.

[0107] In step 412 , the camera 200 captures a still image.

[0108] In step 413, the image processing circuit 217 performs development processing and compression processing on the image data generated in step 412 to generate image data that conforms to the JPEG standard, for example. Then, the image processing circuit 217 performs resizing processing on the image data that has been subjected to the compression processing to reduce the size of the image data. The size of the image data that has been subjected to the resizing processing in step 413 is equal to or greater than the size of the image data that has been subjected to the resizing processing in step 404. This is because the measurement accuracy of the affected area 102 is given priority. Here, the image data is resized to approximately 4.45 megabytes with 1440 pixels × 1080 pixels in 4-bit RGB color. The size of the image data after resizing is not limited thereto. Alternatively, the operation may proceed to the subsequent steps using the generated image data that conforms to the JPEG standard without performing the resizing processing.

[0109] In step 414, the communication unit 219 acquires the image data generated in step 413 and subjected to the resizing process (or not subjected to the resizing process) and the distance information calculated in step 411. In addition, the communication unit 219 also acquires information related to the zoom magnification and information related to the size (the number of pixels) of the image data subjected to the resizing process. When the imaging unit 211 has a single focus without a zoom function, it is not necessary to acquire information related to the zoom magnification. When the image processing device 300 has information related to the size of the image data in advance, it is not necessary to acquire information related to the image data.

[0110] In step 415 , the communication unit 219 transmits the image data acquired in step 414 and at least one information including distance information to the image processing apparatus 300 through wireless communication.

[0111] The operation proceeds to the description of the steps performed by the image processing apparatus 300 .

[0112] In step 451 , the communication unit 313 in the image processing apparatus 300 receives the image data and at least one information including the distance information transmitted from the communication unit 219 in the imaging apparatus 200 .

[0113] In step 452, the operation unit 311 and the auxiliary operation unit 317 in the image processing device 300 extract the affected area 102 of the subject 101 from the image data received in step 441. Since the details of this step are the same as those of step 442, the detailed description of step 452 is omitted here.

[0114] In step 453, the operation unit 311 calculates the area of ​​the affected region 102 as an example of information indicating the size of the affected region 102 extracted by the auxiliary operation unit 317. Since the details of this step are the same as step 443, a detailed description of step 453 is omitted here.

[0115] In step 454, the operation unit 311 performs image analysis to calculate the major axis length and the minor axis length of the extracted affected area and the area of ​​the circumscribed rectangle surrounding the affected area based on the length on the focal plane corresponding to one pixel on the image calculated in step 453. DESIGN-R (registered trademark), which is an evaluation index for bedsores, defines that the size of bedsores is calculated by measuring the value of the product of the major axis length and the minor axis length. In the image processing system of the present invention, analysis of the major axis length and the minor axis length makes it possible to ensure compatibility with data measured with DESIGN-R (registered trademark). Since no strict definition is provided in DESIGN-R (registered trademark), multiple mathematical methods for calculating the major axis length and the minor axis length are considered.

[0116] As an example of a method of calculating the major axis length and the minor axis length, first, the operation unit 311 calculates the minimum bounding rectangle as a rectangle having the minimum area among the circumscribed rectangles surrounding the affected area 102. Then, the operation unit 311 calculates the lengths of the long side and the short side of the rectangle. The length of the long side is calculated as the major axis length, and the length of the short side is calculated as the minor axis length. Then, the operation unit 311 calculates the area of ​​the rectangle based on the length on the focal plane corresponding to one pixel on the image calculated in step 453.

[0117] As another example of a method of calculating the major axis length and the minor axis length, the maximum Feret diameter (Ferret diameter) as the maximum caliper length may be selected as the major axis length and the minimum Feret diameter may be selected as the minor axis length. Alternatively, the maximum Feret diameter as the maximum caliper length may be selected as the major axis length, and the length measured in a direction orthogonal to the axis of the maximum Feret diameter may be selected as the minor axis length. The method of calculating the major axis length and the minor axis length may be arbitrarily selected based on compatibility with measurement results in the prior art.

[0118] The major axis length and minor axis length of the affected area 102 and the area of ​​the rectangle are not calculated for the image data received in step 441. Since the extraction result of the affected area 102 is expected to be confirmed by the user during live view, the image analysis step in step 454 is omitted to reduce processing time.

[0119] When it is expected to acquire information about the actual area of ​​bedsores without evaluating the size based on DESIGN-R (registered trademark), step 454 may be omitted. In this case, it is assumed that there is no information about the size as an evaluation item in DESIGN-R (registered trademark) in the subsequent steps.

[0120] In step 455 , the operation unit 311 generates image data obtained by superimposing information indicating the extraction result of the affected area 102 and information indicating the size of the affected area 102 on the image data serving as the extraction target of the affected area 102 .

[0121] 7A to 7C FIG. 1 is a diagram for describing a method of superimposing information indicating the extraction result of the affected area 102 and information indicating the size of the affected area (including the major axis length and the minor axis length of the affected area 102) on image data. Since a plurality of information indicating the size of the affected area 102 is taken into consideration, the following are described separately. Fig. 7A The superimposed image 701, Figure 7B The superimposed image 702 and Figure 7C The overlay image 703 in FIG.

[0122] exist Fig. 7A In the case of the superimposed image 701, the minimum bounding rectangle is used as a method for calculating the major axis length and the minor axis length. A marker 611 is superimposed at the upper left corner of the superimposed image 701. Figure 6BLikewise, a character string 612 indicating the area value of the affected area 102 is displayed in white characters on a black background on the mark 611 as information indicating the size of the affected area 102. In addition, a mark 712 is superimposed at the upper right corner of the superimposed image 701. The major axis length and the minor axis length calculated based on the minimum bounding rectangle are displayed on the mark 712 as information indicating the size of the affected area 102. A character string 713 indicates the major axis length (cm), and a character string 714 indicates the minor axis length (cm). A rectangular frame 715 indicating the minimum bounding rectangle is displayed around the affected area 102 on the superimposed image 701. Superimposing the rectangular frame 715 with the major axis length and the minor axis length enables the user to confirm the position in the image where the length is being measured.

[0123] In addition, a scale bar 716 is superimposed at the lower right corner of the superimposed image 701. The scale bar 716 is used to measure the size of the affected area 102, and the size of the scale bar on the image data varies with the distance information. Specifically, the scale bar 716 is a bar on which scale marks from 0 cm to 5 cm are indicated in units of 1 cm based on the length on the focal plane corresponding to one pixel on the image calculated in step 453, and the bar matches the size on the focal plane of the imaging device (i.e., on the subject). The user can refer to the scale bar to know the approximate size of the subject or the affected area.

[0124] In addition, the evaluation value of the size of the above-mentioned DESIGN-R (registered trademark) is superimposed at the lower left corner of the superimposed image 701. The evaluation value of the size of DESIGN-R (registered trademark) is classified into the above-mentioned seven levels based on the value obtained by measuring the major axis length (cm) and the minor axis length (the maximum diameter orthogonal to the major axis length) (cm) of the skin damage range and multiplying the major axis length by the minor axis length. In this embodiment, the evaluation value obtained by replacing the major axis length and the minor axis length with the value output by the calculation method is superimposed.

[0125] exist Figure 7B In the case of the superimposed image 702 in FIG. 7 , the maximum Feret diameter 521 is used as the major axis length, and the minimum Feret diameter 522 is used as the minor axis length. A mark 722 is superimposed at the upper right corner of the superimposed image 702. A major axis length character string 723 and a minor axis length character string 724 are displayed on the mark 722. In addition, an auxiliary line 725 corresponding to the measurement position of the maximum Feret diameter 521 and an auxiliary line 726 corresponding to the minimum Feret diameter 522 are displayed in the affected area 102 of the superimposed image 702. Superimposing the auxiliary lines with the major axis length and the minor axis length enables the user to confirm the position in the image where the length is being measured.

[0126] Figure 7CThe superimposed image 703 in FIG. 7 is the same as the superimposed image 702 in terms of the long axis length. However, the short axis length is not measured as the minimum Feret diameter, but is measured as the length measured in the direction orthogonal to the axis of the maximum Feret diameter on the superimposed image 703. A marker 732 is superimposed at the upper right corner of the superimposed image 702. A long axis length character string 723 and a short axis length character string 734 are displayed on the marker 732. In addition, an auxiliary line 725 corresponding to the measurement position of the maximum Feret diameter 521 and an auxiliary line 736 corresponding to the length measured in the direction orthogonal to the axis of the maximum Feret diameter are displayed in the affected area 102 of the superimposed image 702.

[0127] Can be used 7A to 7C Any one of the information shown in to be superimposed on the image data or a combination of multiple information may be used. Optionally, the user may be able to select the information to be displayed. Figure 6B and 7A to 7C The superimposed image shown in is only an example, and the display mode, display position, font type, font size, font color, and positional relationship, etc. of the affected area 102 and the information indicating the size of the affected area 102 may vary according to various conditions.

[0128] In step 456, the communication unit 313 in the image processing device 300 transmits information indicating the extraction result of the extracted affected area 102 and information indicating the size of the affected area 102 to the imaging device 200. In the present embodiment, the communication unit 313 transmits the image data including the information indicating the size of the affected area 102 generated in step 455 to the imaging device 200 through wireless communication.

[0129] The operation returns to the description of the steps performed by the imaging apparatus 200 .

[0130] In step 416 , the communication unit 219 in the imaging apparatus 200 receives the image data including the information indicating the size of the affected area 102 generated in the image processing apparatus 300 .

[0131] In step 417, the display unit 223 displays the image data including the information indicating the size of the affected area 102 received in step 416 for a specific period of time. Here, the display unit 223 displays the image data including the information indicating the size of the affected area 102 respectively in step 416. 7A to 7C Any one of the superimposed images 701 to 703 shown in FIG. 4 is displayed, and the operation proceeds to step 418 after a certain period of time has passed.

[0132] In step 418, it is determined whether there is any affected area information for which no value has been input. The affected area information indicates information indicating the location of the affected area and the evaluation value of each evaluation item of the above-mentioned DESIGN-R (registered trademark). Based on the information indicating the size received in step 416, the evaluation value of the evaluation item regarding the size is automatically input.

[0133] If there is affected part area information for which no value is input in step 418, the operation proceeds to step 419. If all affected part area information is input in step 418, the operation returns to step 402 to start live view again.

[0134] In step 419 , the system control circuit 220 displays a user interface on the display unit 223 to prompt the user to input the affected area information.

[0135] In step 420 , when the user inputs the affected area information, the operation returns to step 418 .

[0136] FIG. 8A to FIG. 8G This is a diagram for describing how the user inputs the affected area information in steps 419 and 420 .

[0137] Fig. 8A This is a screen that prompts the user to input the location of the affected area in the affected area information.

[0138] Part selection items 801 for designating the part of the affected part area: head, shoulder, arm, back, waist, buttocks, and leg are displayed in the display unit 223. An item for completing the input of the affected part area information is provided below the part selection item 801. Selecting this item enables the input of the affected part area information to be terminated even if a part of the affected part area information is not input.

[0139] The user can specify the location where the affected area to be photographed is located using the operation member 224. The item selected by the user is displayed surrounded by a frame line 802. Fig. 8A Since two or more affected areas may exist in one part selected from the part selection items 801, selection of a plurality of items such as buttocks 1, buttocks 2, and buttocks 3 may be further available.

[0140] Figure 8B is Fig. 8A After selecting a part including the affected area, the user is prompted to confirm whether the selected part is appropriate. When the selected part is confirmed by the user operation, the display unit 223 displays Figure 8C The screen shown.

[0141] Figure 8CThis is a screen that prompts the user to input the evaluation value of each evaluation item of DESIGN-R (registered trademark) in the affected area information.

[0142] An evaluation item selection section 804 is displayed on the left side of the screen. Each item: D (depth), E (exudate), S (size), I (inflammation / infection), G (granulation tissue), N (necrotic tissue), and P (cyst) and information indicating whether to input each item are displayed together with the image of the affected area. Figure 8C , an evaluation value "s9" is displayed for S (size) that has been analyzed from the image, and "non" indicating that the item has not been confirmed is displayed for the remaining evaluation items. The shading of the item S (size) indicates that the item (size) has been input.

[0143] The user can specify an evaluation item using the operation member 224. The selected evaluation item (D (depth) here) is displayed surrounded by a frame line 805.

[0144] The evaluation value of the severity level of the evaluation item selected on the left side of the screen is superimposed on the bottom of the screen as the severity level selection unit 806. Figure 8C , d0, d1, d2, D3, D4, D5, and DU are displayed as evaluation values ​​indicating the severity level of D (depth).

[0145] The user can select any evaluation value using the operation member 224. The selected evaluation value is displayed surrounded by a frame line 807, and a description text 808 of the evaluation value (description of evaluation item depth and severity level d2: damage to dermis) is also displayed. The evaluation value can be input by the user who inputs a character string.

[0146] Fig.8D Shown for Figure 8C After selecting an evaluation value, a confirmation notification 809 is displayed to inquire the user whether the selected evaluation value is appropriate.

[0147] When the user confirms with the operating member 224 that there is no problem with the selected evaluation value, the screen changes to Fig. 8E The screen shown.

[0148] exist Fig. 8E , in response to the input of the evaluation value, the display of the evaluation item 810 of D (depth) is changed from “non” to “d2”, and the evaluation item 810 is shaded.

[0149] Likewise, a screen prompting the user to input evaluation values ​​for E (exudate), I (inflammation / infection), G (granulation tissue), N (necrotic tissue), and P (cyst) is displayed until evaluation values ​​are input for all evaluation items.

[0150] In response to the input of the evaluation values ​​of all the evaluation items, the user is notified that the input of the affected area information is completed. Then, the operation returns to step 402 to start the live view process.

[0151] As described above, in the first embodiment, the following function is provided: after photographing the affected area, the user is prompted to input the evaluation values ​​of the evaluation items that are not automatically analyzed and the information related to the position of the affected area, so that the user can input the affected area information. In the above manner, the affected area information that is input using other media in the prior art can be input only using the camera.

[0152] In addition, by determining whether all the affected area information has been input before photographing the next affected area and sequentially prompting the user to input the evaluation items that have not been input, it is possible to prevent omission of input of the affected area information.

[0153] A voice recognition input section may be used as the operation member 224 according to the first embodiment.

[0154] exist Fig. 8A In , when the part of the affected area is input and characters are selected, the part is displayed using characters such as "head" and "shoulder". Fig.8F As shown, a structure may be adopted in which a human body model 811 is displayed on the display unit 223 and a user is caused to specify a location of an affected area using a touch sensor provided on the display unit 223 .

[0155] In addition, if Figure 8G As shown, a structure may be adopted in which the human body model 811 is enlarged, reduced, or rotated so that the site of the affected area can be easily selected.

[0156] Despite Fig. 8E In the example, shading is used as a means for indicating that input of an evaluation value is completed for an evaluation item, but the brightness of the characters may be reduced or the characters may be highlighted. Other display methods may be used as long as the fact that an evaluation value has been input for an evaluation item is explicitly indicated to the user.

[0157] Although DESIGN-R (registered trademark) is used as an available evaluation index for bedsores in this example, the evaluation index is not limited thereto. Other evaluation indexes such as the Bates-Jensen Wound Assessment Tool (BWAT), the Bedsore Scale for Healing (PUSH), or the Bedsore Pain State Tool (PSST) may be used. Specifically, a user interface for inputting evaluation items in BWAT, PUSH, or PSST, etc. may be displayed in response to the extraction result of the area of ​​bedsores and the acquisition of information related to the size of the extracted area.

[0158] Although an example of a structure in which an evaluation value of an evaluation item of bedsore is expected to be input is described in this example, it is expected that evaluation values ​​of evaluation items in other skin diseases can be input as long as visual evaluation items are used. For example, the Severity Score of Atopic Dermatitis (SCORAD) in atopic dermatitis and the Body Surface Area, Psoriasis Area and Severity Index (PASI) in psoriasis are exemplified.

[0159] As described above, according to the present embodiment, there is provided an image processing system in which, in response to a user photographing the affected area 102 using the camera 200, information indicating the size of the affected area is displayed in the display unit 223 in the camera 200. Therefore, the burden on medical personnel in the evaluation of the size of the affected area of ​​bedsores and the burden on patients to be evaluated can be reduced. In addition, compared with a case where a medical staff manually measures the size of the affected area, calculating the size of the affected area based on a program enables individual differences to be reduced to improve the accuracy of the evaluation of the size of bedsores. Furthermore, the area of ​​the affected area as an evaluation value can be calculated, and the calculated area of ​​the affected area can be displayed to more accurately indicate the size of bedsores.

[0160] Since a function for the user to confirm whether the estimated area of ​​the affected area is appropriate during live view display is not essential, a configuration may be adopted in which step 406 , step 407 , and steps 441 to 445 are omitted.

[0161] The image processing apparatus 300 may store information indicating the extraction result of the affected area 102, information indicating the size of the affected area 102, and image data related to a superimposed image on which the information indicating the extraction result of the affected area 102 and the information indicating the size of the affected area 102 are superimposed in the storage unit 312. The output unit 314 can output at least one of the information or image data stored in the storage unit 312 to an output device such as a display connected to the image processing apparatus 300. Displaying the superimposed image in the display enables other users other than the user who photographed the affected area 102 to acquire an image of the affected area 102 in real time, or to acquire an image of the affected area 102 that has been photographed and information indicating the size of the affected area 102. The operation unit 311 in the image processing apparatus 300 may have a function of displaying a scale bar or the like that arbitrarily changes the position and angle for the image data to be sent to the display from the output unit 314. The display of such a scale bar enables the user who views the display to measure the length of an arbitrary position of the affected area 102. It is desirable to automatically adjust the width of the memory of the scale bar based on the distance information received in step 451 , the information on the zoom magnification, the information on the size (the number of pixels) of the image data subjected to the resizing process, and the like.

[0162] Using the image processing apparatus 300 in a state where power is constantly supplied on the fixed side enables acquisition of an image of the affected area 102 and information indicating the size of the affected area 102 at arbitrary timing without the risk of battery exhaustion. In addition, since the image processing apparatus 300, which is generally a fixed device, has a high storage capacity, the image processing apparatus 300 is able to store a large amount of image data.

[0163] In addition, according to the present embodiment, when the user photographs the affected area 102 using the camera 200, the user can input and record information about the affected area 102 that is different from the information acquired by the image analysis of the image. Therefore, the user does not need to input the evaluation of the affected area on the electronic health record or paper medium later while the user views the photographed image data. In addition, presenting the user with items that have not been input prevents the user from forgetting to input information when photographing the affected area.

[0164] (Second embodiment)

[0165] In the image processing system according to the first embodiment, the image processing device 300 performs a process of superimposing information indicating the extraction result of the affected area and information indicating the size of the affected area on the image data. In contrast, in the image processing system according to the second embodiment, the image processing circuit 217 in the imaging device 200 performs a process of superimposing information indicating the extraction result of the affected area and information indicating the size of the affected area on the image data.

[0166] Fig. 9 is a flowchart showing the operation of the image processing system 1 according to the second embodiment.

[0167] exist Fig. 9 In the workflow, no Figure 4 The superimposition processing in steps 444 and 455 by the image processing apparatus 300 in the illustrated workflow is replaced with the superimposition processing in steps 901 and 902 by the imaging apparatus 200. Fig. 9 In the steps described, it is assumed that Figure 4 The steps in the Figure 4 The same processing as in the corresponding steps in .

[0168] In the present embodiment, the data to be sent from the image processing device 300 to the camera 200 in steps 445 and 456 to generate the superimposed image by the camera 200 may not be image data using a color scale. Since the image processing device 300 does not send image data but sends metadata indicating the estimated size of the affected area and data indicating the position of the affected area, communication traffic can be reduced to increase communication speed. The data indicating the estimated position of the affected area is data in a vector format having a smaller size. The data indicating the estimated position of the affected area may be data in a binary raster format.

[0169] Upon receiving metadata indicating the estimated size of the affected area and data indicating the position of the affected area from the image processing apparatus 300 in step 407 or step 416 , the imaging apparatus 200 generates a superimposed image in step 901 or step 902 , respectively.

[0170] Specifically, in step 901, the image processing circuit 217 in the camera device 200 uses Figure 4 The method described in step 444 of generates a superimposed image. The image data on which the information indicating the estimated size and position of the affected area is to be superimposed may be the image data sent from the imaging device 200 to the image processing device 300 in step 406, or may be image data related to the latest frame displayed as a live view image.

[0171] In step 902, the image processing circuit 217 in the camera device 200 uses Figure 4 The superimposed image is generated by the method described in step 455 of . The image data on which the information indicating the estimated size and position of the affected area is to be superimposed is the image data sent from the imaging device 200 to the image processing device 300 in step 415 .

[0172] As described above, according to this embodiment, since the amount of data to be transmitted from the image processing apparatus 300 to the imaging apparatus 200 is reduced, the amount of communication between the imaging apparatus 200 and the image processing apparatus 300 can be reduced to increase the communication speed compared to the first embodiment.

[0173] (Third Embodiment)

[0174] Fig.10 is a diagram schematically showing an image processing system 11 according to the third embodiment. Fig.10The image processing system 11 shown includes, in addition to the camera 200 and the image processing device 300 described above in the first and second embodiments, a terminal device 1000 as an electronic device capable of Web access. The terminal device 1000 is composed of, for example, a tablet terminal and has a Web browser function. The terminal device 1000 can access a Web server and display an acquired Hypertext Markup Language (HTML) file. The terminal device 1000 is not limited to a tablet terminal, and can be a web browser or a device capable of displaying an image using dedicated application software. The terminal device 1000 can be, for example, a smart phone or a personal computer. Although the camera 200 and the terminal device 1000 are described here as separate devices, a single device can be used as the camera 200 and the terminal device 1000. When the terminal device 1000 is a smart phone or a tablet terminal with a camera function, the terminal device 1000 can be used as the camera 200.

[0175] In addition to the processing described above in the first and second embodiments, the operation unit 311 in the image processing device 300 also performs processing for identifying a subject from the image data. In addition, the operation unit 311 performs processing for storing information related to the estimated size and position of the affected area and image data related to the affected area in the storage unit 312 for each identified subject. The terminal device 1000 enables the user to confirm the information indicating the estimated size of the affected area associated with the subject and the image data related to the affected area stored in the storage unit 312 in the image processing device 300 using a Web browser or dedicated application software. For the sake of description, it is assumed here that the terminal device 1000 enables the user to confirm the image data using a Web browser.

[0176] Although in the present embodiment, the function of identifying a subject from image data, the function of storing information or image data related to an affected area for each identified subject, or the function of performing a Web service is performed by the image processing device 300, these functions are not limited to being performed by the image processing device 300. Part or all of these functions may be realized by a computer on a network different from the image processing device 300.

[0177] refer to Fig.10, the subject 101 wears a barcode tag 103 as information for identifying the subject. The captured image data related to the affected area 102 can be associated with the identifier (ID) of the subject indicated by the barcode tag 103. The tag for identifying the subject is not limited to the barcode tag, and may be a two-dimensional code such as a QR code (registered trademark) or a numerical value. Alternatively, a tag on which text is recorded may be used as a tag for identifying the subject, and the tag may be read using an optical character recognition (OCR) reader installed in the image processing device 300.

[0178] The operation unit 311 in the image processing device 300 checks the ID obtained by analyzing the barcode tag included in the captured image data with the subject ID pre-registered in the storage unit 312 to obtain the name of the subject 101. A structure in which the camera 200 analyzes the ID and sends the ID to the image processing device 300 may be adopted.

[0179] The operation unit 311 creates a record based on the image data about the affected area 102, information indicating the size of the affected area 102 of the subject, the subject ID, the name of the acquired subject, and the shooting date and time, and registers the record in the database in the storage unit 312.

[0180] In addition, the operation unit 311 returns information registered in the database in the storage unit 312 in response to a request from the terminal device 1000 .

[0181] Fig.11 is a flowchart showing the operation of the image processing system 11 according to the third embodiment. Fig.11 In the steps described, it is assumed that Figure 4 The steps in the Figure 4 The same processing as in the corresponding steps in .

[0182] refer to Fig.11 , when the imaging device 200 and the image processing device 300 are connected, in step 1101, the imaging device 200 displays an instruction for the user to photograph the barcode label 103 in the display unit 223 and photographs the barcode label 103 in response to the user's release operation. Then, the operation proceeds to step 402. Information related to the patient ID for identifying the patient is included in the barcode label 103. Photographing the affected area 102 after photographing the barcode label 103 enables management of the photographing order based on the photographing date and time, etc. to identify the image in the image of one barcode label before the image of the next barcode label as an image of the same subject using the subject ID. The order of photographing the barcode label 103 after photographing the affected area 102 can be adopted.

[0183] After the system control circuit 220 detects the pressing of the release button in step 410 and performs steps 411 to 414, the communication unit 219 transmits the image data and at least one information including the distance information to the image processing apparatus 300 by wireless communication in step 415. In addition to the image data generated by photographing the affected area 102, the image data generated by photographing the barcode label 103 in step 1001 is also included in the image data transmitted in step 415.

[0184] In step 455 , the image processing apparatus 300 generates image data related to the superimposed image. Then, the operation proceeds to step 1111 .

[0185] In step 1111 , the operation unit 311 performs a process of reading a one-dimensional barcode (not shown) included in the image data about the barcode label 103 captured in step 1001 to read a subject ID for identifying the subject.

[0186] In step 1112 , the read subject ID is checked against the subject ID registered in the storage unit 312 .

[0187] In step 1113, if the verification of the subject ID is successful, the name of the patient and the past affected area information registered in the database in the storage unit 312 are acquired. Here, the last stored affected area information is acquired.

[0188] In step 456 , the communication unit 313 in the image processing apparatus 300 transmits information indicating the extraction result of the extracted affected area 102 , information indicating the size of the affected area 102 , and past affected area information acquired from the storage unit 312 to the imaging apparatus 200 .

[0189] In step 416 , the communication unit 219 in the imaging apparatus 200 receives the image data and the affected area information transmitted from the image processing apparatus 300 .

[0190] In step 417 , the display unit 223 displays the image data including the information indicating the size of the affected area 102 received in step 416 for a certain period of time.

[0191] In step 418, it is determined whether there is any affected area information for which no value is input.

[0192] If there is any affected part area information for which no value is input in step 418, the operation proceeds to step 1102. If all affected part area information is input in step 418, the operation proceeds to step 1104.

[0193] In step 1102 , the system control circuit 220 displays a user interface prompting the user to input the affected area information on the display unit 223 using the past affected area information.

[0194] Fig. 12A and Fig. 12B is a diagram for describing how to display acquired affected area information. Fig. 12A In the example, for a part for which an evaluation value of an evaluation item is input, the character size of an item 1102 in the part selection item 1101 displayed on the left side of the screen is increased. Fig. 12A It indicates that the evaluation values ​​of the evaluation items of the affected area are input for "back" and "buttocks".

[0195] When the user inputs the affected area information in step 420 , in step 1103 , the information is compared with the evaluation values ​​of the past evaluation items to display the result of determination whether the symptoms have been alleviated or worsened.

[0196] exist Fig. 12B In FIG. 1 , the evaluation item selection unit 1103 is displayed in three columns. The evaluation item name, the past evaluation value, and the current evaluation value are displayed in order from the left.

[0197] Here, the past evaluation value is compared with the current evaluation value. For items whose symptoms are judged to be alleviated, a green evaluation value is displayed, and for items whose symptoms are judged to be worsened, a red evaluation value is displayed.

[0198] When the evaluation values ​​of all the evaluation items are input, the user is notified that the input of the affected area information is completed. Then, the operation proceeds to step 1104 .

[0199] In step 1104, the affected area information and the image data in which the evaluation values ​​of the series of evaluation items are input are transmitted to the image processing apparatus 300 through wireless communication. Then, the operation returns to step 402.

[0200] In step 1114 , the image processing apparatus 300 receives the affected area information and the image data transmitted from the imaging apparatus 200 .

[0201] In step 1115, the operation unit 311 creates a record based on the image data obtained by photographing the affected area, the information about the position of the affected area 102, the evaluation value of each evaluation item of the affected area 102, the subject ID, the name of the acquired subject, and the photographing date and time, etc. In addition, the operation unit 311 registers the created record in the database in the storage unit 312.

[0202] In step 1116 , the operation unit 311 transmits the information registered in the database in the storage unit 312 to the terminal device 1000 in response to the request from the terminal device 1000 .

[0203] refer to Fig.13 and Fig.14 A display example of the browser of the terminal device 1000 is described.

[0204] Fig.13 1301. A data selection window 1301 is separated by a separator line 1303 for each date 1302. An icon 1305 is displayed for each shooting time 1304 in the area of ​​each date. A subject ID and a name of a subject are displayed in each icon 1305, and the icon 1305 represents a data set of the same subject shot in the same time zone. A search window 1306 is set on the data selection window 1301. Entering a date, a subject ID, or a name of a subject in the search window 1306 enables searching for a data set. In addition, operating a scroll bar 1307 enables displaying a plurality of data in an enlarged manner in a limited display area. When the user selects and clicks the icon 1305, the browser transitions to a data browsing window, and the user of the browser of the terminal device 1000 can browse images of the data set and information indicating the size of the subject. In other words, a request indicating the indicated subject and the date and time specified in the terminal device 1000 is transmitted from the terminal device 1000 to the image processing device 300. The image processing device 300 transmits image data corresponding to the request and information indicating the size of the subject to the terminal device 1000.

[0205] Fig.14 1401 is a diagram for describing an example of a data browsing window displayed in a browser of the terminal device 1000. The subject ID and the name 1402 of the subject and the shooting date and time 1403 of the data set selected on the data selection window 1301 are displayed on the data browsing window 1401. In addition, an image 1404 based on the image data and data 1405 based on the affected area information in the image 1404 are displayed for each shooting. In addition, the shooting number when the affected area of ​​the same subject is shot multiple times in succession is displayed in the number 1406. Moving the slider 1407 at the right end of the window enables display of data based on the image data and the affected area information at another shooting date and time related to the same subject ID. In addition, changing the setting enables display of data based on the affected area information at multiple shooting dates and times to facilitate comparison of changes in symptoms of the affected area.

[0206] Despite Fig.14 In the embodiment, after the affected area is photographed, the user is prompted to check the subject ID and select the site where the affected area is located. However, the check of the subject ID and the selection of the site where the affected area is located may be performed before the affected area is photographed.

[0207] Fig.15 2 is a flowchart showing a modification of the operation of the image processing system 11 according to the third embodiment. Fig.15 In the steps described, it is assumed that Fig.11 The steps in the Fig.11 The same processing as in the corresponding steps in .

[0208] When the barcode label 103 is photographed in step 1101 , the communication unit 219 transmits image data generated by photographing the barcode label 103 to the image processing apparatus 300 in step 1501 .

[0209] In step 1511 , the communication unit 313 in the image processing apparatus receives image data generated by photographing the barcode label 103 , which is transmitted from the imaging apparatus 200 .

[0210] In step 1512 , the operation unit 311 performs a process of reading a one-dimensional barcode included in the received image data related to the barcode label 103 to read a subject ID for identifying the subject.

[0211] In step 1513 , the read subject ID is checked against the subject ID registered in the storage unit 312 .

[0212] In step 1514, if the verification of the subject ID is successful, the name of the patient registered in the database in the storage unit 312 is acquired. If the verification fails, information indicating the verification failure is acquired instead of the patient's name.

[0213] In step 1515 , the communication unit 313 in the image processing apparatus transmits the name of the patient or information indicating that the verification of the subject ID has failed to the imaging apparatus 200 .

[0214] In step 1502 , the communication unit 219 in the imaging apparatus 200 receives the patient's name transmitted from the image processing apparatus 300 .

[0215] In step 1503 , the system control circuit 220 displays the patient's name on the display unit 223 .

[0216] In step 1504, the system control circuit 220 displays the name of the patient in the display unit 223. Here, the user may be allowed to input a confirmation result of whether the patient's name is correct. If the patient's name is incorrect or the verification of the patient's name fails, the operation may return to step 1101. Displaying the patient's name before capturing an image of the affected area prevents erroneous association between image data or affected area information related to the affected area to be acquired subsequently and the subject ID.

[0217] In step 1505, the system control circuit 220 displays a user interface on the display unit 223 prompting the user to input information related to the location where the affected area exists in the affected area information. Fig. 8A and Figure 8B In that way, part selection items 801 for designating the part of the affected area: head, shoulder, arm, back, waist, buttocks, and leg are displayed to allow the user to select any one of them.

[0218] In step 1506, the user inputs information related to the affected area. Then, the operation proceeds to step 402. After selecting the information related to the part of the affected area to be photographed in the above manner, the step of photographing the affected area is entered to prevent erroneous selection of the information related to the part of the affected area.

[0219] Since the subject ID is checked in step 1513, the image processing device 300 does not need to check the subject ID after acquiring the image data including the affected area. In addition, since the information related to the location of the affected area is input in step 1506, the user does not need to input the information related to the location of the affected area in steps 1507 and 1508 after acquiring the image data including the affected area, and it is sufficient for the user to input the evaluation value of each evaluation item in steps 1507 and 1508.

[0220] As described above, in the image processing system 11 according to the present embodiment, image data related to the affected area 102 and analysis results of the image data can be recognized and stored for each subject, and whether each evaluation item is alleviated or worsened can be confirmed using only the camera in the user's hand. Therefore, the user can confirm the registered management information related to the affected area using only the camera in the user's hand immediately after photographing the affected area. In addition, displaying the severity level currently confirmed in comparison with the last management information enables the user to confirm at a glance whether the symptom is alleviated or worsened.

[0221] The user can confirm the analysis result of the image data related to the affected area 102 in association with the subject ID and the name of the subject from the terminal device 1000 such as a tablet terminal using a web browser or a dedicated application.

[0222] In all the above embodiments, by installing a circuit corresponding to the auxiliary operation unit 317 in the camera device 200, the image processing can be realized using only the camera device 200. Figure 4 , Fig. 9 and Fig.11In this case, the same effect as that in the image processing system composed of the image processing device 200 and the image processing device 300 described above is achieved using only the image capturing device 200. Receiving a new learning model created in an external computer enables improvement in the accuracy of the inference processing of the affected area and extraction of a new type of affected area.

[0223] (Other embodiments)

[0224] The present invention can be implemented by providing a program for implementing one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium and causing one or more processors in a computer of the system or device to read out and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0225] The present invention is not limited to the above embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to make the public aware of the scope of the present invention, the following claims are made.

[0226] This application claims priority from Japanese Patent Application No. 2018-104922 filed on May 31, 2018, Japanese Patent Application No. 2019-018653 filed on February 5, 2019, and Japanese Patent Application No. 2019-095938 filed on May 22, 2019, which are hereby incorporated by reference herein.

Claims

1. An image processing system, comprising a camera device and an image processing device, characterized in that: The camera device comprises: A camera component, configured to receive light from a subject to generate image data, a first communication component for outputting the image data to a communication network, and A display component is used to display an image based on the image data generated by the camera component, and the image processing device includes: a second communication component for acquiring the image data via the communication network, and a computing unit configured to extract an affected area of ​​the subject from the image data, wherein the second communication unit outputs information indicating an extraction result of the affected area extracted by the computing unit to the communication network, The first communication means acquires information indicating the extraction result of the affected area via the communication network, automatically inputting evaluation values ​​of each of one or more evaluation items among a plurality of predetermined evaluation items based on information indicating the extraction result of the affected area, the display section performs display based on information indicating the extraction result of the affected area and performs display for causing a user to input evaluation values ​​of the predetermined plurality of evaluation items in the affected area at the imaging device, and The display section performs display for causing a user to input evaluation values ​​of the predetermined plurality of evaluation items other than the automatically input evaluation values, and the evaluation values ​​input at the imaging device are transmitted to the image processing device via the communication network.

2. The image processing system according to claim 1, characterized in that The display unit displays an image based on the image data on which the extraction result of the affected area is superimposed and which is used by the calculation unit to extract the affected area.

3. The image processing system according to claim 1, characterized in that: The display unit displays a live view image on which the extraction result of the affected area is superimposed and which is generated by the imaging unit.

4. The image processing system according to claim 1, characterized in that: The calculation unit generates information indicating the size of the affected area extracted from the image data, and The second communication means outputs the information indicating the size generated by the operation means to the communication network.

5. The image processing system according to claim 4, characterized in that: The imaging device comprises a generating unit configured to generate distance information related to a distance from the imaging device to the subject. the first communication component outputting the distance information to the communication network, The second communication component acquires the distance information via the communication network, and The calculation unit generates information indicating a size of the affected area based on the distance information.

6. The image processing system according to claim 4, characterized in that: The display section performs display based on information indicating the extraction result of the affected area and information indicating the size.

7. The image processing system according to claim 4, characterized in that: The information indicating the size of the affected area is at least one of the following items: the length of the affected area in at least two directions, the area of ​​the affected area, the area of ​​a circumscribed rectangular area surrounding the affected area, and a scale bar for measuring the size of the affected area.

8. The image processing system according to claim 5, characterized in that: The calculation section converts the size of the affected area on the image data based on information indicating the angle of view or the size of pixels of the image data and the distance information to generate information indicating the size of the affected area.

9. The image processing system according to claim 1, characterized in that: The calculation unit recognizes information indicating the size of the affected area for each subject having the affected area, and stores the recognized information in the storage unit.

10. The image processing system according to claim 9, characterized in that: The calculation section identifies information indicating the size of the affected area based on the subject having the affected area and the date and time when the image data used in the extraction of the affected area was generated, and stores the identified information in the storage section.

11. The image processing system according to claim 9, characterized in that: The computing unit transmits information indicating the size of the affected area corresponding to the object specified in the request to the terminal device in response to a request from the external terminal device.

12. The image processing system according to claim 9, characterized in that: The second communication section also acquires, via the communication network, image data including a code for identifying the subject output from the first communication section, and The calculation unit extracts information for identifying the subject having the affected area from image data including a code for identifying the subject.

13. The image processing system according to claim 1, characterized in that: The calculation unit causes a second display unit different from the display unit to display information indicating a result of extraction of the affected area.

14. The image processing system according to claim 13, characterized in that: The arithmetic means causes the second display means to arrange to display an image based on the image data on which the extraction result of the affected area is superimposed and an image based on the image data acquired by the second communication means.

15. The image processing system according to claim 1, characterized in that The display section causes a user to input evaluation values ​​of the plurality of evaluation items in response to acquisition of information indicating the extraction result of the affected area.

16. A camera device, comprising: An imaging component, used for receiving light from a subject to generate image data; a communication component for outputting the image data to an external device via a communication network; as well as A display component is used to display an image based on the image data generated by the camera component, characterized in that: the communication means acquires information indicating a result of extraction of an affected area of ​​the subject in the image data from the external device via the communication network, automatically inputting evaluation values ​​of each of one or more evaluation items among a plurality of predetermined evaluation items based on information indicating the extraction result of the affected area, the display section performs display based on information indicating the extraction result of the affected area and performs display for causing a user to input evaluation values ​​of the predetermined plurality of evaluation items in the affected area at the imaging device, and The display section performs display for causing a user to input evaluation values ​​of the predetermined plurality of evaluation items other than the automatically input evaluation values, and the evaluation values ​​input at the imaging device are transmitted to the external device via the communication network.

17. The imaging device according to claim 16, wherein: The display unit displays an image based on the image data on which the extraction result of the affected area is superimposed and output to the external device.

18. The imaging device according to claim 16, wherein: The display unit displays a live view image on which the extraction result of the affected area is superimposed and which is generated by the imaging unit.

19. The imaging device according to claim 16, wherein: the communication means acquires information indicating the size of the affected area in the image data from the external device via the communication network, and The display section performs display based on information indicating the extraction result of the affected area and information indicating the size.

20. The imaging device according to claim 19, further comprising: A generating component, used to generate distance information related to the distance from the camera device to the subject, characterized in that: The communication section outputs the distance information to the external device via the communication network.

21. The imaging device according to claim 19, wherein: The information indicating the size of the affected area is at least one of the following items: the length of the affected area in at least two directions, the area of ​​the affected area, the area of ​​a circumscribed rectangular area surrounding the affected area, and a scale bar for measuring the size of the affected area.

22. The imaging device according to claim 16, wherein: The communication unit outputs information for identifying the subject having the affected area to the external device via the communication network.

23. The imaging device according to claim 16, wherein: The display section causes a user to input evaluation values ​​of the plurality of evaluation items in response to acquisition of information indicating the extraction result of the affected area.

24. A control method for an image processing system, the image processing system comprising an image pickup device and an image processing device, the image pickup device comprising an image pickup component, a display component and a first communication component, and the image processing device comprising a computing component and a second communication component, characterized in that: The control method comprises: receiving light from a subject through the imaging component to generate image data; outputting the image data to a communication network via the first communication component; acquiring the image data via the communication network by the second communication component; extracting the affected area of ​​the subject from the image data by the computing unit; outputting information indicating a result of extraction of the affected area to the communication network through the second communication means; acquiring, by the first communication means, information indicating the extraction result of the affected area via the communication network, and automatically inputting evaluation values ​​of each of one or more evaluation items among a plurality of predetermined evaluation items based on the information indicating the extraction result of the affected area; and The display component displays information based on the extraction result indicating the affected area and allows a user to input evaluation values ​​of the predetermined multiple evaluation items in the affected area at the camera device, wherein the display includes allowing a user to input evaluation values ​​of the predetermined multiple evaluation items in addition to the automatically input evaluation values, and the evaluation values ​​input at the camera device are sent to the image processing device via the communication network.

25. A method for controlling a camera device, characterized in that: The control method comprises: receiving light from a subject to generate image data; outputting the image data to an external device via a communication network; acquiring information indicating an extraction result of an affected area of ​​the subject in the image data from the external device via the communication network, and automatically inputting evaluation values ​​of each of one or more evaluation items among a plurality of predetermined evaluation items based on the information indicating the extraction result of the affected area; and The display component is caused to display information based on the extraction result indicating the affected area and to display evaluation values ​​of the predetermined multiple evaluation items in the affected area so that the user can input them at the camera device, wherein the display component is caused to display evaluation values ​​other than automatically input evaluation values ​​of the predetermined multiple evaluation items so that the user can input them, and the evaluation values ​​input at the camera device are sent to the external device via the communication network.

26. A computer-readable non-volatile storage medium storing instructions for causing a computer to perform the steps of a method for controlling an imaging device, wherein: The control method of the camera device comprises: receiving light from a subject to generate image data; outputting the image data to an external device via a communication network; acquiring information indicating an extraction result of an affected area of ​​the subject in the image data from the external device via the communication network, and automatically inputting evaluation values ​​of each of one or more evaluation items among a plurality of predetermined evaluation items based on the information indicating the extraction result of the affected area; and The display component is caused to display information based on the extraction result indicating the affected area and to display evaluation values ​​of the predetermined multiple evaluation items in the affected area so that the user can input them at the camera device, wherein the display component is caused to display evaluation values ​​other than automatically input evaluation values ​​of the predetermined multiple evaluation items so that the user can input them, and the evaluation values ​​input at the camera device are sent to the external device via the communication network.

Images