Remote image display system, remote image display method, and program

Abstract

To provide a remote image display system, a remote image display method, and a program capable of improving accuracy of remote medical care and on-line counseling for cosmetics inexpensively and easily by acquiring a spectrum image of a subject without using a spectrum camera.SOLUTION: A remote image display system for displaying, in an observer's terminal, a captured image of a subject captured at a place different from a place where an observer executes the display includes: a spectral reflection factor estimation unit for estimating a spectral reflection factor of the subject from the captured image of the subject captured with an imaging device for capturing an RGB image; a spectrum image generation unit for generating a spectrum image from a spectrum in a predetermined wavelength of the estimated spectral reflection factor; and an output control unit for displaying, in the observer's terminal, at least one of the captured image and the spectrum image according to the observer's selection.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a remote image display system, a remote image display method, and a program. [Background technology]

[0002] In recent years, there has been a growing demand for online services that utilize systems (hereinafter also referred to as "remote image display systems") that allow observers to remotely view images of subjects online, such as telemedicine (online medical consultations) and online cosmetic consultations. For example, in telemedicine, a patient takes an image of the affected area (an example of an image of the subject) at a location remote from the medical institution, and a doctor (an example of an observer) provides medical treatment based on that image displayed on a terminal at the medical institution. As a result of the increasing demand for such online services, various technologies have been proposed for displaying images taken at remote locations.

[0003] For example, Patent Document 1 discloses a technology that displays on a terminal in a remote hospital an image taken by a patient at home of the subject to be diagnosed, and an image showing the characteristics of predetermined data obtained from the image data obtained by quantifying each pixel of the said image. In this technology, predetermined data necessary for diagnosis, such as blood concentration, its oxygen saturation, and the level of pigment deposition such as melanin, are calculated from the color data of each of the four wavelengths (R (Red), G (Green), B (Blue), IR (Infrared)) for each pixel.

[0004] However, the technology described in Patent Document 1 does not perform spectral analysis, so it is not possible to view the spectrum of a specific wavelength. In visual examination, viewing spectral images for each detailed wavelength can improve the accuracy of the examination. Therefore, for example, in order to improve the accuracy of telemedicine, it is preferable that the spectral image of the image of the patient's affected area be displayed on the terminal of the medical institution.

[0005] Patent Document 2, described below, discloses a technique for evaluating the skin of a subject using a spectral reflectance image (spectral image) obtained by imaging the subject's face with a spectral camera. By using a spectral camera as in this technique, it is possible to easily obtain spectral images of, for example, the affected area of ​​a patient. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 3417235 [Patent Document 2] Patent No. 6001245 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, spectral cameras are expensive and have a complex structure, limiting their installation environments. For example, it is difficult to install spectral cameras in patients' homes or in ambulances. Therefore, there is a need to improve the accuracy of telemedicine inexpensively and easily by acquiring spectral images of subjects without using spectral cameras.

[0008] In view of the above-mentioned problems, the object of the present invention is to provide a remote image display system, a remote image display method, and a program that can improve the accuracy of telemedicine and online cosmetic counseling inexpensively and easily by acquiring spectral images of a subject without using a spectral camera. [Means for solving the problem]

[0009] To solve the above-mentioned problems, a remote image display system according to one aspect of the present invention is a remote image display system that displays an image of a subject taken at a location different from the place where the observer is observing on an observer terminal, comprising: a spectral reflectance estimation unit that estimates the spectral reflectance of the subject from an image captured by an imaging device that captures an RGB image; a spectral image generation unit that generates a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength; and an output control unit that displays at least one of the image of the captured image or the spectral image on the observer terminal according to the observer's selection. The spectral reflectance estimation unit estimates the spectral reflectance of the subject from the captured image at predetermined wavelength intervals, and the spectral image generation unit generates the spectral image at the specified wavelength when the wavelength of the spectral image to be displayed on the observer terminal is specified. .

[0010] A remote image display method according to one aspect of the present invention is a remote image display method for displaying an image of a subject taken at a location different from the location where the observer is performing the observation, on an observer terminal, and includes: a spectral reflectance estimation unit estimating the spectral reflectance of the subject from an image captured by an imaging device that captures an RGB image; a spectral image generation unit generating a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength; and an output control unit displaying at least one of the image, either the captured image or the spectral image, on the observer terminal according to the observer's selection. The spectral reflectance estimation unit estimates the spectral reflectance of the subject from the captured image at predetermined wavelength intervals, and the spectral image generation unit generates the spectral image at the specified wavelength when the wavelength of the spectral image to be displayed on the observer terminal is specified. .

[0011] A program according to one aspect of the present invention is a program that causes a computer to function as a remote image display system that displays captured images of a subject taken at a location different from the location where the observer is observing on an observer terminal, wherein the computer functions as: spectral reflectance estimation means for estimating the spectral reflectance of the subject from captured images taken by an imaging device that captures RGB images; spectral image generation means for generating a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength; and output control means for displaying at least one of the captured images or the spectral image on the observer terminal according to the observer's selection. The spectral reflectance estimation means estimates the spectral reflectance of the subject from the captured image at predetermined wavelength intervals, and the spectral image generation means generates the spectral image at the specified wavelength when the wavelength of the spectral image to be displayed on the observer terminal is specified. . [Effects of the Invention]

[0012] According to the present invention, by acquiring a spectral image of a subject without using a spectral camera, it is possible to improve the accuracy of telemedicine and online counseling for cosmetics at low cost and easily.

Brief Description of the Drawings

[0013] [Figure 1] It is a diagram showing an example of the configuration of a telemedicine system according to an embodiment of the present invention. [Figure 2] It is a diagram for explaining the estimation of spectral reflectance according to an embodiment of the present invention. [Figure 3] It is a diagram for explaining the estimation of spectral reflectance according to an embodiment of the present invention. [Figure 4] It is a diagram for explaining the estimation of spectral reflectance according to an embodiment of the present invention. [Figure 5] It is a diagram for explaining the estimation of spectral reflectance according to an embodiment of the present invention. [Figure 6] It is a diagram showing an example of the display of an affected part image on a doctor terminal according to an embodiment of the present invention. [Figure 7] It is a sequence diagram showing an example of the processing flow in a telemedicine system according to an embodiment of the present invention. [Figure 8] It is a flowchart showing an example of the image display processing flow in an image display system for telemedicine according to an embodiment of the present invention. [Figure 9] It is a flowchart showing an example of the display switching processing flow in an image display system for telemedicine according to an embodiment of the present invention. [Figure 10] It is a diagram showing a first modification example of the display of an affected part image on a doctor terminal according to an embodiment of the present invention. [Figure 11] It is a flowchart showing a first modification example of the image display processing flow in an image display system for telemedicine according to an embodiment of the present invention. [Figure 12] It is a diagram showing a second modification example of the display of an affected part image on a doctor terminal according to an embodiment of the present invention. [Figure 13] This figure shows a second modified example of the display of a diseased area image in a physician's terminal according to an embodiment of the present invention. [Figure 14] This flowchart shows a second modified example of the image display processing flow in the image display system for telemedicine according to an embodiment of the present invention. [Figure 15] This figure shows a modified example of the affected area image display screen according to an embodiment of the present invention. [Figure 16] This figure shows an example of a detailed information display screen according to an embodiment of the present invention. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described in detail below with reference to the drawings. The present invention relates to a system (hereinafter also referred to as a "remote image display system") that allows an observer to remotely observe images of a subject being observed online. This remote image display system can be used, for example, in telemedicine (online medical consultation) or online counseling for cosmetics. In the following embodiment, an example of how the remote image display system is used in telemedicine will be described.

[0015] <1. Configuration of the telemedicine system> Referring to Figure 1, the configuration of a telemedicine system will be described as an example of this embodiment. Figure 1 is a diagram showing an example of the configuration of a telemedicine system according to an embodiment of the present invention. The telemedicine system 1000 is a system for a doctor to provide telemedicine to a patient. In this embodiment, telemedicine is the act of providing medical care to a doctor and a patient at a distance using information and communication technology such as the internet. In the present invention, the doctor corresponds to the observer and the patient corresponds to the subject. In the following, this embodiment will describe an example in which a patient at their home (PH) uses the telemedicine system 1000 to receive telemedicine from a doctor at a hospital (HS) located in a remote location.

[0016] As shown in Figure 1, the telemedicine system 1000 comprises an imaging device 10, a patient terminal 20, a telemedicine image display system 30 (an example of a remote image display system), and a physician terminal 40 (an example of an observer terminal). The imaging device 10 and the patient terminal 20 are located at the patient's home (PH). The physician terminal 40 is located at the hospital (HS). In the telemedicine system 1000, the patient terminal 20, the telemedicine image display system 30, and the physician terminal 40 are each connected by a network NW.

[0017] (1) Imaging device 10 The imaging device 10 is an imaging device that captures RGB images. Examples of imaging devices include digital cameras, webcams, and video cameras. The patient uses the imaging device 10 to image the affected area. The images of the patient's affected area captured by the imaging device 10 are output to the remote medical image display system 30 via the patient terminal 20. The imaging device 10 may be a camera independent of the patient terminal 20, or it may be a camera integrated into the patient terminal 20. If the imaging device 10 is a camera independent of the patient terminal 20, the patient outputs the image captured by the imaging device 10 to the patient terminal 20 using wired communication, wireless communication, or a storage medium, etc. Furthermore, the number of imaging devices 10 used by the patient to image the affected area is not limited to one, but may be multiple.

[0018] (2) Patient terminal 20 The patient terminal 20 is a device used by the patient to receive remote medical consultations from a doctor. The patient terminal 20 can be, for example, a PC (Personal Computer), a smartphone, or a tablet device. The patient terminal 20 is connected to the remote medical consultation image display system 30 via a network NW. The patient accesses the remote medical consultation image display system 30 from the patient terminal 20 and uploads images of the affected area taken using the imaging device 10 to the remote medical consultation image display system 30. The patient receives remote medical consultation from a doctor based on the images uploaded to the remote medical consultation image display system 30. As shown in Figure 1, the patient terminal 20 includes a communication unit 21, an input unit 22, an output unit 23, a storage unit 24, and a control unit 25.

[0019] (2-1) Communications Department 21 The communication unit 21 has the function of sending and receiving various types of information. For example, the communication unit 21 transmits images and audio of the patient's affected area to the remote medical image display system 30. The communication unit 21 also receives images transmitted from the remote medical image display system 30. The images received by the communication unit 21 are, for example, images and audio of a doctor performing remote medical treatment at hospital HS.

[0020] (2-2) Input section 22 The input unit 22 has the function of receiving input from the patient. The input unit 22 is implemented by an input device such as a keyboard, mouse, or touch panel. This input device may be a device that is pre-installed as hardware in the patient terminal 20, or it may be a device that is externally connected to the patient terminal 20.

[0021] (2-3) Output section 23 The output unit 23 has the function of outputting various types of information. The output unit 23 is implemented, for example, by a display device such as a display or an audio output device such as a speaker. The display device and audio output device may be devices that are pre-installed as hardware in the patient terminal 20, or they may be devices that are externally connected to the patient terminal 20. For example, the output unit 23 may display an image of the patient's affected area captured by the imaging device 10 on the display device, or in telemedicine, display an image of the doctor on the display device and output the doctor's voice from the audio output device.

[0022] (2-4) Storage section 24 The storage unit 24 has the function of storing various types of information. The storage unit 24 is composed of a storage medium, such as an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read / write Memory), ROM (Read Only Memory), or any combination of these storage media.

[0023] (2-5) Control Unit 25 The control unit 25 has the function of controlling the overall operation of the patient terminal 20. The control unit 25 is implemented, for example, by causing the CPU (Central Processing Unit) that the patient terminal 20 has as hardware to execute a program.

[0024] (3) Image display system for telemedicine 30 The telemedicine image display system 30 is a system that displays images of the affected area of ​​a patient receiving telemedicine from a physician at a location different from where the physician is conducting the telemedicine, on the physician's terminal 40 used by the physician for telemedicine. The telemedicine image display system 30 is, for example, a terminal such as a server device. The telemedicine image display system 30 is connected to the patient terminal 20 and the physician terminal 40 via a network NW. As shown in Figure 1, the remote medical image display system 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0025] (3-1) Communications Department 31 The communication unit 31 has the function of sending and receiving various types of information. For example, the communication unit 31 receives images and audio of the patient's affected area from the patient terminal 20, and receives images, audio, and information entered by the doctor from the doctor terminal 40. The communication unit 31 writes the images of the patient's affected area received from the patient terminal 20 to the past image database 324 (an example of a past image storage unit in the claim) and stores them. Writing to the past image database 324 may be done automatically in the background, for example, or it may be done based on the doctor's operation. Furthermore, the communication unit 41 transmits information to be output to the physician terminal 40 (for example, images and audio of the affected area of ​​the patient) to the physician terminal 40, and transmits information to be output to the patient terminal 20 (for example, images and audio of the physician) to the patient terminal 20.

[0026] (3-2) Storage section 32 The storage unit 32 has the function of storing various types of information. The storage unit 32 is composed of a storage medium, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. As shown in Figure 1, the storage unit 32 includes an imaging device spectral sensitivity database 321, a spectral reflectance database 322, a light source database 323, and a past image database 324.

[0027] (3-2-1) Imaging device spectral sensitivity database 321 The imaging device spectral sensitivity database 321 has pre-written and stored pairs of identification information for each imaging device 10 and the spectral sensitivity of the imaging device 10 indicated by that identification information.

[0028] (3-2-2) Spectroscopic Reflectance Database 322 The spectral reflectance database 322 has pre-written and stored pairs of identification information for each color sample and the spectral reflectance of the color sample indicated by that identification information, or identification information for each imaging target (e.g., the surface of the human body) and the spectral reflectance of the imaging target indicated by that identification information.

[0029] If the spectral reflectance of the color sample is stored in the spectral reflectance database 322, the patient will use the imaging device 10 to image both the affected area and the color sample together.

[0030] (3-2-3) Light Source Database 323 The light source database 323 stores pre-written pairs of identification information for each light source and the spectral distribution of that light source (light source spectral distribution) indicated by that identification information. Examples of light sources include sunlight and fluorescent lamps.

[0031] (3-2-4) Past Image Database 324 The past image database 324 stores images of the affected area of ​​patients taken in the past, previously estimated spectral reflectances, estimated spectral intensity results for each wavelength of said spectral reflectance, and images corresponding to various cases.

[0032] (3-3) Control unit 33 The control unit 33 has the function of controlling the overall operation of the telemedicine image display system 30. The control unit 33 is implemented, for example, by causing the CPU, which is provided as hardware in the telemedicine image display system 30, to execute a program. As shown in Figure 1, the control unit 33 includes a data acquisition unit 331, a spectral reflectance estimation unit 332, an image selection unit 333, a spectral image generation unit 334, an output control unit 335, and a display switching unit 336.

[0033] (3-3-1) Data acquisition unit 331 The data acquisition unit 331 has the function of acquiring various types of data. For example, the data acquisition unit 331 acquires data necessary for estimating spectral reflectance from various databases. Specifically, the data acquisition unit 331 acquires the spectral sensitivity of the imaging device 10 used by the patient to image the affected area from the imaging device spectral sensitivity database 321. The data acquisition unit 331 also acquires the spectral reflectance of a color sample or the spectral reflectance of the object being imaged from the spectral reflectance database 322. Furthermore, the data acquisition unit 331 acquires the spectral distribution of the light source when the patient imaged the affected area from the light source database 323. Finally, the data acquisition unit 331 acquires image data from the past image database 324.

[0034] (3-3-2) Spectral reflectance estimation section 332 The spectral reflectance estimation unit 332 has the function of estimating the spectral reflectance of the affected area from the image of the affected area of ​​the patient captured by the imaging device 10. The spectral reflectance estimation unit 332 estimates the spectral reflectance for each pixel of the image. The spectral reflectance estimation unit 332 associates the estimated spectral reflectance with the image and writes it to the past image database 324 for storage. Specifically, the spectral reflectance estimation unit 332 estimates the spectral reflectance of the patient's affected area from the spectral sensitivity of the imaging device (the spectral sensitivity of each RGB image) stored in the imaging device spectral sensitivity database 321, the basis vector group stored in the spectral reflectance database 322, the spectral distribution of the imaging light source stored in the light source database 323, and the image data of the patient's affected area captured in the RGB image database 324 by the data acquisition unit 331.

[0035] In this embodiment, the spectral reflectance estimation unit 332 estimates the spectral reflectance of the affected area at predetermined wavelength intervals from the image of the affected area acquired by the data acquisition unit 331, for example. As an example, the spectral reflectance estimation unit 332 estimates the spectral reflectance at 10 nm intervals in the range of 400 nm to 700 nm. That is, the spectral reflectance estimation unit 332 samples wavelength units from 400 nm to 700 nm in 10 nm increments and estimates the spectral reflectance as a 31-dimensional vector.

[0036] Here, we will explain in detail the process for estimating spectral reflectance. Generally, when an object is imaged using a general-purpose imaging device (hereinafter referred to as a camera), the brightness value p at the image coordinate x of the acquired image data is... m,k (x) is expressed by the following equation (1), where m is the number assigned to the camera, k is the camera channel, s(λ,x) is the spectral reflectance of wavelength λ at image coordinate x, l(λ) is the spectral distribution of wavelength λ in the light source, c m,k (x) represents the spectral sensitivity of channel k of camera m at image coordinate x. Note that the channel indicates the color elements that the camera can capture; for example, if the camera captures an RGB image, it represents R, G, and B.

[0037]

number

[0038] As shown in equation (1), the brightness value p of channel k of camera m at image coordinate x m,k (x) is the spectral reflectance s(λ,x) of channel k of camera m at image coordinate x, the spectral distribution l(λ) of the light source, and the spectral sensitivity c of channel k of camera m at image coordinate x. m,k The product with (x) is expressed as the integral value obtained by integrating with respect to the wavelength λ. According to Reference 1 (Do It Yourself Hyperspectral Imaging with Everyday Digital Cameras, Seoung Wug Oh, 2016 IEEE Conference on Computer Vision and Pattern Recognition), the spectral distribution l(λ) of a light source is defined by the following equation (2) using the spectral distribution basis vector and basis coefficients. Here, a j e is the basis coefficient multiplied by the j-th basis vector. j The values ​​shown represent the j-th spectral distribution basis vector.

[0039]

number

[0040] (2) In the formula, according to Reference 1, N a = 6 is used. N a When N j = 6, the spectral distribution l(λ) of the light source is expressed as data obtained by adding the products of each basis coefficient a

[0041] and the 1st to 6th spectral distribution basis vectors. That is, in this case, the spectral distribution l(λ) of the light source is expressed using the 1st to 6th principal components.

[0042] According to Reference 1, the spectral reflectance s(λ, x) of the object at the image coordinate x is defined by the following formula (3) using the spectral reflectance basis vector and the basis coefficient. Here, r j (x) is the basis coefficient multiplied by the j-th basis vector at the image coordinate x, and b j is the j-th spectral distribution basis vector, respectively.

[0043]

Equation

[0044] (3) In the formula, according to Reference 1, N r = 8 is used. N rWhen =8, the spectral reflectance s(λ,x) of the object at image coordinate x is obtained by applying the respective basis coefficients r from the first spectral reflectance basis vector to the eighth spectral reflectance basis vector. j It is expressed as data obtained by adding up the results of multiplying by (x). In other words, in this case, the spectral reflectance s(λ,x) of the object is expressed using the first to eighth principal components.

[0045] Here, the number of principal components used to represent the spectral reflectance s(λ,x) of the object can be arbitrarily determined according to the computational processing capability of the control unit 33 and the required accuracy of the spectral reflectance data to be represented, as long as it does not exceed the upper limit of the number of principal components that can be treated.

[0046] The upper limit of the number of principal components that can be treated in spectral reflectance data is determined, for example, based on the number of imaging conditions (the spectral sensitivity of the camera during imaging). This upper limit is the number obtained by multiplying the number of cameras by the number of channels and subtracting 1, when RGB images are captured by multiple cameras, each with different spectral sensitivities. For example, when RGB images are captured by three cameras, the upper limit is (number of cameras = 3) × (number of channels = 3) - 1, which is 8.

[0047] By applying equations (2) and (3) described above to equation (1), the brightness value p at image coordinate x can be obtained. m,k (x) is expressed by equation (4) below. Here, the bolded A represents the integral term represented by the integral symbol replaced by a matrix. A m,k (i,j) represents the (i,j) component in the integral term with respect to channel k of camera m.

[0048]

number

[0049] By replacing the term indicated by the Σ (sigma) symbol in equation (4) with a vector (matrix product), we obtain the brightness value p at image coordinate x as shown in equation (5) below. m,k(x) can be expressed as a determinant. Here, the bolded r is a matrix representing the basis coefficients of each of the spectral reflectance basis vectors at image coordinate x, the bolded a is a matrix representing the basis coefficients of each of the spectral distribution basis vectors, and the bolded A represents each of the integral terms for channel k of camera m, N r ×N a This shows the matrix.

[0050]

number

[0051] The left side of equation (5) is the brightness value p at image coordinate x. m,k (x) is shown, and the right-hand side shows the spectral information. Brightness value p m,k (x) represents the RGB values ​​of a single pixel in an RGB image. In other words, equation (5) shows the relationship between the RGB values ​​of a single pixel and the spectral information. In contrast to equation (5), the relationship between the RGB values ​​of each of the n pixels constituting an RGB image and the spectral information is expressed by the following equation (6). Here, the bolded p is a matrix representing the luminance values ​​at each image coordinate, and the bolded R represents the basis coefficients of each of the spectral reflectance basis vector groups at each image coordinate. r A matrix of ×n, where the bolded 'a' represents the matrix showing the basis coefficients for each of the spectral distribution basis vectors, and the bolded A represents the integral terms for each of the channels k of camera m, N. r ×N a This shows the matrix. Note that n represents the total number of pixels in the RGB image.

[0052]

number

[0053] The spectral reflectance estimation unit 332 estimates, using the least squares method, a matrix R (basis coefficients for spectral reflectance) and a matrix a (basis coefficients for the spectral distribution of the light source) for the image data acquired from the data acquisition unit 331, such that equation (6) above holds true. The spectral reflectance estimation unit 332 can estimate the spectral reflectance s(λ,x) of the affected area of ​​the patient by substituting the estimated matrix R into equation (3). Furthermore, the spectral reflectance estimation unit 332 can estimate the spectral distribution l(λ) of the imaging light source by substituting the estimated matrix a into equation (2).

[0054] Here, with reference to Figures 2 to 5, the method by which the spectral reflectance estimation unit 332 estimates the matrix R and matrix a in equation (6) using the least squares method will be explained. Figures 2 to 5 are diagrams illustrating the estimation of spectral reflectance according to an embodiment of the present invention.

[0055] Figure 2 shows equation (7), which is obtained by applying the least squares method to equation (6). In equation (7), the bold R with a hat on the left side represents the estimated basis coefficient of the spectral reflectance, and the bold a with a hat represents the estimated basis coefficient of the spectral distribution. The right side of equation (7) shows the combination of matrix R and matrix a that minimizes the sum of the squares of the difference (distance in vector space) between the left and right sides of equation (6) for all channels and all cameras.

[0056] The spectral reflectance estimation unit 332 calculates the brightness value of the RGB image in the image data acquired from the data acquisition unit 331 using the brightness value p of equation (7). m,k Substitute the values ​​into the equation. The spectral reflectance estimation unit 332 also substitutes the spectral distribution basis vectors, spectral reflectance basis vectors obtained from the spectral reflectance database 322, and the spectral sensitivity in the image data obtained from the data acquisition unit 331 into matrix A of equation (7). Then, the spectral reflectance estimation unit 332 solves equation (7) with the values ​​substituted into it to calculate the estimated values ​​of the basis coefficients for spectral reflectance and the basis coefficients for spectral distribution.

[0057] Figure 3 shows equation (8), which is equation (7) with a term indicating smoothing added. In equation (8), the smoothing parameters α and β are coefficients multiplied by the term indicating smoothing (smoothing term), and are arbitrary positive real numbers determined according to the degree (strength) of smoothing.

[0058] In the least squares method, the results are highly susceptible to disturbances such as noise. Therefore, if there are errors in the image data acquired from the data acquisition unit 331, the spectral reflectance estimation unit 332 will have difficulty making an appropriate estimation even after solving equation (7). For this reason, the spectral reflectance estimation unit 332 may solve equation (8), which adds a smoothing term to equation (7). Here, the smoothing term may consist of a term relating to the spectral reflectance s(λ,x) and a term relating to the spectral distribution l(λ). The smoothing term relating to the spectral reflectance s(λ,x) is obtained by summing the integral of the square of the second derivative of the spectral reflectance s(λ,x) with respect to wavelength λ over all pixels. The smoothing term relating to the spectral distribution l(λ) is obtained by summing the integral of the square of the second derivative of the spectral distribution l(λ) with respect to wavelength λ over all pixels.

[0059] The spectral reflectance estimation unit 332 can perform a more appropriate estimation even when there are errors in the image data by solving equation (8), which is equation (7) with a smoothing term added. The smoothing term may consist of both a smoothing term relating to spectral reflectance and a smoothing term relating to spectral distribution, or it may consist of only one of them.

[0060] Figure 4 shows equation (9), which is equation (8) with constraints applied. In equation (8), the spectral reflectance s(λ,x) and spectral distribution l(λ) can never be negative. Therefore, as shown in equation (9), the spectral reflectance estimation unit 332 imposes the constraint that both the spectral reflectance s(λ,x) and spectral distribution l(λ) in equation (8) must be positive values, and calculates estimated values ​​for the basis coefficients of the spectral reflectance and the spectral distribution. This allows for estimation that excludes cases where at least one of the spectral reflectance s(λ,x) and spectral distribution l(λ) is negative, even if theoretically possible.

[0061] Figure 5 shows equation (10), which is equation (9) in matrix form. In equation (10), the smoothing term with respect to the spectral reflectance s(λ,x) is expressed as the product of the second differential matrix W, the spectral reflectance basis vector matrix B, and the spectral reflectance basis coefficient matrix R. The smoothing term with respect to the spectral distribution l(λ) is expressed as the product of the second differential matrix W, the spectral distribution basis vector matrix E, and the spectral distribution basis coefficient matrix a. Here, matrix B v,i This represents the i-th spectral reflectance basis vector in band v. Matrix E v,j ∫ represents the j-th spectral distribution basis vector in band v. The bands represent the spectrally separated frequency bands in the spectral image, and consist of 31 bands, for example, obtained by sampling the frequency band from 400 nm to 700 nm at 10 nm intervals.

[0062] In general, it is difficult to solve both matrix R and matrix a simultaneously for equation (9). Therefore, the spectral reflectance estimation unit 332 alternately optimizes matrix R and matrix a to converge both matrices R and a.

[0063] (3-3-3) Image selection section 333 The image selection unit 333 has the function of selecting an image to be displayed on the physician's terminal 40. For example, the image selection unit 333 selects an image selected by the physician as the image to be displayed on the physician's terminal 40.

[0064] Specifically, when a physician selects an image, the image selection unit 333 selects the image to be displayed on the physician's terminal 40. In this case, the image captured by the imaging device 10 is acquired as the image to be displayed on the physician's terminal 40.

[0065] On the other hand, if the physician selects a spectral image, the image selection unit 333 selects the spectral image as the image to be displayed on the physician's terminal 40. In this case, the spectral image generated by the spectral image generation unit 334, which will be described later, is acquired as the spectral image to be displayed on the physician's terminal 40.

[0066] In this embodiment, the physician can select two images to be displayed on the physician terminal 40 on the affected area image display screen, which will be described later. If the physician selects a spectral image on the affected area image display screen, they can specify the wavelength of spectral reflectance used to generate the spectral image to be displayed on the physician terminal 40.

[0067] (3-3-4) Spectral image generation unit 334 The spectral image generation unit 334 has the function of generating a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength. The spectral image generation unit 334 associates the generated spectral image with the captured image and writes it to the past image database 324 for storage.

[0068] For example, if the wavelength of the spectral image to be displayed on the physician terminal 40 is specified, the spectral image generation unit 334 generates a spectral image at the specified wavelength. As an example, suppose the physician specifies 540 nm as the wavelength of the spectral image to be displayed on the physician terminal 40. In this case, the spectral image generation unit 334 generates a spectral image from the spectral reflectance at the wavelength of 540 nm estimated by the spectral reflectance estimation unit 332. Furthermore, if the wavelength of the spectral image to be displayed on the physician terminal 40 is specified within a predetermined range, the spectral image generation unit 334 generates a spectral image from the integral value of the spectrum at each wavelength within the specified range. For example, suppose a physician specifies the wavelength of the spectral image to be displayed on the physician terminal 40 to be in the range of 540 nm to 550 nm. In this case, the spectral image generation unit 334 generates a spectral image from the integral value of the spectral reflectance at wavelength 540 nm estimated by the spectral reflectance estimation unit 332 and the spectrum at wavelength 540 nm.

[0069] Furthermore, if the spectral reflectance at the wavelength specified by the physician has not been estimated by the spectral reflectance estimation unit 332, the spectral image generation unit 334 generates a spectral image based on the spectral reflectance near the specified wavelength from among the estimated spectral reflectances. For example, suppose a doctor specifies 543 nm as the wavelength for the spectral image to be displayed on the doctor's terminal 40. In this embodiment, the spectral reflectance estimation unit 332 estimates the spectral reflectance at 10 nm intervals within the range of 400 nm to 700 nm. Therefore, the spectral reflectance at a wavelength of 543 nm is not estimated by the spectral reflectance estimation unit 332. In this case, the spectral image generation unit 334 generates a spectral image using a weighted average of the spectral reflectances at 540 nm and 550 nm, which are near the wavelength of 543 nm.

[0070] The weighting values ​​for each spectral reflectance can be set to any value. For example, if the specified wavelength is 543 nm, the wavelength of 543 nm is closer to the wavelength of 540 nm. In this case, the spectral image generation unit 334 sets a weight for the spectral reflectance at wavelength 540 nm that is larger than the weight for the spectral reflectance at wavelength 550 nm. Furthermore, if the specified wavelength is 545 nm, then 545 nm is an intermediate value between 540 nm and 550 nm. In this case, the spectral image generation unit 334 sets the same value for the weight of the spectral reflectance at 540 nm and the weight of the spectral reflectance at 550 nm.

[0071] The spectral image generation unit 334 may generate a spectral image by colorizing the spectral reflectance, or it may generate a spectral image by converting the light source into an RGB image according to the light source selected from the light source database 323.

[0072] (3-3-5) Output control unit 335 The output control unit 335 has a function to control the output of the affected area image to be displayed on the physician terminal 40. For example, the output control unit 335 displays at least one of the captured image or spectral image as the affected area image on the physician terminal 40, according to the physician's selection. The output control unit 335 may display the affected area image received by the communication unit 31 from the patient terminal 20 directly on the physician terminal 40, or it may retrieve the affected area image stored in the past image database 324 and display it on the physician terminal 40. In this embodiment, the physician can select two images of the affected area to be displayed on the affected area image display screen shown on the display screen of the physician terminal 40. However, it is assumed that there is only one display area on the affected area image display screen where the affected area images are displayed. That is, in the initial display of the affected area image display screen after the physician has selected two images, the output control unit 335 displays only one of the two selected images in the display area. Furthermore, images of affected areas that were not displayed in the display area during the initial display can be displayed in the display area of ​​the affected area image display screen after the initial display by the function of the display switching unit 336 described later.

[0073] (3-3-6) Display switching section 336 The display switching unit 336 has the function of switching the display of affected area images to be shown on the physician's terminal 40. For example, suppose there is only one display area for images selected by the physician on the affected area image display screen, and multiple images are selected by the physician to be displayed in that display area. In this case, the display switching unit 336 switches and displays the multiple images selected by the physician in one display area. This configuration allows doctors to conduct remote consultations while comparing multiple images of the affected area.

[0074] For example, suppose a doctor selects two images of the affected area to be displayed on the affected area image display screen: an image captured by a physician and a spectral image. In this case, the display switching unit 336 alternately switches between displaying the image captured by a physician and the spectral image in the display area. The display switching unit 336 switches the affected area image displayed in the display area when, for example, a display switching operation is input by a doctor. For example, suppose a display switching button is provided on the affected area image display screen. In this case, the display switching unit 336 switches the affected area image displayed in the display area each time the doctor performs a display switching operation by pressing the display switching button.

[0075] Furthermore, the display switching unit 336 may sequentially switch the images of the affected area displayed in the display area at predetermined time intervals, even without input from a physician regarding the display switching operation. With this configuration, the display switching unit 336 can eliminate the need for a physician to input a display switching operation.

[0076] (4) Doctor's Terminal 40 The physician terminal 40 is a terminal used by a physician to provide remote medical care to a patient. The physician terminal 40 can be, for example, a PC, smartphone, or tablet. The physician terminal 40 is connected to the remote medical image display system 30 via a network NW. The physician accesses the remote medical image display system 30 from the physician terminal 40 and provides remote medical care to the patient based on the images of the affected area of ​​the patient displayed on the physician terminal 40. As shown in Figure 1, the physician terminal 40 includes a communication unit 41, an input unit 42, an output unit 43, a storage unit 44, and a control unit 45.

[0077] (4-1) Communications Department 41 The communication unit 41 has the function of sending and receiving various types of information. For example, the communication unit 41 transmits images of a doctor to the remote medical image display system 30. The communication unit 41 also receives images and audio transmitted from the remote medical image display system 30. The images received by the communication unit 41 include, for example, images of the affected area of ​​a patient receiving remote medical treatment and audio of the patient.

[0078] (4-2) Input section 42 The input unit 42 has the function of receiving input from a physician. The input unit 42 is implemented by an input device such as a keyboard, mouse, or touch panel. This input device may be a device that is pre-installed as hardware in the physician terminal 40, or it may be a device that is externally connected to the physician terminal 40. The physician operates the input unit 42 to make various inputs to the screen displayed on the output unit 43. For example, the physician accesses the telemedicine image display system 30 and inputs identification information to identify the patient being treated. This patient identification information may include, for example, a patient ID number or a patient number assigned during emergency medical care. In addition, if multiple affected areas exist for the same patient, the physician may input an affected area number in addition to the patient number to identify the affected area. The physician also inputs an operation to select the type of affected area image to be displayed on the output unit 43.

[0079] (4-3) Output section 43 The output unit 43 has the function of outputting various types of information. The output unit 43 is implemented, for example, by a display device such as a display or an audio output device such as a speaker. The display device and audio output device may be devices that are pre-installed as hardware in the physician terminal 40, or they may be devices that are externally connected to the physician terminal 40. For example, in telemedicine, the output unit 43 displays an image of the patient's affected area on the display device and outputs the patient's voice from the audio output device.

[0080] Now, with reference to Figure 6, the display of the affected area image in the physician terminal 40 according to this embodiment will be described. Figure 6 is a diagram showing an example of the display of the affected area image in the physician terminal 40 according to an embodiment of the present invention. For example, the affected area image display screen 50 shown in Figure 6 is displayed on the physician terminal 40.

[0081] On the affected area image display screen 50, the "Image 1" selection field 51 is a field for selecting whether to display the image of the patient's affected area captured by the imaging device 10, or a spectral image generated based on the spectral reflectance estimated from the said image. Field 52 is for entering the minimum wavelength of the spectral image to display when a spectral image is selected in field 51. Field 53 is for entering the maximum wavelength of the spectral image to display, when a spectral image is selected in field 51. The input format in fields 52 and 53 can be, for example, text input or a dropdown menu. In the case of a dropdown menu, the wavelength can be selected at predetermined wavelength intervals (e.g., 10 nm intervals).

[0082] If only the minimum wavelength in column 52 is set and no setting is made in column 53, a color-scaled image of the spectral image of the wavelength in column 52 will be displayed. On the other hand, if the minimum wavelength and maximum wavelength are set in columns 52 and 53 respectively, an image will be displayed that is a color scale of the integral value of the spectral image at each wavelength from column 52 to column 53, i.e., the integrated value of the spectrum of each pixel.

[0083] Similarly, the selection field 54 for "Image 2" is a field for selecting whether to display the image of the patient's affected area captured by the imaging device 10, or the spectral image generated based on the spectral reflectance estimated from the said image. Field 55 is for entering the minimum wavelength of the spectral image to display when a spectral image is selected in field 54. Field 56 is for entering the maximum wavelength of the spectral image to display, when a spectral image is selected in field 54. The input format in fields 55 and 56 can be, for example, text input or a dropdown menu. In the case of a dropdown menu, the wavelength can be selected at predetermined wavelength intervals (e.g., 10 nm intervals).

[0084] If only the minimum wavelength in column 55 is set and no setting is made in column 56, a color-scaled image of the spectral image of the wavelength in column 55 will be displayed. On the other hand, if the minimum wavelength and maximum wavelength are set in columns 55 and 56 respectively, an image will be displayed that is a color scale of the integral values ​​of the spectral images for each wavelength from column 55 to column 56, i.e., the integrated values ​​of the spectra of each pixel.

[0085] The display area 57 is the area where either the display image 58a selected in the selection field 51 or the display image 58b selected in the selection field 54 is displayed.

[0086] The display switching button 59 switches the image displayed in the display area 57 each time it is pressed with a mouse or the like. For example, each time the display switching button 59 is pressed with a mouse or the like, it sequentially switches between the display image 58a selected in the selection field 51 and the display image 58b selected in the selection field 54, and displays them in the display area 57.

[0087] Note that the combination of Image 1 and Image 2 on the affected area image display screen 50 is not limited to the combination of captured image and spectral image shown in Figure 6. For example, the combination of Image 1 and Image 2 may be a combination of spectral image and captured image, a combination of captured image and captured image, or a combination of spectral image and spectral image.

[0088] (4-4) Storage section 44 The storage unit 44 has the function of storing various types of information. The storage unit 44 is composed of a storage medium, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media.

[0089] (4-5) Control Unit 45 The control unit 45 has the function of controlling the overall operation of the physician terminal 40. The control unit 45 is implemented, for example, by causing the CPU, which is provided as hardware in the physician terminal 40, to execute a program.

[0090] <2. Processing Flow> Next, the processing flow according to this embodiment will be explained with reference to Figures 7 to 9. In the processing flow described below, it is assumed that there are two images that can be selected on the affected area image display screen, and one affected area image that can be displayed on the affected area image display screen.

[0091] (1) Processing flow in the telemedicine system First, the processing flow in the telemedicine system 1000 will be explained with reference to Figure 7. Figure 7 is a sequence diagram showing an example of the processing flow in the telemedicine system 1000 according to an embodiment of the present invention.

[0092] As shown in Figure 7, first the patient performs an imaging operation on the imaging device 10 (step S101). The imaging device 10 receives an imaging operation from the patient and images the affected area of ​​the patient (step S102). After imaging, the imaging device 10 transmits the image of the patient's affected area to the patient terminal 20 (step S103). The patient accesses the remote medical image display system 30 from the patient terminal 20. The patient operates the upload screen displayed on the patient terminal 20 to upload the captured images to the remote medical image display system 30 (step S104). When a user inputs an upload command, the patient terminal 20 transmits the captured image received from the imaging device 10 to the remote medical image display system 30 via the network NW from the communication unit 21 (step S105).

[0093] The communication unit 31 of the remote medical image display system 30 writes the captured images received from the patient terminal 20 to the past image database 324 for storage (step S106). The data acquisition unit 331 acquires the data necessary for estimating the spectral reflectance from each database (step S107). Specifically, the data acquisition unit 331 acquires the spectral sensitivity of the imaging device 10 from the imaging device spectral sensitivity database 321, the spectral reflectance of the object being imaged from the spectral reflectance database 322, the spectral distribution of the light source from the light source database 323, and the captured image from the past image database 324. The spectral reflectance estimation unit 332 estimates the spectral reflectance of the affected area of ​​the patient shown in the captured image based on the data acquired by the data acquisition unit 331 (step S108).

[0094] After the patient uploads images of the affected area to the remote medical image display system 30, the physician accesses the remote medical image display system 30 from the physician's terminal 40. The physician operates the affected area image display screen displayed on the physician's terminal 40 to select the images to be displayed on the affected area image display screen (step S111). The physician's terminal 40, which has received an image selection operation from the physician, transmits selection information indicating the information of the image selected by the physician to the remote medical image display system 30 (step S112). The remote medical image display system 30 performs image display processing based on the selection information received from the physician's terminal 40 (step S113). Details of the image display processing will be described later. After image display processing, the remote medical image display system 30 displays the image selected by the physician as the affected area image on the physician's terminal 40 (step S114).

[0095] After the image of the affected area is displayed on the physician terminal 40, the physician operates the affected area image display screen displayed on the physician terminal 40 to switch the display of the affected area image (step S121). When a physician inputs a display switching operation, the physician terminal 40 sends switching information indicating that the physician has performed the display switching operation to the remote medical image display system 30 (step S122). The remote medical image display system 30 performs a display switching process based on the switching information received from the physician's terminal 40 (step S123). Details of the display switching process will be described later. After the display switching process, the remote medical image display system 30 displays the switched image as the affected area image on the doctor's terminal 40 (step S124).

[0096] (2) Image display processing flow in the telemedicine image display system 30 Next, referring to FIG. 8, the flow of image display processing in the remote medical image display system 30 will be described. FIG. 8 is a flowchart showing an example of the flow of image display processing in the remote medical image display system 30 according to an embodiment of the present invention. In the image display processing, the constant N indicates the number of selectable images on the affected part image display screen. The variable n indicates the number of times the image selection process has been performed.

[0097] As shown in FIG. 8, first, the image selection unit 333 performs an initialization process (step S201) to select the image selected by the doctor. In the initialization process, the number of selectable images (for example, 2) on the affected part image display screen is set for the constant N, and 0 is set for the variable n. The image selection unit 333 determines whether n < N (step S202). If n < N (step S202 / YES), the image selection unit 333 proceeds to step S203. On the other hand, if n < N is not satisfied (step S202 / NO), the image selection unit 333 proceeds to step S207.

[0098] In step S203, the image selection unit 333 determines whether the type of the image n selected by the doctor is a captured image. If the type of the image n selected by the doctor is a captured image (step S203 / YES), the image selection unit 333 proceeds to step S204. On the other hand, if the type of the image n selected by the doctor is not a captured image (step S203 / NO), the image selection unit 333 proceeds to step S205.

[0099] In step S204, the data acquisition unit 331 acquires the captured image from the past image database 324 and proceeds to step S206. In step S205, the spectral image generation unit 334 generates a spectral image based on the spectral reflectance estimated by the spectral reflectance estimation unit 332 and proceeds to step S206. In step S206, the image selection unit 333 increments the value of the variable n by 1. After the increment, the image selection unit 333 repeats the process from step S202 until n < N is no longer true.

[0100] In step S207, the output control unit 335 causes the image selected by the doctor as Image 1 to be displayed on the doctor terminal 40. After the display of Image 1, the image display process ends. Note that the output control unit 335 may cause the image selected by the doctor as Image 2 to be displayed on the doctor terminal 40.

[0101] (3) Flow of display switching process in the remote medical image display system 30 Next, referring to FIG. 9, the flow of the display switching process in the remote medical image display system 30 will be described. FIG. 9 is a flowchart showing an example of the flow of the display switching process in the remote medical image display system 30 according to an embodiment of the present invention.

[0102] As shown in FIG. 9, first, the display switching unit 336 determines whether Image 1 is being displayed on the doctor terminal 40 (step S301). If Image 1 is being displayed on the doctor terminal 40 (step S301 / YES), the display switching unit 336 switches the image to be displayed on the doctor terminal 40 from Image 1 to Image 2 (step S302). On the other hand, if Image 2 is being displayed on the doctor terminal 40 (step S301 / NO), the display switching unit 336 switches the image to be displayed on the doctor terminal 40 from Image 2 to Image 1 (step S303). After the image switching, the display switching process ends.

[0103] As described above, the remote medical image display system 30 according to the present embodiment is a system that causes a diseased part image showing the diseased part of a patient receiving remote medical treatment by a doctor to be displayed on the doctor terminal 40 used by the doctor for remote medical treatment at a location different from the location where the doctor conducts the remote medical treatment. The remote medical image display system 30 includes a spectral reflectance estimation unit 332, a spectral image generation unit 334, and an output control unit 335. The spectral reflectance estimation unit 332 estimates the spectral reflectance of the affected area from the image of the affected area captured by the imaging device 10, which captures RGB images. The spectral image generation unit 334 generates a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength. The output control unit 335, according to the physician's selection, causes at least one of the captured image or spectral image to be displayed on the physician's terminal 40 as an image of the affected area.

[0104] With this configuration, the telemedicine image display system 30 according to this embodiment can generate spectral images from RGB images, eliminating the need for an expensive spectral camera to acquire spectral images. Furthermore, patients and doctors do not need to prepare equipment for using a spectral camera, making the system easily accessible. In addition, doctors can improve the accuracy of telemedicine because the spectral images generated by the functions of the telemedicine image display system 30 provide a consistent set of characteristics when visualizing the color of the patient's affected area.

[0105] Therefore, the remote medical image display system 30 according to this embodiment makes it possible to improve the accuracy of remote medical care inexpensively and easily by acquiring spectral images of the affected area of ​​the patient without using a spectral camera.

[0106] <3. Variant> Embodiments of the present invention have been described above. Next, modifications of the embodiments of the present invention will be described. Each modification described below may be applied to the embodiments of the present invention individually or in combination. Furthermore, each modification may be applied in place of the configuration described in the embodiments of the present invention, or it may be applied in addition to the configuration described in the embodiments of the present invention.

[0107] (1) First variation In the embodiments described above, an example was described in which there is only one display area for the image selected by the physician on the affected area image display screen, but the invention is not limited to such an example. For example, there may be multiple display areas for the image selected by the physician on the affected area image display screen. In this case, if the doctor has selected multiple images to be displayed in the display area, the output control unit 335 will display each of the selected images in its corresponding display area. With this configuration, multiple images of the affected area are displayed on the affected area image display screen. This allows doctors to perform remote consultations while comparing multiple images of the affected area without having to switch between displays.

[0108] Here, with reference to Figures 10 and 11, a first modified example of the display of affected area images on the physician terminal 40 and the image display processing flow in the telemedicine image display system 30 will be described. In the example shown in Figures 10 and 11, it is assumed that there are two images that can be selected on the affected area image display screen, and two affected area images that can be displayed on the affected area image display screen.

[0109] (1-1) Display of affected area image on physician terminal 40 Figure 10 shows a first modified example of the display of an image of a diseased area in a physician's terminal 40 according to an embodiment of the present invention. The functions of selection fields 61, 62, 63, 64, 65, and 66 in the affected area image display screen 60 shown in Figure 10 are the same as those of selection fields 51, 52, 53, 54, 55, and 56 in the affected area image display screen 50, which were explained with reference to Figure 6, so their explanation is omitted.

[0110] The display area 67a is the area where the display image 68a selected in the selection field 61 is displayed. In the example shown in Figure 10, the display image 68a is the captured image. The display area 67b is the area where the display image 68b selected in the selection field 64 is displayed. In the example shown in Figure 10, the display image 68b is a spectral image.

[0111] Note that the combination of Image 1 and Image 2 on the affected area image display screen 60 is not limited to the combination of captured image and spectral image shown in Figure 10. For example, the combination of Image 1 and Image 2 may be a combination of spectral image and captured image, a combination of captured image and captured image, or a combination of spectral image and spectral image.

[0112] The affected area image display screen 60 shown in Figure 10 can display all affected area images selected by the physician. Therefore, the affected area image display screen 60 does not have a display switching button.

[0113] (1-2) Image display processing in image display systems for telemedicine Figure 11 is a flowchart showing a first modified example of the image display processing flow in the image display system 30 for telemedicine according to an embodiment of the present invention. The processes from steps S401 to S406 shown in Figure 11 are the same as the processes from steps S201 to S206 explained with reference to Figure 8, so their explanation will be omitted.

[0114] In the flowchart shown in Figure 11, step S407, which is a different process from the flowchart shown in Figure 8, the output control unit 335 displays the image selected by the doctor as image 1 and the image selected by the doctor as image 2 on the doctor's terminal 40.

[0115] (2) Second variation In the above-described embodiment, there were two types of images that could be selected on the affected area image display screen: captured images and spectral images. However, the system is not limited to this example. For example, the affected area image display screen may allow selection of images showing the progress of the patient's affected area or images showing search results related to symptoms (e.g., search results for cases).

[0116] The affected area image display screen allows the user to select an image showing the progress of the patient's affected area. In this case, the past image database 324 stores images of the patient's affected area taken in the past, associating them with the spectral reflectance estimated from those images. When the physician selects a past image of the patient (i.e., an image showing progress) as the affected area image they want to display on the physician terminal 40, the output control unit 335 retrieves the corresponding image based on the information stored in the past image database 324 and displays it on the physician terminal 40. For example, if an image taken is selected as an image showing progress, the output control unit 335 retrieves the corresponding image taken from the past image database 324 and displays it on the physician terminal 40. On the other hand, if a spectral image is selected as an image showing progress, the output control unit 335 retrieves the spectral image generated by the spectral image generation unit 334 based on the spectral reflectance stored in the past image database 324 and displays it on the physician terminal 40.

[0117] With this configuration, the affected area image display screen shows two images of the same patient, but taken at different times. For example, an image showing the current state of the patient's affected area and an image showing the patient's affected area in the past are displayed side by side. This allows the doctor to observe the progression of the patient's affected area by comparing the displayed images. The progression of the patient's affected area includes, for example, whether the symptoms have improved or worsened, and whether the treatment with medication is effective. Furthermore, past images of the affected area may be of the same person as the patient being treated, or they may be of a different person.

[0118] Furthermore, the affected area image display screen allows the user to select an image that represents the search results for a case. In this case, the past image database 324 stores the captured image, the spectral reflectance estimated from the captured image, and the case (symptoms) related to the affected area shown in the captured image in association with each other. The physician enters a case related to the image of the affected area that they want to display on the physician terminal 40 into the affected area image display screen. In this case, the output control unit 335 searches the past image database 324 using the entered case as a key and obtains an image to be displayed on the physician terminal 40 based on the information associated with the entered case. The output control unit 335 then displays the obtained image on the physician terminal 40. For example, if an captured image is selected as the image that represents the search results for a case, the output control unit 335 obtains the captured image associated with the entered case from the past image database 324 and displays it on the physician terminal 40. On the other hand, if a spectral image is selected as the image representing the search results for a case, the output control unit 335 acquires the spectral image generated by the spectral image generation unit 334 based on the spectral reflectance associated with the input case and displays it on the physician terminal 40.

[0119] With this configuration, the affected area image display screen will show, for example, an image of the patient's affected area alongside images illustrating various case studies. For example, the affected area image of the patient being treated and the affected area image of a different patient will be displayed side-by-side in two separate display frames for comparison. This allows for comparisons such as comparing the patient being treated with a healthy person, or comparing them with patients whose condition has already been confirmed through medical examination. This allows the doctor to determine whether the patient's affected area matches the case shown in the comparison images by comparing the displayed images. In other words, the doctor can provide medical care by referring to past cases. Furthermore, the doctor can improve the accuracy of determining the patient's affected area by repeatedly searching for cases and comparing them with many images.

[0120] Here, with reference to Figures 12 to 14, a second modified example of the display of affected area images on the physician terminal 40 and the image display processing flow in the telemedicine image display system 30 will be described. In the example shown in Figures 12 to 14, it is assumed that there are two images that can be selected on the affected area image display screen, and two affected area images that can be displayed on the affected area image display screen.

[0121] (2-1) Display of affected area image on physician terminal 40 Figures 12 and 13 show a second modified example of the display of affected area images in the physician terminal 40 according to an embodiment of the present invention. Figure 12 shows an example where an image showing the progress of the patient's affected area is selected as Image 2. Figure 13 shows an example where an image showing the search results for a case is selected as Image 2.

[0122] The functions of the selection fields 71, 72, 73, and display area 76a in the affected area image display screen 70 shown in Figure 12, and the display image 77a displayed in the display area 76a, are the same as the functions of the selection fields 61, 62, 63, and display area 67a in the affected area image display screen 60, and the display image 68a displayed in the display area 67a, as explained with reference to Figure 10, so their explanation is omitted.

[0123] The selection field 74 for "Image 2" is a field for selecting whether to display an image of the patient's affected area captured by the imaging device 10, a spectral image generated based on the spectral reflectance estimated from the said image, or a past image of the patient's affected area (i.e., an image showing the progress). For past images of the patient's affected area, it may be possible to select in selection field 74 whether to display an image or a spectral image. Selection field 75 is for selecting the image to display as an image indicating follow-up observation when follow-up observation is selected in selection field 74. When displaying an image as an image indicating follow-up observation, the physician selects an image indicating follow-up observation in selection field 74, and then operates the dialog box that appears when the icon in selection field 75 is pressed to select the image file to be displayed in display area 76b. Alternatively, the physician may set the image file to be displayed in display area 76b by dragging and dropping it into selection field 75. When displaying a spectral image as an image indicating follow-up observation, the physician selects the spectral image in the same way as when displaying an image. As a result, the selected spectral image is displayed in display area 76b. Alternatively, even when displaying a spectral image in display area 76b as an image indicating follow-up observation, the physician may select an image in selection field 75. In this case, the spectral image generated based on the spectral reflectance associated with the selected image will be displayed in display area 76b. Display area 76b is the area where the display image 77b selected in selection field 75 is displayed. In the example shown in Figure 12, display image 77b is the image taken during the previous medical examination.

[0124] Furthermore, the combination of Image 1 and Image 2 on the affected area image display screen 70 is not limited to the combination of the captured image and the follow-up image shown in Figure 12. For example, the combination of Image 1 and Image 2 may be any combination of two types selected from the three types: the captured image, the spectral image, and the follow-up image.

[0125] The functions of the selection fields 81, 82, 83, and display area 86a in the affected area image display screen 80 shown in Figure 13, and the display image 87a displayed in display area 86a, are the same as the functions of the selection fields 61, 62, 63, and display area 67a in the affected area image display screen 60, and the display image 68a displayed in display area 67a, as explained with reference to Figure 10, so their explanation is omitted.

[0126] The selection field 84 for "Image 2" is a field for selecting whether to display the image of the patient's affected area acquired by the imaging device 10, the spectral image generated based on the spectral reflectance estimated from the image, or the image showing the search results for the case. The selection field 84 may also allow the user to select whether to display the image showing the search results for the case, either the image or the spectral image. The search field 85 is where the doctor enters the case to be searched for when the case search is selected in the selection field 84. The doctor enters the case in the search field 85, for example, by text input or a dropdown menu. To display an image as an image representing the search results for the case, the doctor selects, for example, the image representing the search results for the case in the selection field 84, and then enters the case in the search field 85. As a result, the image associated with the entered case is displayed in the display area 86b. To display a spectral image as an image representing the search results for the case, the doctor selects, for example, the spectral image representing the search results for the case in the selection field 84, and then enters the case in the search field 85. As a result, the spectral image generated based on the spectral reflectance associated with the entered case is displayed in the display area 86b. Display area 86b is the area where the display image 87b, which represents the case entered in the search field 85, is displayed. In the example shown in Figure 13, the display image 87b is an image of a patient whose case is a stroke.

[0127] Furthermore, the combination of Image 1 and Image 2 on the affected area image display screen 80 is not limited to the combination of the captured image and the image showing the case search results shown in Figure 13. For example, the combination of Image 1 and Image 2 may be any combination of two types selected from the three types: the captured image, the spectral image, and the image showing the case search results.

[0128] (2-2) Image display processing in image display systems for telemedicine Figure 14 is a flowchart showing a second modified example of the image display processing flow in the image display system 30 for telemedicine according to an embodiment of the present invention. The processes in steps S501 to S504, S506, and S510 shown in Figure 14 are the same as the processes in steps S201 to S204, S205, and S206 described with reference to Figure 8, so their explanation will be omitted.

[0129] In the flowchart shown in Figure 14, in step S505, the image selection unit 333 determines whether the type of image n selected by the physician is a spectral image. If the type of image n selected by the physician is a spectral image (step S505 / YES), the image selection unit 333 proceeds to step S506. On the other hand, if the type of image n selected by the physician is not a spectral image (step S505 / NO), the image selection unit 333 proceeds to step S507.

[0130] In step S507, the image selection unit 333 determines whether the type of image n selected by the physician is an image indicating follow-up observation. If the type of image n selected by the physician is an image indicating follow-up observation (step S507 / YES), the image selection unit 333 proceeds to step S508. On the other hand, if the type of image n selected by the physician is not an image indicating follow-up observation (step S507 / NO), the image selection unit 333 proceeds to step S509.

[0131] In step S508, the data acquisition unit 331 acquires the corresponding captured image as a past image from the past image database 324 and proceeds to step S510. In step S509, the output control unit 335 (or data acquisition unit 331) acquires the image associated with the case searched from the past image database 324, and proceeds to step S510.

[0132] In step S511, the output control unit 335 displays the image selected by the doctor as image 1 and the image selected by the doctor as image 2 on the doctor's terminal 40.

[0133] (3) Third variation In the above-described embodiment, an example was explained in which the captured image obtained from the past image database 324 or the spectral image generated by the spectral image generation unit 334 is displayed directly in the display area of ​​the affected area image display screen. However, the invention is not limited to this example. For example, the output control unit 335 may correct the image before displaying it in the display area.

[0134] Specifically, the output control unit 335 corrects the color of the captured image based on the spectral reflectance estimated by the spectral reflectance estimation unit 332 and light source data relating to the light source at the location where the physician conducts the remote consultation. The light source data relating to the light source at the location where the physician conducts the remote consultation (e.g., the examination room) is pre-written and stored in, for example, the light source database 323. Specifically, the output control unit 335 corrects the color of the captured image to match the color of the medical examination room, based on each of the spectral images (i.e., spectral reflectance) and the spectral distribution of the light source in the examination room used by the doctor. This configuration allows physicians to observe the patient's affected area in the same way as they would in a normal examination environment, without having to worry about color changes due to differences in the spectral distribution of the light source. This enables physicians to make more accurate diagnoses of the condition of the affected area. Furthermore, physicians can shorten examination times and respond more quickly.

[0135] (4) Fourth variation The above-described embodiment explains an example in which a patient receives remote medical treatment at their home, but the invention is not limited to this example. For example, the location where a patient receives remote medical treatment may be an accident scene, a disaster scene, an emergency scene, or inside an ambulance. Furthermore, the person who takes images of the patient's affected area is not limited to the patient themselves, but may be another person present at the scene (for example, an emergency medical technician).

[0136] (5) Fifth variation The above-described embodiment illustrates an example in which a remote image display system is used for telemedicine, but the invention is not limited to this example. The remote image display system may also be used for online counseling for cosmetics. Here, we will describe an online counseling system for cosmetics that uses a remote image display system. In the online counseling system, the person seeking advice consults with a beauty advisor (BA) online about selecting and using cosmetics that are suitable for their skin color and condition. The configuration of the online counseling system can be the same as that of the telemedicine system 1 described above, except that the patient (subject) is replaced with the person seeking advice and the doctor (observer) is replaced with the BA. Furthermore, for the past image database 324, it is sufficient to store images of bare skin or with makeup, associating them with the skin condition at that time (such as the presence or absence of dark circles under the eyes or blemishes). This configuration allows the beauty advisor (BA) to accurately understand the client's skin color and condition.

[0137] (6) Sixth variation In the above-described embodiment, an example was explained in which the affected area image display screen 50 can display and confirm either an image of the patient's affected area or a spectral image generated based on the spectral reflectance estimated from the image. However, the invention is not limited to this example. The affected area image display screen 50 may also display further information associated with the image or spectral image that the physician wishes to display. Information associated with the image or spectral image may include, for example, environmental information and patient information. Environmental information may include, for example, information indicating the time the image was taken, the temperature or room temperature at the shooting location, humidity, and atmospheric pressure. Patient information may include, for example, information indicating the patient's body temperature and blood pressure at the time the image was taken. The color of a patient's skin surface can change depending on the shooting environment and the patient's physical condition. Therefore, being able to check information such as temperature, room temperature, humidity, atmospheric pressure, body temperature, and blood pressure along with the captured image or spectral image allows physicians to provide medical care that takes this information into consideration. In this modified version, it is possible to display environmental information and patient information together with the captured image or spectral image on the affected area image display screen 50. This allows physicians to further improve the accuracy of telemedicine compared to when only the image is displayed.

[0138] The method for acquiring environmental information and patient information is not particularly limited. For example, environmental information and patient information may be information detected by a sensor device capable of detecting each type of information, information measured by the patient using a measuring device capable of measuring each type of information, or information acquired using a service capable of acquiring each type of information. Environmental information and patient information may be input to the patient terminal 20 from each device or service, or input to the patient terminal 20 by the patient. Furthermore, if the patient terminal 20 is a device such as a smartphone that has imaging and measurement functions, the patient terminal 20 itself may acquire environmental information and patient information.

[0139] The environmental information and patient information entered into the patient terminal 20 are uploaded to the telemedicine image display system 30 in association with the captured image, for example, in step S105 shown in Figure 7, and stored in the storage unit 32. Furthermore, in step S111 shown in Figure 7, it is assumed that environmental information or patient information is associated with the image selected by the physician on the affected area image display screen 50. In this case, the output control unit 335 of the telemedicine image display system 30 also displays the environmental information or patient information on the physician's terminal 40. For example, the output control unit 335 displays a button on the affected area image display screen 50, and when the physician presses the button, it displays a screen showing the environmental information or patient information (hereinafter also referred to as the "detailed information display screen") in a separate window. Alternatively, the output control unit 335 may display the environmental information or patient information together with the image on the affected area image display screen 50. Furthermore, in step S121 shown in Figure 7, it is assumed that environmental information or patient information is associated with the image switched by the physician on the affected area image display screen 50. In this case, the telemedicine image display system 30 switches the display of environmental information or patient information along with the displayed image.

[0140] Here, with reference to Figures 15 and 16, an example of the affected area image display screen and detailed information display screen in this modified example will be described. Figure 15 is a diagram showing a modified example of the affected area image display screen according to an embodiment of the present invention. Figure 16 is a diagram showing an example of the detailed information display screen according to an embodiment of the present invention.

[0141] The affected area image display screen 50 shown in Figure 15 is further provided with detailed information buttons 91 and 92 compared to the affected area image display screen 50 shown in Figure 6. Detailed information button 91 is a button for displaying the detailed information display screen for the image selected in the selection field 51. Detailed information button 92 is a button for displaying the detailed information display screen for the image selected in the selection field 54. When the detailed information button 91 or the detailed information button 92 is pressed, the detailed information display screen 93, as shown in Figure 16, is displayed. The detailed information display screen 93 shows an example of patient information, indicating that the patient's body temperature is "36.5℃" and blood pressure is "80~110". The detailed information display screen 93 also shows an example of environmental information, indicating that the ambient temperature / room temperature is "25.0℃" and humidity is "50%".

[0142] Note that the location where the detailed information buttons 91 and 92 are displayed is not limited to the location shown in Figure 15, but may be displayed at any location. Also, patient information and environmental information may be displayed on the affected area image display screen 50 without using a separate window. Furthermore, patient information and environmental information may be displayed by pressing the display image 58a displayed in the display area 57 of the affected area image display screen 50. Furthermore, the examples described with reference to Figures 15 and 16 are not limited to the affected area image display screen 50, but may also be applied to the affected area image display screen 60 shown in Figure 10, the affected area image display screen 70 shown in Figure 12, and the affected area image display screen 80 shown in Figure 13.

[0143] Embodiments of the present invention have been described above. It should be noted that some or all of the configurations of the remote image display system in the above-described embodiments may be implemented using a computer. In this case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into the computer system and executed. Here, "computer system" includes hardware such as an OS and peripheral devices. "Computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Furthermore, "computer-readable recording medium" may also include those that dynamically hold programs for a short period, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period, such as volatile memory inside the computer system acting as a server or client. The program may also be for implementing some of the functions described above, or it may be a program that can implement the functions described above in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0144] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the spirit of this invention. [Explanation of symbols]

[0145] 10…Imaging device 20...Patient terminal 21... Communications Department 22...Input section 23…Output section 24...Storage section 25... Control Unit 30…Image display system for telemedicine 31… Communications Department 32...Storage section 33…Control Unit 40… Doctor's terminal 41... Communications Department 42...Input section 43…Output section 44...Storage section 45... Control Unit 321… Imaging device spectral sensitivity database 322…Spectral Reflectance Database 323...Light Source Database 324…Past Image Database 331...Data acquisition unit 332...Spectral reflectance estimation section 333...Image selection section 334...Spectral image generation unit 335…Output control unit 336...Display switching section 1000... Telemedicine system

Claims

1. A remote image display system that displays images of a subject taken at a location different from where the observer is observing, on the observer's terminal, A spectral reflectance estimation unit estimates the spectral reflectance of the subject from the captured image captured by an imaging device that captures RGB images, A spectral image generation unit generates a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength, An output control unit that, in accordance with the observer's selection, displays at least one of the captured image or the spectral image on the observer's terminal, Equipped with, The spectral reflectance estimation unit estimates the spectral reflectance of the subject from the captured image at predetermined wavelength intervals, The spectral image generation unit generates the spectral image at the specified wavelength when the wavelength of the spectral image to be displayed on the observer terminal is specified. Remote image display system.

2. If there is only one display area for the image selected by the observer, and multiple images are selected by the observer to be displayed in the display area, a display switching unit is provided to switch between and display the multiple images selected by the observer in the single display area. The remote image display system according to claim 1, further comprising:

3. The output control unit, when there are multiple display areas for images selected by the observer and multiple images to be displayed in the display areas have been selected by the observer, displays each of the multiple images selected by the observer in the corresponding display area. The remote image display system according to claim 1 or claim 2.

4. A past image storage unit stores previously captured images of the subject and the spectral reflectance estimated from the captured images in association with each other. Furthermore, When the observer selects a past image of the subject as the image they wish to display on the observer terminal, the output control unit retrieves the corresponding image based on the information stored in the past image storage unit and displays it on the observer terminal. A remote image display system according to any one of claims 1 to 3.

5. A past image storage unit stores the captured image, the spectral reflectance estimated from the captured image, and the symptoms relating to the subject captured in the captured image in association with each other. Furthermore, When the observer inputs symptoms related to an image they wish to display on the observer terminal, the output control unit retrieves the image corresponding to the input symptoms based on the information stored in the past image storage unit and displays it on the observer terminal. A remote image display system according to any one of claims 1 to 3.

6. The spectral image generation unit generates the spectral image from the integral values ​​of the spectrum at each wavelength within the specified range when the wavelength of the spectral image to be displayed on the observer terminal is specified within a predetermined range. The remote image display system according to claim 1.

7. The output control unit corrects the color of the captured image based on the estimated spectral reflectance and light source data relating to the light source at the location where the observer is performing the observation. A remote image display system according to any one of claims 1 to 6.

8. The output control unit causes the observer terminal to display at least one of the following: the time the image was taken, the temperature or room temperature of the location where the image was taken, humidity, or atmospheric pressure. A remote image display system according to any one of claims 1 to 7.

9. The output control unit causes the observer terminal to display at least one of the subject's body temperature and blood pressure at the time the image that the observer wants to display on the observer terminal is captured. A remote image display system according to any one of claims 1 to 7.

10. A remote image display method that displays captured images of a subject taken at a location different from the location where the observer is performing the observation, on the observer's terminal, The spectral reflectance estimation unit estimates the spectral reflectance of the subject from the captured image captured by the imaging device that captures the RGB image, The spectral image generation unit generates a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength, The output control unit, in accordance with the observer's selection, causes at least one of the captured image or the spectral image to be displayed on the observer's terminal; Includes, The spectral reflectance estimation unit estimates the spectral reflectance of the subject from the captured image at predetermined wavelength intervals, The spectral image generation unit generates the spectral image at the specified wavelength when the wavelength of the spectral image to be displayed on the observer terminal is specified. Remote image display method.

11. A program that causes a computer to function as a remote image display system, which displays images of a subject taken at a location different from where the observer is observing, on the observer's terminal. The aforementioned computer, A spectral reflectance estimation means for estimating the spectral reflectance of the subject from the captured image captured by an imaging device that captures RGB images, A spectral image generation means for generating a spectral image from the spectrum of the estimated spectral reflectance at a predetermined wavelength, Output control means for displaying at least one of the captured image or the spectral image on the observer terminal, according to the observer's selection, To make it function as, The spectral reflectance estimation means estimates the spectral reflectance of the subject from the captured image at predetermined wavelength intervals, The spectral image generation means generates the spectral image at the specified wavelength when the wavelength of the spectral image to be displayed on the observer terminal is specified. program.

Images