Softness measurement device and softness measurement method

Abstract

A softness measurement device according to one embodiment of the present invention comprises: an acquisition unit which acquires a measurement value obtained by a force sensor that is attached to a nail of a finger of an operator and measures the pressure of a fingertip on the basis of the strain of the nail, and also acquires a captured image including an RGB image and a depth image when the force sensor is attached and the finger presses a measurement target; a region determination unit which determines a region of the finger in the RGB image; a three-dimensional coordinate calculation unit which calculates the three-dimensional finger coordinate of the finger in the depth image corresponding to the position of the finger which is determined to be the region of the finger; a normal vector calculation unit which calculates a normal vector on the basis of depth information on the depth image when the finger presses the measurement target; a displacement amount calculation unit which calculates, as a displacement amount, a component along the normal vector to the three-dimensional finger coordinate in the captured image at the start of the pressing by the finger; a calculation unit which calculates an index of softness of the measurement target on the basis of the displacement amount, the measurement value, and the radius of curvature of the finger; and a communication control unit which transmits the captured image, the displacement amount, and the index.

Description

Softness measuring device and softness measuring method

[0001] This invention relates to a softness measuring device and a softness measuring method.

[0002] Due to the ongoing depopulation caused by population decline and the worsening shortage of doctors, the traditional model of medical consultation, which relies on proximity and face-to-face interaction, is becoming unsustainable. Therefore, efforts are being made to realize telemedicine, which utilizes ICT (Information and Communication Technology) to overcome spatial and temporal constraints, enabling a single medical professional (e.g., a doctor) to efficiently treat many patients.

[0003] For example, there are attempts to use video calls to visually examine a patient's physical condition from a remote location. In this attempt, doctors can visually examine the patient's physical condition via video call from a remote location. However, observation of physical condition also involves methods other than visual inspection, such as auscultation and palpation, which are not supported. Palpation is important for understanding physical conditions that cannot be judged visually. Furthermore, palpation has a psychological effect of reassuring the patient. Currently, fully remote palpation using robotic hands without human intervention is being explored, but there are challenges in terms of cost and safety, as well as the aforementioned psychological effect. Therefore, the significance of remote palpation through sensory sharing via a collaborator is great.

[0004] One method of palpation in telemedicine involves a collaborator (nurse or caregiver) near the patient. In this method, the collaborator uses a hardness measuring device such as a probe to obtain palpation information, which is then transmitted to the doctor in the remote location. For example, Non-Patent Document 1 discloses a measurement method using a probe-type sensor as a portable and human-applicable softness measurement technology. Furthermore, Non-Patent Document 2 discloses a measurement method using a finger-mounted tactile sensor.

[0005] YAWASA hardness measuring equipment (https: / / tecgihan.co.jp / products / tactile-sensibility-measurement / yawasa / yws / ) Remote palpation system (Hokkaido University) (https: / / www.hokudai.ac.jp / news / pdf / 240110_pr2.pdf)

[0006] In Non-Patent Documents 1 and 2, the practitioner, acting as a collaborator, touches the patient's affected area via a sensor, which impairs their fingertip sensation and prevents them from accurately perceiving the condition of the affected area. Furthermore, they are always limited to a palpation method that involves pressing the sensor surface against the affected area.

[0007] This invention was made in view of the above circumstances, and its purpose is to provide a technology that allows for the measurement of the softness of the affected area being examined in parallel with conventional palpation movements, without impairing the tactile sensation of the collaborator.

[0008] To solve the above problems, a softness measuring device according to one aspect of the present invention includes: an acquisition unit that acquires a measurement value from a force sensor attached to the fingernail of the operator and which measures the pressure of the fingertip by the strain of the nail, and an image including an RGB image and a depth image when the force sensor is attached and the target to be measured is pressed with the finger; a region determination unit that determines the region of the finger in the RGB image; a three-dimensional coordinate calculation unit that calculates the three-dimensional finger coordinates of the finger in the depth image corresponding to the finger region and the position of the finger determined to be the finger; a normal vector calculation unit that calculates a normal vector based on the depth information of the depth image when the target to be measured is pressed with the finger; a displacement amount calculation unit that calculates a component along the normal vector between the three-dimensional finger coordinates of the image and the three-dimensional finger coordinates of the image when the finger pressure is started as a displacement amount; a calculation unit that calculates an index of the softness of the target to be measured based on the displacement amount, the measurement value, and the radius of curvature of the finger; and a communication control unit that transmits the image, the displacement amount, and the index.

[0009] According to one aspect of this invention, it is possible to measure the softness of the affected area being examined in parallel with conventional palpation actions, without impairing the collaborator's sense of touch in their fingertips.

[0010] Figure 1 is a schematic diagram showing an example of the configuration of a softness measurement system according to the embodiment. Figure 2 is a block diagram showing an example of the hardware configuration of a palpation device and a diagnostic device included in the softness measurement system according to the embodiment. Figure 3 is a diagram showing an example of a practitioner's hand equipped with a force sensor according to the embodiment. Figure 4 is a block diagram showing the software configuration of the palpation device according to the embodiment in relation to the hardware configuration of the palpation device 1 shown in Figure 2. Figure 5 is a diagram showing an example of three-dimensional finger coordinates stored in the acupressure start point coordinate storage unit according to the embodiment. Figure 6 is a diagram showing an example of finger curvature radius stored in the finger curvature radius storage unit according to the embodiment. Figure 7 is a diagram showing an example of coefficients and correction coefficients stored in the fingertip physical property value correction coefficient storage unit according to the embodiment. Figure 8 is a block diagram showing the software configuration of the diagnostic device according to the embodiment in relation to the hardware configuration of the diagnostic device shown in Figure 2. Figure 9 is a flowchart showing an example of processing by the palpation device during a palpation operation according to the embodiment. Figure 10 is a diagram showing an example of the hand and palpation finger detected by the region determination unit from an RGB image according to the embodiment, and the acquired palpation finger coordinates. Figure 11 is a diagram showing an example of the position of the palpation finger and the average coordinate calculation target region according to the embodiment. Figure 12 shows an example of a normal vector calculated by the normal vector calculation unit according to the embodiment. Figure 13 shows the displacement amount calculated by the displacement amount calculation unit in the embodiment.

[0011] Next, embodiments of the present invention will be described below with reference to the drawings. In the following embodiments, parts with the same number will be assumed to perform the same function, and therefore repeated explanations will be omitted. For example, when there are multiple identical or similar elements, a common reference numeral may be used to describe each element without distinction, or a sub-number may be used in addition to the common reference numeral to describe each element separately.

[0012] First, the softness measurement system according to the embodiment will be described. (Configuration) Figure 1 is a schematic diagram showing an example of the configuration of the softness measurement system 100 according to the embodiment. In Figure 1, the softness measurement system 100 is used to initiate a video call between a remote location where a physician (DO) is located and a local location where a patient (PA) and a practitioner (OP) such as a nurse or caregiver are located. The practitioner (OP) films the patient (PA), and the physician (DO) gives visual instructions to the practitioner (OP). Based on these instructions, the practitioner (OP) performs a palpation of the patient (PA) using a palpation device, which is a softness measurement device. The palpation device transmits the palpation information obtained from the palpation to a diagnostic device located at the remote location. The diagnostic device transmits the softness based on the palpation information to the physician (DO) via a tactile display (SD). Thus, the softness measurement system 100 is not a remotely operated robot, but a system that supports palpation instructions from a physician (DO) to a practitioner (OP) in a situation where the practitioner (OP) touches the patient (PA) body parts on behalf of the physician (DO) at the remote location.

[0013] Next, the hardware and software configurations of the palpation device 1 and diagnostic device 2 included in the softness measurement system 100 will be described.

[0014] Figure 2 is a block diagram showing an example of the hardware configuration of the palpation device 1 and diagnostic device 2 included in the softness measurement system 100 according to the embodiment. First, the hardware configuration of the palpation device 1 will be described. The palpation device 1, which is a softness measurement device, is a computer that analyzes the input data and generates and outputs output data. For example, the palpation device 1 is a PC, tablet computer, smartphone, wearable device, or AR (Augmented Reality) glasses that can be carried by the operator.

[0015] As shown in Figure 2, the palpation device 1 comprises a control unit 11, a program storage unit 12, a data storage unit 13, a communication interface 14, and an input / output interface 15. The control unit 11, program storage unit 12, data storage unit 13, communication interface 14, and input / output interface 15 are connected to each other via a bus so as to be able to communicate with each other. Furthermore, the communication interface 14 may be connected to the diagnostic device 2 via a network so as to be able to communicate with each other. In addition, the input / output interface 15 is connected to the input device 151, the output device 152, the force sensor PS, and the depth camera DC so as to be able to communicate with each other.

[0016] The control unit 11 controls the palpation device 1. The control unit 11 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 11 may be an integrated circuit capable of executing various programs.

[0017] The program storage unit 12 can use a combination of non-volatile memory that allows writing and reading at any time, such as EPROM (Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), and SSD (Solid State Drive), as a storage medium, and non-volatile memory such as ROM (Read Only Memory). The program storage unit 12 stores programs necessary to execute various processes. In other words, the control unit 11 can realize various controls and operations by reading and executing programs stored in the program storage unit 12.

[0018] The data storage unit 13 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM (Random Access Memory), as storage media. The data storage unit 13 is used to store data acquired and generated during the process in which the control unit 11 executes a program and performs various processing.

[0019] The communication interface 14 includes one or more wired or wireless communication modules. For example, the communication interface 14 may include a communication module that connects via a wired connection to an external device such as the diagnostic device 2 over a network. The communication interface 14 may also include a wireless communication module that connects wirelessly to an external device such as the diagnostic device 2. Furthermore, the communication interface 14 may include a wireless communication module that uses short-range wireless technology to connect wirelessly to an external device. In other words, the communication interface 14 can be any general communication interface that can communicate with the diagnostic device 2, etc., under the control of the control unit 11 and send and receive various information.

[0020] The input / output interface 15 is connected to the input device 151, output device 152, force sensor PS, depth camera DC, etc. The input / output interface 15 is an interface that enables the transmission and reception of information between the input device 151, output device 152, force sensor PS, and depth camera DC. The input / output interface 15 may be integrated with the communication interface 14. For example, the palpation device 1 and at least one of the input device 151, output device 152, force sensor PS, and depth camera DC may be wirelessly connected using short-range wireless technology, and information may be transmitted and received using said short-range wireless technology.

[0021] The input device 151 may include, for example, a keyboard or pointing device for the operator (OP) to input various information to the palpation device 1. The input device 151 may also include a reader for reading data to be stored in the program storage unit 12 or data storage unit 13 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0022] The output device 152 includes a display that shows display images, including physician DO, captured by the diagnostic device 2, based on the results calculated by the control unit 11. The output device 152 also includes a printer that prints the information displayed on the display.

[0023] The force sensor PS is a sensor attached to the fingernail of the surgeon (OP). The force sensor PS measures the pressure at the fingertip based on the strain of the surgeon's fingernail.

[0024] Figure 3 shows an example of a surgeon's (OP) hand equipped with a force sensor PS according to the embodiment. As shown in Figure 3, the force sensor PS is attached to the surgeon's (OP) fingernail. By using such a force sensor PS, the surgeon (OP) can directly touch the affected area of ​​the patient (PA) during palpation, without hindering the sensation in their fingertips. Another advantage is that the method of acupressure when palpating the affected area is not significantly restricted.

[0025] The depth camera DC is a camera capable of capturing RGB images and depth images. The depth camera DC may be a so-called RGBD camera. Alternatively, the depth camera DC may be a camera that combines an RGB camera capable of capturing RGB images with a depth camera capable of capturing depth images. Alternatively, the depth camera DC may be a camera other than a stereo camera, such as a LiDAR camera. In other words, the depth camera DC can be any camera capable of capturing both RGB images and depth images.

[0026] Next, the hardware configuration of the diagnostic device 2 will be described. The diagnostic device 2 is a computer that analyzes the input data, generates output data, and outputs it. For example, the diagnostic device 2 may be a PC, tablet computer, smartphone, wearable device, or AR (Augmented Reality) glasses that can be used by the physician (DO).

[0027] As shown in Figure 2, the diagnostic device 2 comprises a control unit 21, a program storage unit 22, a data storage unit 23, a communication interface 24, and an input / output interface 25. The control unit 21, program storage unit 22, data storage unit 23, communication interface 24, and input / output interface 25 are connected to each other via a bus so as to be able to communicate with one another. Furthermore, the communication interface 24 may be connected to the diagnostic device 2 via a network so as to be able to communicate with one another. In addition, the input / output interface 25 is connected to an input device 251, an output device 252, and a tactile display SD so as to be able to communicate with one another.

[0028] The control unit 21 controls the palpation device 1. The control unit 21 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 21 may be an integrated circuit capable of executing various programs.

[0029] The program storage unit 22 can use a combination of non-volatile memory that allows writing and reading at any time, such as EPROM (Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), and SSD (Solid State Drive), as a storage medium, and non-volatile memory such as ROM (Read Only Memory). The program storage unit 22 stores programs necessary to execute various processes. In other words, the control unit 21 can realize various controls and operations by reading and executing programs stored in the program storage unit 22.

[0030] The data storage unit 23 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM (RanDOM Access Memory), as storage media. The data storage unit 23 is used to store data acquired and generated during the process in which the control unit 21 executes a program and performs various processing.

[0031] The communication interface 24 includes one or more wired or wireless communication modules. For example, the communication interface 24 may include a communication module that connects via a network to an external device such as a palpation device 1 via a wired connection. The communication interface 24 may also include a wireless communication module that connects wirelessly to an external device such as a diagnostic device 2. Furthermore, the communication interface 24 may include a wireless communication module that uses short-range wireless technology to connect wirelessly to an external device. In other words, the communication interface 24 can be any general communication interface that can communicate with the diagnostic device 2, etc., under the control of the control unit 21 and send and receive various types of information.

[0032] The input / output interface 25 is connected to an input device 251, an output device 252, a tactile display SD, etc. The input / output interface 25 is an interface that enables transmission and reception of information between the input device 251, the output device 252, and the tactile display SD. The input / output interface 25 may be integrated with the communication interface 24. For example, the diagnostic device 2 and at least one of the input device 251, the output device 252, and the tactile display SD are wirelessly connected using a short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology.

[0033] The input device 251 may include, for example, a keyboard, a pointing device, etc. for a doctor DO to input various information to the diagnostic device 2. The input device 251 may include a camera capable of photographing the doctor DO. Further, the input device 251 may include a reader for reading data to be stored in the program storage unit 22 or the data storage unit 23 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0034] The output device 252 includes a display for displaying the result calculated by the control unit 21, a photographed image including the patient PA photographed by the palpation device 1, etc. The output device 252 also includes a printer for printing the information displayed on the display.

[0035] The tactile display SD is a display that reproduces the softness when the operator OP palpates the affected part of the patient PA by applying a stimulus to the fingertip of the doctor DO or the like based on the palpation information transmitted by the palpation device 1. The tactile display SD may have the shape shown in FIG. 1. Note that the tactile display SD is not limited to the shape shown in FIG. 1, and may be any display capable of reproducing the softness when the operator OP palpates the affected part of the patient PA based on the palpation information.

[0036] Next, the software configuration of the palpation device 1 will be described. FIG. 4 is a block diagram showing the software configuration of the palpation device 1 according to the embodiment in association with the hardware configuration of the palpation device 1 shown in FIG. 2. The control unit 11 includes a softness measurement unit 111 and a communication control unit 112. Further, the softness measurement unit 111 includes an input information processing unit 1111, a background removal unit 1112, a region determination unit 1113, a three-dimensional coordinate calculation unit 1114, a region removal unit 1115, a normal vector calculation unit 1116, a displacement amount calculation unit 1117, and a Young's modulus calculation unit 1118.

[0037] The softness measurement unit 111 is a control unit for measuring the softness of the measurement target by pressing the affected part to be measured by the operator OP. For example, the softness measurement unit 111 takes the measured value of the force sensor PS and the captured images (RGB image and depth image) of the depth camera DC as inputs, and outputs the captured images and the softness index to the communication control unit 112. Also, the softness measurement unit 111 may output the information calculated and detected by each unit described below to the communication control unit 112. Further, the softness measurement unit 111 outputs the captured image from the depth camera DC during finger pressure to the communication control unit 112.

[0038] The input information processing unit 1111 is an acquisition unit that acquires the measured value of the force sensor PS, the captured images of the depth camera DC, etc. For example, the input information processing unit 1111 takes the above measured value and captured image as inputs, and outputs the input information to the acquisition information storage unit 131. That is, the input information processing unit 1111 controls to store the measured value and the captured image in the acquisition information storage unit 131. Also, the input information processing unit 1111 may control to display the acquired information on the display of the output device 152.

[0039] The background removal unit 1112 is a control unit that removes the background from the depth image. For example, the background removal unit 1112 takes the depth image as input and outputs the depth image with the background removed. For example, the background removal unit 1112 acquires the depth image stored in the acquired information storage unit 131 and removes the background from the acquired depth image. Details of the background removal method will be described later. The background removal unit 1112 then outputs the depth image with the background removed to the 3D coordinate calculation unit 1114 and the region removal unit 1115.

[0040] The region determination unit 1113 is a control unit that calculates the positions of the operator's hands and fingers during palpation. For example, the region determination unit 1113 takes an RGB image as input and outputs the RGB image and the positions of the hands and fingers during palpation within the RGB image. The region determination unit 1113 acquires the RGB image stored in the acquired information storage unit 131 and calculates the positions of the operator's hands and fingers during palpation from the acquired RGB image. Details of the calculation method will be described later. The region determination unit 1113 then outputs the RGB image, including the calculated positions of the hands and fingers during palpation, to the 3D coordinate calculation unit 1114 and the region removal unit 1115.

[0041] The 3D coordinate calculation unit 1114 is a calculation unit that calculates the 3D coordinates of the palpation finger. For example, it takes a depth image with the background removed and an RGB image including the calculation results for the hand and palpation finger as input, and outputs the 3D finger coordinates of the palpation finger in the depth image. Based on the depth image with the background removed and the RGB image including the calculation results for the hand and palpation finger received from the background removal unit 1112 and the region determination unit 1113, the 3D coordinate calculation unit 1114 calculates the 3D finger coordinates in the depth image. Details of the method for calculating the 3D finger coordinates will be described later. The 3D coordinate calculation unit 1114 then outputs the calculated 3D finger coordinates to the displacement amount calculation unit 1117.

[0042] The region removal unit 1115 is a removal unit that performs hand / palpation finger region expansion processing and removes corresponding regions in the depth image. For example, the region removal unit 1115 takes a depth image with the background removed and an RGB image including the calculation results for the hand and palpation finger as input, and outputs a depth image and RGB image from which the regions recognized as palpation fingers and hands have been removed. The region removal unit 1115 performs expansion processing on the depth image with the background removed received from the background removal unit 1112, and removes the regions recognized as palpation fingers and hands in the expanded region. The expansion processing may be, for example, morphological transformation processing. The region removal unit 1115 then outputs the depth image and RGB image from which the regions recognized as palpation fingers and hands have been removed to the normal vector calculation unit 1116.

[0043] The normal vector calculation unit 1116 is a calculation unit that calculates normal vectors. For example, the normal vector calculation unit 1116 takes a depth image and an RGB image from which the areas recognized as palpable fingers and hands have been removed as input, and outputs an RGB image containing normal vector information. The normal vector calculation unit 1116 calculates normal vectors based on the depth information of the depth image from which the areas recognized as palpable fingers and hands have been removed, which is received from the area removal unit 1115. The area removal unit 1115 then outputs the RGB image containing the calculated normal vector information to the displacement amount calculation unit 1117.

[0044] The displacement calculation unit 1117 is a calculation unit that calculates the amount of displacement. For example, the displacement calculation unit 1117 takes an RGB image containing the coordinates of the pressure point start point, the 3D finger coordinates, and the calculated normal vector n as input, and outputs the displacement of the palpated finger. The displacement calculation unit 1117 reads the pressure point start point coordinates stored in the pressure point start point coordinate storage unit 132. Furthermore, the displacement calculation unit 1117 calculates the displacement of the palpated finger based on the pressure point start point coordinates and the RGB image containing the 3D finger coordinates received from the 3D coordinate calculation unit 1114 and the normal vector calculation unit 1116, and the calculated normal vector n. Details of the method for calculating the displacement will be described later. The displacement calculation unit 1117 then outputs the displacement to the Young's modulus calculation unit 1118.

[0045] The Young's modulus calculation unit 1118 is a calculation unit that calculates an index of the softness of the object being measured. For example, the Young's modulus calculation unit 1118 takes the finger curvature radius, displacement amount, and measured value of the force sensor PS as input and outputs an index of softness. The Young's modulus calculation unit 1118 acquires the measured value of the force sensor PS from the acquired information storage unit 131 and the finger curvature radius from the finger curvature radius storage unit 133. Furthermore, the Young's modulus calculation unit 1118 calculates the Young's modulus, which is an index of softness, based on the displacement amount, the measured value of the force sensor PS, and the finger curvature radius. Details of the Young's modulus calculation method will be described later. The Young's modulus calculation unit 1118 then outputs the Young's modulus to the communication control unit 112.

[0046] The communication control unit 112 is a control unit that communicates various information with the diagnostic device 2 through the communication interface 14. For example, the communication control unit 112 takes information including captured images and Young's modulus as input, converts this information into information suitable for communication, and outputs this information. The communication control unit 112 may also control the information received from the diagnostic device 2 to be displayed on the display of the output device 152.

[0047] The data storage unit 13 includes an acquired information storage unit 131, a pressure point start point coordinate storage unit 132, a finger curvature radius storage unit 133, and a fingertip physical property value correction coefficient storage unit 134.

[0048] The acquired information storage unit 131 is a storage unit used to store information acquired by the input information processing unit 1111. For example, the acquired information storage unit 131 stores the measured values ​​of the force sensor PS and the images captured by the depth camera DC.

[0049] The acupressure start point coordinate storage unit 132 is a storage unit used to store the coordinates of the acupressure start point. For example, the acupressure start point coordinate storage unit 132 stores the three-dimensional finger coordinates when acupressure is started on the affected area being measured by the palpating finger.

[0050] Figure 5 shows an example of three-dimensional finger coordinates stored in the acupressure start point coordinate storage unit 132 according to this embodiment. As shown in Figure 5, the three-dimensional coordinates of the acupressure start point on the affected area, which is the target of measurement for palpation, are stored in the acupressure start point coordinate storage unit 132.

[0051] The finger curvature radius memory unit 133 is a memory unit that stores the finger curvature radius of the operator (OP). For example, the finger curvature radius memory unit 133 stores the results of the operator (OP) measuring the curvature radius of their own finger.

[0052] Figure 6 shows an example of a finger curvature radius stored in the finger curvature radius storage unit 133 according to this embodiment. As shown in Figure 6, the finger curvature radius of the operator (OP) is stored in the finger curvature radius storage unit 133.

[0053] The fingertip physical property correction coefficient storage unit 134 is a storage unit that stores coefficients and correction coefficients. The Young's modulus of the operator's fingertip varies depending on the contact conditions with the affected area being measured. For example, under low finger pressure, the pressure is mainly from the relatively soft epidermis, whereas under high finger pressure, the epidermis is compressed and the pressure is mainly from hard tissues such as bone. Therefore, the fingertip Young's modulus is higher under high finger pressure compared to low finger pressure. Also, due to the principle of the force sensor PS that measures the force of the fingertip from nail strain, there is an error in the measured value when using flexible materials. Therefore, for example, since the Young's modulus of the operator's fingertip varies (hardness differs) depending on the contact conditions, a coefficient value for Young's modulus correction is calculated by conducting a calibration experiment using a gel that mimics human skin. Accordingly, the fingertip physical property correction coefficient storage unit 134 stores the coefficient value for Young's modulus correction calculated by the calibration experiment.

[0054] Figure 7 shows an example of coefficients and correction coefficients stored in the fingertip physical property correction coefficient storage unit 134 according to the embodiment. As shown in Figure 7, the fingertip physical property correction coefficient storage unit 134 stores a finger pressure coefficient a, a variable coefficient b, and a constant c. Details of the Young's modulus correction using these will be described later.

[0055] Finally, the software configuration of the diagnostic device 2 will be described. Figure 8 is a block diagram showing the software configuration of the diagnostic device 2 according to the embodiment, in relation to the hardware configuration of the diagnostic device 2 shown in Figure 2. The control unit 21 comprises a communication control unit 211, a diagnostic control unit 212, and an output control unit 213.

[0056] The communication control unit 211 is a control unit that communicates various information with the palpation device 1 through the communication interface 24. For example, the communication control unit 211 takes information including captured images and Young's modulus as input, converts this information into information suitable for output, and outputs this information. That is, the communication control unit 211 controls the information received from the palpation device 1 to be displayed on the display of the output device 152. The communication control unit 211 may also acquire captured images (images taken of the doctor DO) from the camera included in the input device 251, convert the acquired captured images into information suitable for communication, and transmit this information to the palpation device 1 through the communication interface 24. The communication control unit 211 also stores the information received from the palpation device 1 (captured images, Young's modulus, etc.) in the acquired information storage unit 231.

[0057] The diagnostic control unit 212 is a control unit that performs a diagnosis based on an index of softness. For example, the diagnostic control unit 212 takes Young's modulus as input and outputs the diagnosis result. The diagnostic control unit 212 uses AI technology or the like to perform a diagnosis of the affected area being measured based on Young's modulus, which is an index of softness. The diagnostic control unit 212 then outputs the diagnosis result to the output control unit 213.

[0058] The output control unit 213 is a control unit that controls the output of diagnostic results. For example, the output control unit 213 takes the diagnostic results as input and outputs those diagnostic results. The output control unit 213 controls the output control unit 213 to display the diagnostic results on its own display.

[0059] The data storage unit 23 includes an acquired information storage unit 231. The acquired information storage unit 231 is a storage unit used to store various types of information received by the communication control unit 211.

[0060] [Operation] Figure 9 is a flowchart showing an example of processing by the palpation device 1 during a palpation operation according to this embodiment. The operation shown in this flowchart is realized when the control unit 11 of the palpation device 1 reads and executes a program stored in the program storage unit 12.

[0061] This operation is initiated, for example, when the operator OP palpates the affected area of ​​the patient PA in response to instructions from the physician DO using the diagnostic device 2. In this embodiment, the communication control unit 112 and the communication control unit 211 are assumed to be constantly sending and receiving captured images and other information necessary for telemedicine. The communication control unit 112 also transmits information output from the softness measurement unit 111 (e.g., captured images, Young's modulus) to the diagnostic device 2 as appropriate. The communication control unit 211 controls the received information to be displayed on the display of the output device 252.

[0062] In step ST101, the input information processing unit 1111 acquires the finger curvature radius. The operator (OP) measures the finger curvature radius by measuring the size of their own finger. The input information processing unit 1111 then acquires the measurement result entered using the input device 151. Note that the finger curvature radius can be measured by any common method, such as a predetermined sensor or device capable of measuring the finger curvature radius, so a detailed explanation is omitted here. The input information processing unit 1111 stores the acquired measurement result in the finger curvature radius storage unit 133. Note that the operation of step ST101 can be performed at any time before palpation.

[0063] In step ST102, the input information processing unit 1111 acquires RGB images and depth images. The input information processing unit 1111 acquires RGB images and depth images taken by the depth camera DC when the operator OP is palpating the affected area of ​​the patient PA. The input information processing unit 1111 stores the captured images, including the acquired RGB images and depth images, in the acquisition information storage unit 131.

[0064] In step ST103, the background removal unit 1112 removes the background from the depth image. The background removal unit 1112 acquires the depth image stored in the acquired information storage unit 131. Then, the background removal unit 1112 removes the background from the depth image by predetermined threshold processing. For example, the background removal unit 1112 removes objects that are more than a certain distance (i.e., more than a predetermined threshold) away from the depth camera DC. The background removal unit 1112 outputs the depth image from which the background has been removed to the 3D coordinate calculation unit 1114 and the region removal unit 1115.

[0065] In step ST104, the region determination unit 1113 calculates the position of the hand / palpating finger. The region determination unit 1113 acquires the RGB image stored in the acquired information storage unit 131. The acquired RGB image is an image that corresponds in time to the depth image acquired by the background removal unit 1112. From the acquired RGB image, the region determination unit 1113 calculates the position of the hand or finger (palpating finger) that is palpating the affected area of ​​the operator OP. For example, the region determination unit 1113 detects the region of the hand and palpating finger using AI (deep learning) object detection technology. Any method such as YOLO can be used for the deep learning object detection technology, so a detailed explanation is omitted here. Furthermore, the region determination unit 1113 sets the center of the detected palpating finger region as the palpating finger coordinate. The palpating finger coordinate in the RGB image may be acquired using other measurement means such as AR markers, infrared reflection markers, or hand skeleton recognition models. Furthermore, the region determination unit 1113 may perform smoothing using a moving average or the like with respect to the positions of the hand and the palpating finger.

[0066] Figure 10 shows an example of the hand and palpation fingers detected by the region determination unit 1113 from an RGB image according to the embodiment, and the acquired palpation finger coordinates. In the example of Figure 10, the region of the hand detected by the region determination unit 1113 is the region enclosed by a solid line, and the region of the palpation fingers is the region enclosed by a dotted line. Furthermore, in the example of Figure 10, the palpation finger coordinates are shown in the center of the palpation finger region. The region determination unit 1113 tracks the positions of the hand and palpation fingers from the RGB image as shown in Figure 10, and continues to acquire palpation finger coordinates as it tracks. The region determination unit 1113 outputs region information including the RGB image, the positions of the hand and palpation fingers, and palpation finger coordinates to the 3D coordinate calculation unit 1114 and the region removal unit 1115.

[0067] Here, as shown in Figure 10, in one embodiment, the operator OP is shown palpating the affected area to be measured with one finger (palpating finger), but of course, the affected area may be palpated with multiple fingers. In this case, the area determination unit 1113 calculates the position of each palpating finger using the technique described above.

[0068] In step ST105, the 3D coordinate calculation unit 1114 calculates the 3D coordinates of the palpation finger. The 3D coordinate calculation unit 1114 calculates the 3D finger coordinates on the depth image corresponding to the position of the palpation finger detected by the region determination unit 1113. In this embodiment, to address the issue of outliers, the 3D coordinate calculation unit 1114 calculates the average value of the 3D coordinates of the surrounding n x n pixels specified in advance from the position of the palpation finger as the 3D finger coordinates. Here, n is any positive integer. Furthermore, since the surrounding n x n pixels also include the background and objects other than the palpation finger, the 3D coordinate calculation unit 1114 invalidates areas other than those recognized as the palpation finger or hand based on the region determination result by the region determination unit 1113. That is, the 3D coordinate calculation unit 1114 calculates the 3D finger coordinates using the average value of the 3D coordinates of the pixels of the palpation finger and hand among the n x n pixels surrounding the palpation finger.

[0069] Figure 11 shows an example of the position of the palpation finger and the area to be calculated for average coordinates according to the embodiment. As shown in Figure 11, the area within the dotted line portion inside the area detected as the palpation finger is the area to be calculated for 3D finger coordinates. The area determination unit 1113 calculates 3D finger coordinates from the area to be calculated. The area determination unit 1113 outputs the depth image and the calculated 3D finger coordinates to the displacement amount calculation unit 1117.

[0070] In step ST106, the region removal unit 1115 performs hand / palpation finger region expansion processing to remove the corresponding region on the depth image. In the depth image from which the background has been removed by the background removal unit 1112, the region removal unit 1115 specifies a range of n x m pixels surrounding the palpation finger position by processing in the same manner as in step ST105. This range of n x m pixels becomes the normal target range. Within the specified range, the region removal unit 1115 removes the regions recognized as palpation fingers and hands by the region determination unit 1113. As a result, the region removal unit 1115 extracts only the depth information of the object being palpated. Here, in order to prevent contours from remaining in the regions detected as palpation fingers or hands, the region removal unit 1115 applies morphological transformation (expansion) processing to expand the region beyond the actual recognized area. Then, the region removal unit 1115 removes the regions recognized as palpation fingers and hands from the expanded region. The region removal unit 1115 outputs the depth image and RGB image, from which the regions recognized as the palpation finger and hand have been removed, to the normal vector calculation unit 1116.

[0071] In step ST107, the normal vector calculation unit 1116 calculates the normal vector. The normal vector calculation unit 1116 performs appropriate decimation on the depth information (point cloud) of the affected area that is the target of palpation in the depth image obtained by removing the background and the areas recognized as the palpation fingers and hands from the depth image, and then calculates the normal vector n using principal component analysis (PCA).

[0072] Figure 12 shows an example of a normal vector calculated by the normal vector calculation unit 1116 according to the embodiment. First, the left figure in Figure 12 is an RGB image, and the right figure shows an example of a depth image in which the background and the regions recognized as palpation fingers and hands have been removed by processing up to step ST106. As shown in the right figure of Figure 12, the background is removed by the background removal unit 1112, and the hands and palpation fingers are removed by the region removal unit 1115. The normal vector calculation unit 1116 then calculates a normal vector based on the depth information within n × m pixels (within the dashed frame in Figure 12). The left arrow in Figure 12 shows an example of the calculated normal vector. The normal vector calculation unit 1116 may perform smoothing on the calculated normal vector as appropriate, such as by moving average.

[0073] In this embodiment, an example of calculating the normal vector using principal component analysis was described. However, the normal vector calculation unit 1116 may calculate the normal vector using any method capable of calculating the normal vector. For example, one method is to generate a mesh based on depth information (point cloud) and use the average value of the normal vectors of the individual faces.

[0074] In step ST108, the displacement calculation unit 1117 determines whether the measured value of the force sensor PS is greater than 0. If it is determined that the measured value of the force sensor PS is 0, that is, that the palpating finger is not touching the affected area, the process returns to step ST102. On the other hand, if it is determined that the measured value of the force sensor PS is greater than 0, that is, that the palpating finger is touching the affected area, the process proceeds to step ST109.

[0075] In step ST109, the displacement calculation unit 1117 determines whether the acupressure start point has already been recorded. If it determines that the acupressure start point has not been recorded, the process proceeds to step ST110. If it determines that the acupressure start point has already been recorded, the process proceeds to step ST111.

[0076] In step ST110, the displacement amount calculation unit 1117 causes the finger pressure start point coordinates to be stored in the finger pressure start point coordinate storage unit 132. When the measured value of the force sensor PS exceeds 0 in step ST108, that is, the three-dimensional finger coordinates at the moment when the palpating finger contacts the affected area are p ０ Let it be, and the time at that time be t ０ Let it be. Then, the displacement amount calculation unit 1117 stores this three-dimensional finger coordinate p ０ and the time t ０ in the finger pressure start point coordinate storage unit 132. Then, the process returns to step ST102.

[0077] In step ST111, the displacement amount calculation unit 1117 reads out the finger pressure start point coordinates. The displacement amount calculation unit 1117 reads out the finger pressure start point coordinates p ０ stored in the finger pressure start point coordinate storage unit 132.

[0078] In step ST112, the displacement amount calculation unit 1117 calculates the displacement amount. For example, the three-dimensional finger coordinates p ｉ calculated in step ST105 (at time t ｉ ). Then, the displacement amount calculation unit 1117 calculates the component along the normal vector n of p ｉ -p ０ as the displacement amount at time t ｉ . The displacement amount calculation unit 1117 outputs the calculated displacement amount to the Young's modulus calculation unit 1118.

[0079] In step ST113, the Young's modulus calculation unit 1118 reads out the fingertip physical property correction value coefficient. The Young's modulus calculation unit 1118 reads out the fingertip physical property correction value coefficient stored in the fingertip physical property value correction coefficient storage unit 134.

[0080] In step ST114, the Young's modulus calculation unit 1118 reads out the finger curvature radius. The Young's modulus calculation unit 1118 reads out the finger curvature radius stored in the finger curvature radius storage unit 133.

[0081] In step ST115, the Young's modulus calculation unit 1118 calculates the Young's modulus. The Young's modulus calculation unit 1118 calculates the Young's modulus based on the displacement amount, the measured value P of the force sensor PS ｉBased on the radius of curvature of the finger, the Young's modulus, an indicator of softness, is calculated using the following formula based on Hertz's contact theory. Note that this formula approximates the finger as a sphere and assumes contact with an infinite plane.

[0082]

[0083] Here, v ｉ and v ０ This represents the Poisson's ratio of the finger and the affected area, and E ｉ and E ０ This represents the Young's modulus of the finger and the affected area, F ｎ This is the load, i.e., the measured value P from the force sensor PS. ｉ Represents R ｉ δ represents the radius of curvature of the finger, ｚｎ This represents the amount of displacement. Furthermore, in measurements of the human body, due to the finiteness of thickness, the heterogeneity of tissue, and the presence of underlying tissue (such as bone), the calculated Young's modulus is not strictly the same as the physical property value of the tissue (it becomes an apparent Young's modulus). However, in one embodiment, since the purpose is to transmit the fingertip sensation of the operator, the apparent Young's modulus is stored and transmitted as is.

[0084] Furthermore, the last term in the above equation is the fingertip material property term. The Young's modulus calculation unit 1118 may approximate the fingertip material property term as an inverse function according to the coefficient values ​​stored in the fingertip material property value correction coefficient storage unit 134, as shown in the following equation. In the following approximation, the Young's modulus is assumed to depend on the finger pressure and the amount of displacement.

[0085]

[0086] Here, a, b, and c are the finger pressure coefficient a, the displacement coefficient b, and the constant c, as explained with reference to Figure 7. Furthermore, the Young's modulus calculation unit 1118 may approximate the fingertip physical property term with a function other than the reciprocal function, such as an exponential function or any other arbitrary function.

[0087] In one embodiment, the Young's modulus calculation unit 1118 was shown to calculate Young's modulus as an indicator of softness. However, the indicator of softness is not limited to Young's modulus, and any other indicator (such as Asker C hardness) may be used.

[0088] Figure 13 shows an example of the screen displayed on the output device 252 of the diagnostic device 2 in the embodiment. As shown in Figure 13, by measuring the amount of displacement and an index of softness (e.g., Young's modulus), the output device 252's display will show the Young's modulus, the amount of indentation, etc.

[0089] Furthermore, in accordance with the Young's modulus transmitted by the communication control unit 112, the communication control unit 211 performs presentation reaction force control of the tactile display SD. This allows the physician DO to diagnose the patient PA based on an indicator of the softness of the affected area, in addition to the image from the depth camera DC.

[0090] Alternatively, the diagnostic control unit 212 may perform a diagnosis based on the measured Young's modulus using AI technology or the like. The diagnostic control unit 212 then outputs the diagnosis result to the output control unit 213. The output control unit 213 controls the display of the output device 252 to display the diagnosis result.

[0091] In this embodiment, an example was described in which the affected area of ​​a patient PA was subjected to acupressure as the measurement target, but the measurement target is not limited to the affected area. For example, the measurement target can of course be an object other than the human body (e.g., an animal, a cushion, furniture, etc.). For example, if a uniform material with sufficient thickness is used as the measurement target instead of the human body, the Young's modulus calculation unit 1118 can calculate the accurate Young's modulus rather than the apparent Young's modulus.

[0092] [Effects] According to the embodiment, the palpation device 1 measures finger pressure based on nail strain using a force sensor PS. This does not impair the operator OP's fingertip sensation. Furthermore, because the operator OP's fingertips are free, the method of applying acupressure is not significantly restricted.

[0093] Furthermore, the palpation device 1 can measure the softness of the affected area by optically measuring the amount of indentation by the operator OP based on the measurement value of the force sensor PS and the image captured by the depth camera DC.

[0094] [Other Embodiments] Furthermore, the processing performed by the palpation device 1 in the above embodiment may also be performed by the diagnostic device 2. For example, the palpation device 1 transmits the measurement results of the force sensor PS and the images captured by the depth camera DC to the diagnostic device 2. The diagnostic device 2 may perform the processing described above based on the transmitted information.

[0095] It is also possible to construct the operation of each component of the palpation device 1 in the above embodiment as a program, and install and run it on a computer used as a user device or on a computer used as an external device.

[0096] Furthermore, the method described in the above embodiment can be distributed by storing the program (software means) that can be executed by a computer in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, MO, etc.), or a semiconductor memory (ROM, RAM, flash memory, etc.), and by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only the execution program but also tables and data structures) to be executed by the computer. The computer realizing this device reads the program stored in the storage medium, and, if necessary, constructs the software means using the configuration program, and executes the above-described process by controlling its operation with this software means. The storage medium referred to in this specification is not limited to distribution mediums, but also includes storage mediums such as magnetic disks and semiconductor memories provided inside the computer or in devices connected via a network.

[0097] In short, this invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriately as possible, in which case the combined effects can be obtained. Moreover, the embodiments described above include inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple constituent elements disclosed.

[0098] 100...Measurement system 1...Palpation device 11...Control unit 111...Measurement unit 1111...Input information processing unit 1112...Background removal unit 1113...Area determination unit 1114...3D coordinate calculation unit 1115...Area removal unit 1116...Normal vector calculation unit 1117...Displacement amount calculation unit 1118...Young's modulus calculation unit 112...Communication control unit 12...Program storage unit 13...Data storage unit 131...Acquired information storage unit 132...Acupressure start point coordinate storage unit 133...Finger curvature radius storage unit 134...Fingertip physical property value correction coefficient storage unit 14...Communication interface 15...Input / output interface 151...Input device 152...Output device PS...Force sensor DC...Depth camera 2...Diagnostic device 21...Control unit 211...Communication control unit 212...Diagnostic control unit 213...Output control unit 22...Program storage unit 23...Data storage unit 231...Acquired information storage unit 24...Communication interface 25...Input / output interface 251...Input device 252...Output device SD...Haptic display OP...Operator PA...Patient DO...Physician

Claims

1. A softness measuring device comprising: an acquisition unit that acquires a measurement value from a force sensor attached to the fingernail of the operator and which measures the pressure of the fingertip by the strain of the nail, and an image including an RGB image and a depth image when the force sensor is attached and the target to be measured is pressed with the finger; an area determination unit that determines the area of ​​the finger in the RGB image; a three-dimensional coordinate calculation unit that calculates the three-dimensional finger coordinates of the finger in the depth image corresponding to the finger area and the determined position of the finger; a normal vector calculation unit that calculates a normal vector based on the depth information of the depth image when the target to be measured is pressed with the finger; a displacement amount calculation unit that calculates a component along the normal vector between the three-dimensional finger coordinates of the image and the three-dimensional finger coordinates of the image when the finger pressure is started as a displacement amount; a calculation unit that calculates an index of the softness of the target to be measured based on the displacement amount, the measurement value, and the radius of curvature of the finger; and a communication control unit that transmits the image, the displacement amount, and the index.

2. The softness measuring device according to claim 1, wherein the three-dimensional coordinate calculation unit calculates the average value of the three-dimensional coordinates of predetermined surrounding pixels from the center of the area determined to be the finger area as the three-dimensional finger coordinate.

3. The softness measuring device according to claim 1, wherein the normal vector calculation unit calculates the normal vector from the region of the object to be measured, after removing the background of the depth image and the operator's hand, including the fingers.

4. A softness measurement method executed by the processor of a softness measuring device, comprising: acquiring a measurement value from a force sensor attached to the fingernail of a practitioner and measuring the pressure of the fingertip by the strain of the nail; an image captured including an RGB image and a depth image when the force sensor is attached and the finger is applying pressure to the object to be measured; determining the finger region of the RGB image; calculating the three-dimensional finger coordinates of the finger in the depth image corresponding to the finger region and the determined finger position; calculating a normal vector based on the depth information of the depth image when the finger is applying pressure to the object to be measured; calculating a displacement amount as a component along the normal vector between the three-dimensional finger coordinates of the image captured and the three-dimensional finger coordinates of the image captured when the finger application of pressure by the finger begins; calculating an index of the softness of the object to be measured based on the displacement amount, the measurement value, and the radius of curvature of the finger; and transmitting the image captured, the displacement amount, and the index.

Images