Compression-induced imaging on a mobile platform for subsurface and surface object characterization

Abstract

The mobile platform imaging device generates an image of an object using compression of a target area. The tactile sensor has an optical waveguide including a flexible, transparent first layer. Light is guided into the waveguide. When the first layer is deformed, light is scattered from the first layer. The first layer is deformed by pressing the tactile sensor against the object. The force sensor detects the force pressing the tactile sensor against the object and outputs corresponding force information. The first communication unit receives the force information from the force sensor. The receiver holds a mobile device having a second communication unit and an imager, which can generate image information using light scattered from the first layer. The first communication unit communicates with the second communication unit, and the mobile device communicates with an external network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to and further develops the teachings of co-invented and assigned International Patent Application No. PCT / US2011 / 043203, entitled “Apparatus and Method for Surface and Subsurface Tactile Sensation Imaging,” filed on July 7, 2011 and published as Publication No. WO 2012 / 006431 on January 12, 2012, and corresponding U.S. Patent No. 9,652,696, issued on May 16, 2017. The entire contents of that application and patent are incorporated herein by reference in their entirety as if expressly set forth herein. This application also relates to and claims priority to U.S. Provisional Application No. 62 / 538,032, filed on July 28, 2017, the disclosure of which is also incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to compression-induced imaging and characterization of subsurface and surface objects, and more particularly to a mobile platform-based device and related methods for determining the size and stiffness of a solid target beneath the surface of a softer solid medium. The device and method can be used to non-invasively characterize tumors in humans or animals. Background Art

[0004] Elasticity is one of the properties used to identify malignant tumors. Unhealthy tissues tend to be stiffer than corresponding healthy tissues (Itoh et al. 2006, Dargahi 2004, Rivaz 2008, Krouskop 1998, Regini 2010). Therefore, quantifying tissue elasticity will greatly help doctors classify and identify unhealthy tissues.

[0005] One researcher reported that elasticity scores derived using ultrasound elastography detected malignancies in 116 of 120 cases (Regini 2010). Even though ultrasound elastography has shown good results, there are limitations. The main limitation of elastography is that tissue compression affects the elasticity score, which can lead to misdiagnosis (Itoh et al. 2006). In addition, elastography requires a trained operator in a hospital setting, which means it is not portable and is expensive and complicated to perform.

[0006] The tactile imaging system (TIS) of the above-mentioned earlier application No. WO 2012 / 006431 has a camera, a probe and a force sensor, and can be connected to a laptop computer. However, the actual embodiment of the system is somewhat bulky, relatively expensive and complex. Therefore, there is still room for improvement. Summary of the invention

[0007] According to one aspect of the present application, there is provided a mobile platform imaging device that uses compression of a target area to generate an image of an object, comprising: a tactile sensor including an optical waveguide including at least a flexible and transparent first layer, and at least one light source configured to guide light into the optical waveguide. The optical waveguide is configured so that when the first layer is deformed, at least some of the light guided into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by pressing the tactile sensor against the object; preferably a rigid frame that holds the waveguide, or a waveguide with sufficient rigidity can be formed to support the waveguide in use. A force sensor that detects the force applied to press the tactile sensor against the object and outputs corresponding force information. A first communication unit is connected to receive force information from the force sensor. A receptacle is provided for holding a mobile device having a second communication unit and an imager, the imager being positioned so that the imager can use at least some of the light scattered from the first layer to generate image information; wherein the first communication unit is capable of communicating with the second communication unit, and the mobile device is capable of communicating with an external network.

[0008] The mobile platform imaging device can be combined with a mobile phone as a mobile device. The mobile phone can then be programmed to use at least some of the light scattered from the first layer to generate image information (i.e., include an imager), receive force information, synchronize the image information with the force information, and transmit the associated information to an external device. The transmitted information can then also include selected information such as the current date, the current time, the user ID of the current user of the mobile device, and the target ID of the object.

[0009] The image information may be selected from the group consisting of a single image, a plurality of images, and a video image.

[0010] The mobile platform imaging device may be programmed to exchange a plurality of messages with an external device, generate a hash from the contents of the plurality of messages, and encrypt the image and / or force information using an encryption key determined from the hash.

[0011] The force sensor may be between the tactile sensor and a receptacle for holding the mobile device.The mobile platform imaging device may also include a housing, and the force sensor may then be between the housing and the receptacle for holding the mobile device.

[0012] According to another aspect, a method for determining surface or sub-surface dimensions of an object using the mobile platform imaging device described above is provided, comprising: obtaining image information and corresponding force information under multiple applied forces; and estimating the dimensions of the object using 3D interpolation.

[0013] The obtaining step may include obtaining a series of images over a time period while varying the force, and selecting images from a portion of the time period during which the force varies more smoothly and / or more linearly with time than during other portions of the time period.

[0014] According to another aspect, a method for aligning a light source of the mobile platform imaging device described above is provided, comprising applying a tactile sensor to a known object with a preselected force, processing image information to identify asymmetry in the scattered light, and adjusting the light source or at least one light source to reduce the asymmetry.

[0015] One or more light sources illuminate the light guide; the wavelength of the light can vary for different applications, or multiple wavelengths of light can be used in a single embodiment and controlled to provide varying image responses. The light guide is transparent and flexible, where the light guide and camera are configured to have a line of sight to capture the deflected light. The light source can have a diffuser to generate uniform light and / or filters can be added to control the light emission into or out of the waveguide. The diffuser can be a hollow tube where the light source is placed inside the tube, or it can be a white plastic / paper diffuser material placed between the light source and the waveguide.

[0016] The waveguide can be held in place by a rigid frame, wherein the clear transparent layer can be part of the frame, so that when the waveguide is pressed against an area of ​​interest, the front or portion of the waveguide facing the area of ​​interest will deform, while the waveguide will have sufficient rigidity to maintain line of sight between the waveguide and the camera to allow detection of the deflected light and calculation of the applied force. The stiffness can be provided by the waveguide itself (i.e., the waveguide is shaped / configured to provide stiffness), a glass or plastic plate behind the waveguide, or the camera lens itself. In one embodiment, a rectangular waveguide with a clear rigid backing is shown. If the waveguide is bent to provide a Figure 3A If the material has sufficient stiffness as shown, a rigid clear backing layer would not be required.

[0017] According to another aspect, a method for detecting cuts or damage to a waveguide of a mobile platform imaging device is provided, comprising applying a tactile sensor to a known object with a preselected force, processing image information to identify asymmetry in scattered light, and when the asymmetry exceeds a threshold, indicating a cut or damage to the waveguide.

[0018] According to another aspect, a method for determining the softness of a target of a mobile platform imaging device is provided, comprising: determining a contact area based on image information, determining stress based on the contact area and force information of a tactile sensor, determining a strain of a flexible waveguide based on the image information and the determined stress; and determining a strain of an inclusion based on the strain, contact area, and stress of the flexible waveguide.

[0019] The step of determining the strain of the flexible waveguide may use at least one of the sum of the intensities of the pixels of the scattered light and the intensity of the maximum intensity pixel of the scattered light.

[0020] According to another aspect, a method for determining absolute elasticity of an object using compression induced imaging system (CIS) information is provided, comprising a forward modeling approach using a finite element method followed by an inverse modeling approach with a deep compact network.

[0021] According to another aspect, a method of obtaining a tumor risk index of an object includes at least one of: calculating the tumor risk index as a weighted sum of the size and rigidity of the object; and a machine learning method of determining the risk index using a convolutional neural network.

[0022] According to another aspect, a method of communication is provided, comprising: exchanging multiple messages by a first device and a second device, generating a hash from the contents of the multiple messages, and at least one of: encrypting information using an encryption key determined from the hash and sending the encrypted information to the second device; and receiving information from the second device and attempting to decrypt the information using the encryption key determined from the hash.

[0023] The smartphone platform removes the need for laptops and dedicated cameras and makes more compact and portable devices possible. Smartphone accessories can be made with just an elastomeric probe, a light source, and a frame. By leveraging the ubiquitous smartphones of today's society, compression-induced imaging systems (CIS) on mobile platforms may be more cost-effective than stand-alone tactile imaging systems. Smartphones already have processors, cameras, and transceivers capable of short-range (Bluetooth, Wi-Fi) and long-range (cellular) communications. By using these, the cost, weight, and volume of providing dedicated equipment in a CIS unit can be saved.

[0024] The optical waveguide can be automatically moved toward the target by a motor (such as a linear actuator or screw drive, or by using a pulley or timing belt) with the click of a switch. This will allow the force to be applied automatically, rather than manually pressing. In one application, the motor drove the optical waveguide toward the target area in a uniform motion of about 5 to 7 mm in about three seconds to obtain a tactile image.

[0025] The optical waveguide can have different shapes, such as hemispherical, semi-cylindrical or rectangular. The waveguide can also be bent to provide sufficient rigidity, for example, the waveguide can be a substantially flat sheet that is then bent into a semi-cylindrical shape, which can impart rigidity (i.e., structural rigidity) to the waveguide. The front surface of the waveguide will contact the target area, while the rear surface will provide sufficient rigidity to obtain a tactile image.

[0026] Images, pressure readings, and metadata can be sent to the cloud. Utilizing the cloud as an intermediate "second device" is a very beneficial step. This allows us to use the compression-induced imaging system at any location, at any time, without having to ensure that the physical recipient computer is available and online. The data can then be retrieved from the cloud for later processing at the recipient computer.

[0027] In an embodiment, the device includes a light source and an elastomeric light guide located in front of a smartphone, and a handle and force sensor located on the back, with the smartphone sliding into a slot between the two. Force is applied from the back, and the applied force measured by the force sensor is wirelessly transmitted to a smartphone application. A microcontroller with Bluetooth capability is used to transmit the applied force information to the smartphone, thereby avoiding the need for a mechanical data connection, and thus avoiding the problem of ensuring that the correct connector is available for a particular smartphone. The elastomer is positioned so that the smartphone camera is aligned with the center of the elastomer. The force and image synchronization problem is solved using a smartphone application ("app"), which both initiates the capture of tactile images and polls the applied force information. The applied force information is then sent to the application via Bluetooth communication. The stored images and metadata (user number, case number, tumor location, date, time, applied force) are encrypted by the application and wirelessly transmitted to a cloud server. The local processing computer then downloads the data and processes the data. The malignancy score will be calculated and the encrypted score will be sent back to the smartphone.

[0028] Under ideal conditions, the malignancy score can be calculated and returned to the smartphone in near real time. However, if any communication link between the smartphone and the local processing computer is unavailable, the data can be retained and forwarded when the required communication link appears.

[0029] The "tumor risk index" aggregates all the determined mechanical properties into a single number. This number is used to determine the likelihood that the mass being examined is a malignant tumor.

[0030] The mobile platform compression-induced imaging system can mimic the human finger to detect the mechanical properties (size, shape, stress, strain, mobility and elasticity) of an object. The object can directly contact the probe or be compressed indirectly by the system.

[0031] The present application relates to a system for characterizing certain mechanical properties of an object by applying gentle pressure from an overlying surface. A device for estimating the size and softness of a surface or sub-surface target would be useful in a variety of applications. By placing the device on a mobile platform such as a smartphone, the technology can be more easily accessed. A compression induced imaging system (CIS) allows an operator to quickly capture the mechanical properties of an object after compression with the convenience of a mobile platform.

[0032] Our system is inspired by some previously proposed systems, but with some important differences. Our system uses a dynamic optical sensing mechanism instead of a static electromechanical pressure sensor. This allows us to more accurately dynamically image and assess mechanical properties. In addition, our system leverages the mobile device's sensors and the mobile device's communication equipment to reduce cost and increase accessibility.

[0033] One possible application of a mobile platform compression induced imaging system (CIS) is the detection of human malignancies. Therefore, embodiments of the new mobile system are oriented toward medicine, as they have the potential to help physicians pre-screen tumors and train palpation techniques. The mechanical properties that the system can provide are a valuable resource in the tumor screening process. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The overall architecture of an embodiment of a mobile platform compression induced imaging system (CIS) is shown.

[0035] Figure 2 yes Figure 1 00143] A block diagram of an embodiment of a front-end attachment of the system shown in FIG.

[0036] FIG. 3A to FIG. 3C Shown for Figure 2 The elastic sensor element of the front end attachment.

[0037] Figure 4 is a diagram showing possible locations of LEDs for one embodiment of an elastomeric sensing element.

[0038] FIG. 5A to FIG. 5C Images from the sensor under test conditions are shown.

[0039] Figure 6 It is a circuit diagram.

[0040] Figure 7 is a schematic side view of the mobile platform CIS.

[0041] Figure 8 It is a diagram of the communication process.

[0042] Fig.9A , Fig. 9B and Fig. 9Care front, side and rear views of a frame forming part of one embodiment of a CIS.

[0043] Fig. 10A and Fig. 10B are perspective and front views of another frame forming part of an embodiment of a CIS.

[0044] Fig.11 It is a functional block diagram of an implementation method of the data transmission method.

[0045] Fig.12 A user interface on a smartphone is shown.

[0046] Fig.13 is a diagram of the data acquisition flow between the mobile device and the front-end microprocessor.

[0047] Fig.14 is a flow diagram for an interface between a mobile device, a server, and a processing unit.

[0048] Fig.15 It is a graph of the forward modeling method.

[0049] Fig.16 It is a diagram of the inverse modeling approach.

[0050] Fig.17 is a graph of a neural network. DETAILED DESCRIPTION

[0051] Referring to the accompanying drawings and first to Figure 1 , one embodiment of the overall architecture of a mobile platform compression induced imaging system (CIS), generally indicated by reference numeral 20, basically includes a mobile device 22, a front-end attachment 24 of the mobile device 22, a cloud server 26, and a local processing unit 28. The mobile device 22 includes an imaging unit (typically a camera) 32, a software unit 34, and a wireless communication unit 30, all of which are standard in smartphones and are not described in detail for the sake of brevity.

[0052] The front end attachment 24 includes a compression sensing probe 36, an applied force measurement unit 38, a power unit 40, a wireless communication unit 42, and a processing unit 44 (see Figure 2 ).

[0053] The front-end accessory 24 is where all physical data acquisition (force and image data) occurs and provides a chassis for the mobile device (iPhone or other smart phone) 22. The processing unit (microcontroller) 44 communicates via Bluetooth with the software unit 34 of the smart phone 22. The smart phone 22 transmits all data and metadata to the cloud server 26 via Wi-Fi if available, otherwise by cellular data transmission.

[0054] As will be described in more detail below, the front end attachment 24 is at least partially enclosed or mounted to the smartphone 22 in order to ensure precise alignment between the compression sensing probe 36 and the camera 32, wherein the camera 32 must image the probe 36. The microcontroller 44 enables force sensor readings and allows Bluetooth communication with the smartphone software unit (application) 34. Data collection and image capture are controlled by the user within the application 34 using the smartphone's user interface (see Fig.12 ). When the user has collected the desired data, the user may then send it (or it may be automatically sent at periodic intervals) to the cloud server 26 via Wi-Fi.

[0055] The cloud server 26 includes a cloud-based database, which may be conventional and not shown in detail. The database is used to store tactile images and metadata until the local computer 28 is ready to process the data. The processed data is then returned to the originating smartphone 22. The user of the system can obtain the desired information on the screen of the smartphone 22.

[0056] Now also refer to Figure 2 , the front end attachment 24 includes a compressive sensing probe 36, which includes a layer of transparent flexible elastomer 46 and one or more light sources 48 for feeding light into the elastomer 46. In an embodiment, the light source 48 is a light emitting diode, which requires a constant current driver circuit system 50. Of course, other light sources may require different driver and control circuit systems. The compressive sensing probe 36 is Figure 7 is shown in more detail in .

[0057] The transparent, flexible elastomeric layer 46 forms an elastic optical waveguide, which is the main functional component of the main sensing probe 36 of the device 20. In one embodiment of the probe 36, the optical waveguide 46 is an elastomer such as polydimethylsiloxane (PDMS), which mainly includes [-Si(CH3)2-], with a backing layer 72 of glass or plastic (see Figure 7), although again this is not required in all embodiments. The probe 36 generates light corresponding to the compression of the elastomeric layer 46. Compression sensing is based on an optical phenomenon called total internal reflection. If two different media have different refractive indices, and a beam of light passes through the two media, then when the light reaches the interface between the two media, typically some of the light is transmitted by refraction and the rest is reflected, the amount depending on the angle of incidence. However, if the light starts in the medium of the higher refractive index, then there is a critical angle above which refraction is not possible and the light is completely reflected, i.e., total internal reflection. In the present device, the elastomeric layer 46 is configured so that light from the light source 48 is guided by total internal reflection along the layer between the PDMS-air interface in the front and the glass / plastic-air interface in the back, as long as the elastomeric layer 46 does not deform. Elastomeric waveguides are described in U.S. Patent No. 9,652,696.

[0058] However, when the elastomeric layer 46 of the probe 36 is compressed or deformed due to the applied force, the light is internally reflected from the deformed front surface at a steeper angle, at which angle some of the light escapes through the glass / plastic-air interface behind the elastomeric layer. Some of this escaping light is captured by the imager 32 in the mobile device 22.

[0059] Also refer to FIG. 3A to FIG. 3C , collectively referred to as FIG3 , possible embodiments of the elastomer layer 46 include a rectangular elastomer 52, a hemispherical body 56, and a rectangular body 58 that is bent into a portion of a cylinder. The sheets 52 and 58 have a light source 48 that is positioned so that the light enters from the boundary at the narrow edge, almost parallel to the large face, thereby guiding the light by total internal reflection. In the curved shape 56 (or 58 if the light is irradiated in a circumferential direction), the light is injected near the convex surface, almost tangential to the surface, so that it can be guided around the curved surface by total internal reflection. In all of these embodiments, there is a glass or plastic plate between the elastomer and the camera. The plate can be flat, or it can be curved, for example to match the curvature of the elastomer sheet 58. For all elastomers, light is injected from the side to introduce as much internal reflection as possible.

[0060] The elastic member 46 is formed by a frame (see Figure 7 ) is held in the proper position and shape by a frame which also provides a mount for the light source 48. The frame may have a clear, rigid layer 72 (e.g., glass or plastic) between the light guide 46 and the camera 32. Another embodiment is a curved elastic body 58 held by a rigid frame, such as Figure 3A The requirement is to have a line of sight between the optical waveguide and the camera for capturing the deflected light waves.

[0061] A variety of different light sources 48 can be used. The described embodiment uses ultra-bright white LEDs. The main function of the LEDs is to provide a complete integrated light to illuminate the entire area of ​​the sensing probe 36. The intensity of each individual LED affects the pixel value of each captured image. Therefore, it is a very desirable condition to have uniform intensity in each LED and in all probes. Image processing can be calibrated for non-uniform illumination as long as the non-uniformity is stable over time, but this may require significant additional processing power and processing time. In addition, one or more light sources can be provided to emit light of multiple wavelengths that can be used to provide different imaging results. The system can be customized to provide a selected wavelength change to the user, or it can automatically cycle between different wavelengths by software based on a specific wavelength adjustment calculation.

[0062] Now also refer to Figure 4 A possible embodiment of the rectangular elastomeric element 52 is a 38 mm (1.5 inch) square, 12 mm to 30 mm (0.5 to 1.2 inch) thick PDMS block. Four LEDs 48 are provided, one of which is located at the center of each narrow face of the PDMS block 52 (about 19 mm from the corner edge). The number of LEDs 48 on each side can be increased to increase the area of ​​illumination and / or the uniformity of illumination inside the PDMS. Typically, the probe will be covered with nitrile for sterilization reasons.

[0063] Alternatively, a diffusing element such as a white tube with a light source inside can be used to evenly distribute the light in the PDMS. Other diffusing barriers can be used to generate uniform light into the sensing probe. Filters can be added between the light source and the sensing probe to adjust the wavelength of the light directed into the sensing probe.

[0064] Light source alignment is important for compressive sensing probes if good images are to be obtained that do not require extensive post-processing. In theory, with no load or deformation, total internal reflection should not allow any light to escape, but due to the presence of debris in the elastomer, and due to the angular range of the light injected into the elastomer, some light will escape the elastomer even though nothing is touching the surface of the elastomer. This low intensity light is found to form an interference pattern. With the light source properly aligned and balanced, the interference pattern will typically form concentric circles, such as Figure 5A When the light source is not properly aligned, the interference image becomes skewed and asymmetric.

[0065] To calibrate the light source, a compression-induced image is obtained. Typically, those lights that form a patch of light in the middle are the brightest and can be represented as grayscale. Then, using available methods such as Otsu's thresholding, the grayscale is segmented into a black background and a white image, and optionally one or more intermediate (gray) rings, as Figure 5BAs shown. The center of the segmented image is then determined. The center of the image is then moved by adjusting the illumination direction of the light source until it is at or close to the center of the probe. For example, in Matlab software, the image can be approximated as an ellipse, and the "regionprops" function can be used to determine the center and eccentricity of the ellipse. In order to measure the "roundness" of the region, Matlab's "regionprops" command also returns the eccentricity of the region by specifying an ellipse and calculating the ratio of the distance between the focus of the ellipse and the length of its major axis. If the region is a circle, the eccentricity value is zero, and if it is a straight line, the value will be one. Figure 5B The small circles in the figure represent the centroids of the layers (excluding the background). The eccentricity of the central (white) segment is 0.0066, the eccentricity of the inner (light grey) ring is 0.27, and the eccentricity of the outer (dark grey) ring is 0.83. Figure 5B It can be seen that the two inner regions are fairly well centered. Figure 5C Three small circles marking the centers of the background (black), outer ring (darker grey), and inner ring (lighter grey) regions are shown, with eccentricities of 0.76, 0.64, and 0.86, in order from left to right.

[0066] If the image is evenly divided into three regions, such as Figure 5B and 5C As shown, the regional centroid and eccentricity score can then be calculated for each region. For the aligned images, the eccentricity scores and centroid positions of the two inner regions are very close. The proposed method for minimizing the uneven interference and thus aligning the LED is to adjust its position so as to produce a three-segment image with the centroid close to the center of the PDMS region and the eccentricity score below the maximum acceptable level. This level can be found experimentally by comparing the quality of the alignment with the accuracy of the inclusion size estimation. The LED adjustment mainly involves tilting or moving the LED within its tube to bring it closer to or further away from the PDMS. The light source is then adjusted to obtain a circular, uniform, compression-induced image.

[0067] This method can be used to detect damage to the sensing probe 36. If the image is severely asymmetric without any load, such as Figure 5C As shown, we suspect that the probe elastomer 46 is damaged, such as a cut or crack. We can use this method to detect damaged probes (elastomers). When the probe is applied to a rigid object, PDMS requires consistent deformation for image processing to estimate the object stiffness. In daily use, PDMS experiences various forms of stress and strain. The material may eventually fatigue and tear along the material. Tearing is defined as the continuous separation of PDMS, allowing air to enter the PDMS. The tear can be characterized by the length through the PDMS surface and the depth through the PDMS cross section.

[0068] exist Figure 5C In the PDMS image, damaged PDMS produces an image with bright spots near the edge of the PDMS. This causes the centroids of different regions to be far off center and produce higher eccentricity scores due to the rectangular shape. If these two values ​​deviate sufficiently, the PDMS can be identified as misaligned or damaged by the image analysis software, and the user can be prompted by the software to remove the hygiene cover and inspect the sensing probe for LED misalignment or PDMS damage.

[0069] The overall intensity of the image changes depending on the illumination from the LED and the ambient light. A threshold for the overall image brightness will be set to avoid frequent false alarms.

[0070] Now also refer to Figure 6 The main purpose of the constant current driver circuit 50 is to provide a constant current so that the light source can provide a uniform light level even if a natural voltage drop occurs (for example, because the power unit 40 relies on a battery and the battery is low). Figure 6 , and uses a Linear Technologies LT1932 LED driver 57, which is controlled by a SparkFun Electronics DEV-12640 Pro Micro development board 59 with an Arduino processor. The Pro Micro 59 also receives input from a force measurement unit 38, which is a TE Connectivity FC22 load sensing cell. The LED circuit is designed to enable the user to control the light intensity of the LED through a dimming option. Figure 6 As shown, the Pro Micro 59 also has a jack for input from an image sensor (IDS Imaging Development Systems GmbH UI-1220SE) in case for any reason it is preferred to use the dedicated imager on the front accessory 24 rather than the camera 32 on the smartphone 22. In the illustrated embodiment, the Arduino processor includes a Bluetooth transceiver. Alternatively, a separate Bluetooth transceiver may be provided.

[0071] The applied force can be measured by one or more force sensors 38 in the front end attachment 24. The force sensor 38 can be a pressure sensor, a load sensing unit, a strain sensor, a piezoelectric sensor, a strain gauge, etc. The direction of the applied force is important. The front end 24 or the mobile device 22 can have attitude and position sensors to determine the direction of the applied force in an absolute manner. If the direction of the applied force is measured by various sensors 38, then a more accurate result will be obtained, so that the direction of the applied force relative to the device 20 and in particular any unevenness of the applied pressure can be detected.

[0072] One possible embodiment is the FC22 compressive load sensing unit with vertical applied pressure. The FC22 uses micromachined silicon piezoresistive strain gauges (Measurement Specialties, 2012). The FC22 measures direct force, which is more reliable than alternative sensors that measure force through pressure bladders. Because the FC22 series includes micro-fusion technology, it eliminates organic epoxies that are sensitive to aging, which provides excellent long-term span and essentially unlimited cycle life expectancy. An exemplary FC22 force sensor or applied force measurement unit 38 would be integrated into the frame of the front end attachment 24, which then creates a stable platform for force measurement. To improve the accuracy of the analog force sensor data, a buffer can be used to store multiple values ​​of the force sensor, which can then be averaged.

[0073] The force sensor 38 is connected to a processor 44, such as an exemplary Pro Micro microcontroller 59. The force data will be synchronized with the CIS image. In an embodiment, a Bluetooth-enabled microprocessor 59 is connected to the force sensor 38. When a capture image command is generated on the smartphone 22, the smartphone 22 sends a command to capture a CIS image to the smartphone camera 32, and sends a Bluetooth command to the microprocessor 59 to obtain the applied force information from the force sensor 38. The force data is then sent by Bluetooth to the software unit 34 of the smartphone 22, where the force data is written into the CIS image metadata. There may also be a hardware trigger from the smartphone 22 to the microcontroller 59. The hardware trigger will result in a more accurate force / image synchronization.

[0074] The processing unit 44 has three main functions: obtaining the raw force sensor data and converting it into usable force information, reading the light brightness value, and controlling the light brightness. The processing unit can be a microprocessor, FPGA or computer.

[0075] One possible embodiment of brightness control is accomplished by the application software 34 in the mobile device 22. The dimming function can be controlled by varying the output of the PWM signal between 0-255 in increments of 30, where 0 is low and 255 is high. It is also contemplated that the processor 34 of the smart phone 22 can be used as the microprocessor 59, and the software application on the phone 22 is programmed to receive the force sensor data and the brightness data and use it to control the brightness.

[0076] Now also refer to Figure 8 , the main function of the communication unit 42 is to provide useful data to the mobile device 22 and obtain commands from the mobile device 22. This can be a wireless communication such as Bluetooth or Wi-Fi, or it can be a wired connection between the front-end accessory 24 and the mobile device 22. One possible embodiment is a Bluetooth connection between the microcontroller 59 of the front-end accessory 24 and the processor 34 of the smart phone 22, followed by a Wi-Fi HTTP or HTTPS connection between the smart phone 22 (especially its communication unit 30) and the cloud server 26.

[0077] Initial communication begins with the microcontroller 59 in the front-end accessory 24 and the smartphone 22 connecting via a Bluetooth Smart connection. There are two main components in a Bluetooth connection: a peripheral device and a central device. In this embodiment, the smartphone 22 is the central device and the microcontroller 59 is the peripheral device: this is similar to a client and server relationship.

[0078] An unconnected peripheral device advertises data such as its name, signal strength and universally unique identifier (UUID). Once connected, the peripheral device will share its services; the service that will be used is the Universal Asynchronous Receiver / Transmitter service. It basically acts as a data pipe between the microcontroller 59 and the smartphone 22. The service has two properties, "Tx" and "Rx", which will be used to transmit force data and control dimming. Suitable security protocols to ensure that the smartphone 22 is connected to the correct front end unit 24 and to avoid external interference or interception of the wireless transmission are well known and will not be described here for the sake of brevity.

[0079] To use this method, an instruction set is created to handle the different functions required to obtain force data from the force sensor 38 and control the dimming of the LED 48.

[0080] If available, the communication between the smartphone 22 and the cloud server 26 (running a PHP script in this embodiment) will be done over Wi-Fi. Images are sent in, for example, JPEG format, and the associated data is sent in a JSON file using the HTTP POST method. To retrieve data from the cloud server 26, the smartphone application 34 will use the HTTP REQUEST method. By using the JSON file, the data is parsed in a way that makes retrieving information simple. Once communication is established, the software must ensure that the messages are in the same "language": in an embodiment, all data passed between the platforms conforms to the UTF-8 string encoding.

[0081] The power unit 40 provides all necessary power to the front end attachment 24. That would include the light source(s) 48, the constant current driver 50, the force sensor 38, the processing unit 44 and the communication unit 42. The power unit 40 may be a battery or a fuel cell.

[0082] In the case where the mobile device 22 is a smartphone, the camera, video camera, accelerometer, gyroscope and communication hardware available in a typical smartphone can be utilized, in particular the camera 32, application software 34 and communication unit 30. One possible embodiment is the iPhone. The iPhone 6 has an 8 megapixel and 1.5 μm pixel video camera, accelerometer, gyroscope and compass, Wi-Fi 802.11.

[0083] from Figure 7 As can be seen, the camera 32 observes the elastomer 46. The front end attachment 24 is designed so that the elastomer 46 is visible from the mobile device camera 32. The camera 32 is able to focus on the edge (side) of the elastomer 46. The camera's auto focus is disabled and the focal length is set in software or firmware. The focal length is fixed for each combination of camera model and front end device model and can therefore be pre-programmed. The camera 32 captures light escaping from the elastomer 46 as the elastomer 46 is compressed and deformed by the target object. The camera 32 can capture a single image, multiple still images, or one or more video images.

[0084] The main function of the application software 34 is to obtain CIS image data and metadata such as applied force, user number, case number, date, time and dimming value from the force sensor 38 and send this information to the cloud server 26. The data is encrypted and compressed in the application software 34. In addition, the application software 34 will receive the malignancy score and other output information returned from the local processing unit 28.

[0085] The built-in wireless communication capabilities of the mobile device are used to send the acquired data and receive the results. The communication can be done via Wi-Fi, LTE, 3G or Bluetooth. Other ftp or http sockets can also be used.

[0086] The compression induced images obtained from the CIS imager are sent via Wi-Fi to a remote cloud server 26 or a local computer 28 running a mechanical property estimation algorithm. The obtained images with the corresponding histopathological results are stored in a cloud database. This database is used to calculate the malignancy score. This allows the server 26 to maintain a comprehensive database for more accurate performance evaluation. Therefore, the communication should be secure. In order to achieve efficient and secure communication between two entities, a communication scheme based on symmetric encryption is often adopted. Both symmetric or asymmetric encryption can be used to protect the data.

[0087] For symmetric encryption, secret keys are often transmitted with the help of asymmetric encryption methods, which require the support of public key infrastructure. Even if only the key is encrypted asymmetrically, it will cause a lot of computational and communication overhead for the CIS. Worse, if there is any risk that the secret key is leaked, then the CIS must generate another secret key and transmit it using expensive asymmetric encryption methods.

[0088] To overcome these problems, the present embodiment uses a different secure communication protocol. The proposed communication protocol is quite safe without introducing a public key infrastructure. In addition, a secret key is generated in a very efficient manner. Specifically, during the initialization process, there is no shared secret key between the smart phone 22 and the cloud server 26. Instead, the smart phone 22 and the cloud server 26 use plain text to randomly chat with each other without sending any sensitive or secret information. After several rounds of interaction, both the smart phone 22 and the cloud server 26 use the hash value of their chat content to generate a secret key. In order to generate the same secret key, the attacker must intercept all the plain text in the chat. Fortunately, severe packet loss often occurs in wireless communication, so it is difficult for an attacker to obtain all the chat content. If the chat entities cannot hear each other clearly, they can ask for repeated messages. Eavesdroppers can't do that. The robustness of security is based on the lack of robustness of wireless communication (the characteristics of packet loss). In this way, a secret key can be generated without introducing a public key infrastructure. In order to make communication more secure, after each successful communication, both entities update their secret keys. Therefore, even if an attacker steals the secret key, it will expire quickly, and once the attacker loses data packets, the attacker will no longer be able to match any future key updates.

[0089] The CIS data can be sent directly or through a cloud service 26 to a local processing unit 28. Preferably, the CIS unit 20 is connected to a cloud server 26 and the local processing unit 28 obtains the data from the cloud server 26. In the cloud, the encrypted compression induced image with metadata can be stored. The mobile device 22 and the local processing unit 28 communicate with the cloud server 26. The cloud server database stores the data and metadata until it is ready for data processing on the local computer 28.

[0090] One embodiment of the cloud server 26 acts as a repository for all data and has two main components, a PHP script and a MySQL database. There are two PHP scripts: one receives the data sent from the smart phone 22 and stores them in the appropriate directory; the second goes to the appropriate directory and returns the resulting information to the smart phone 22.

[0091] The local processing unit 28 obtains the CIS data and processes the data to obtain mechanical property information. The CIS data and metadata are stored and processed in the computer 28. The local processing unit 28 can be in the cloud 26, but in this embodiment, for security purposes, the local processing unit 28 is located in the responsible doctor's own location. Therefore, the data in the cloud 26 will never be decrypted. The processed data is then returned to the originating mobile device 22. The user of the system is then able to obtain the desired information on the mobile device screen.

[0092] Now also refer to Fig.9A , Fig. 9B and Fig. 9C 9, one embodiment of the mobile platform compression induced imaging system 20 includes a frame 60. The purpose of the frame 60 is to hold the front end 24 and enable it to be seamlessly attached to the mobile device 22. The elastomeric element 46 of the sensing probe 36 is visible to the imaging unit 32 of the mobile device 22. The force sensor 38 is positioned to obtain accurate information of the applied force.

[0093] The frame 60 includes a cradle 62 having raised sides 64, on which the smartphone 22 can be placed, with the camera 32 aligned with the opening 66. Thus, here the frame 60 and cradle 62 define a receptacle in which the mobile device 22 can be received. If the smartphone 22 has an autofocus sensor alongside the camera 32, then the opening 66 can be elongated as shown so that the camera can focus on a target at the edge of the sensing probe 36. Alternatively, the autofocus can be overridden in software and the camera 32 forced to a known focus setting.

[0094] 9 is designed for an Apple iPhone 6, but many other available models of smartphones 22 have similar general configurations and will require at most slight modifications to the dimensions of the bracket 62. The illustrated frame 60 was created in SolidWorks CAD software and printed using an Objet 3D printer.

[0095] The front attachment frame 60 is the backbone of the system as it contains all of the LED circuits 50, PDMS probes 36, force sensors 38, and smartphone 22 and keeps them aligned. The goal of this design was to create a lightweight, easy to hold device that could accurately capture tactile images and force sensing information.

[0096] The cradle 62 holds the smartphone securely in place to allow some pressure to be applied to the phone without moving it or damaging any components. The cradle 62 does not interfere with the view of the camera or smartphone screen, or with access to any buttons, thereby maintaining full phone functionality. If wired communication is used between the smartphone 22 and the front-end processor 59, the cradle 62 may have a connector plug that fits into a port in the smartphone 22 when the smartphone 22 is inserted into the cradle 62. The cradle 62 is used here as a handle for the user to hold the CIS assembly 22, 24 and press the sensor 36 (particularly the elastomer 46) against the target.

[0097] Directly below the smartphone cradle 62 is a storage compartment 68 for the electrical components. The compartment 68 is sized to contain the microcontroller 59, the battery pack 40, the power switch, the LED driver 50, the force sensor circuitry, and some wiring. The compartment 68 is enclosed by a cover for protection and aesthetics, but has external access points for the power switch and the necessary wiring for the LEDs and force sensors.

[0098] On the front side of the frame 60 is an area 70 for image capture. The hole 66 for the camera 32 is centered on the location of the camera 32 on the smartphone 22 itself. The distance from the camera 32 to the sensor elastomer 46 is selected to be within the focal length range of the camera 32, which in the embodiment of FIG. 9 is provided by the manufacturer as a minimum of 31 mm. Additional lenses (not shown) may be added in front of the smartphone camera 32 to reduce the minimum focal length or to capture an image of an appropriate size.

[0099] A glass or other rigid, transparent sheet material 72 may be incorporated between the elastomer 46 and the camera 32 (see Figure 7), serving as a rigid support for the elastomer 46. The glass / plastic 72 supports the back of the elastomer 46 while allowing the camera 32 to see through. The glass / plastic 72 and the elastomer 46 are placed in a rectangular cutout 74 in a holder 76 at the front end of the frame 60, and the elastomer 46 fits securely in the cutout against the glass / plastic 72. Four holes 78 symmetrically placed in the holder 76 around the cutout 74 are access points for the LEDs 48. The LEDs 48 are secured within a metal spacer lining the holes 78. All wiring for the LEDs is directed away from the imaging area and arranged with as few and as small lines as possible so that the holder 76 can move freely. It is also conceivable that a rigid support for the elastomer 46 can be provided by a holder 76 around the periphery of the elastomer 46.

[0100] 9 , the applied force is estimated from the accelerometer and gyroscope of the smartphone 22. Alternatively, or in addition, a thin force sensor may be placed on the frame 60 where it contacts the tissue at the edge of the elastomer 46.

[0101] Now also refer to Fig. 10A and Fig. 10B , collectively referred to as FIG. 10, and reference Figure 7 , the second embodiment of the frame 80 is similar in principle to the first embodiment and will not be described again unnecessarily. In the frame 80, the corners of the smartphone 22 including the camera 32 are inserted into slots in the bracket 82. The frame 80 and the bracket 82 again define a receptacle for holding the mobile device or smartphone 22. A housing 84 behind the bracket 82 and aligned with the holder 76 for the elastomer sensor 46 contains all the electronics and serves as a handle. The housing 84 is not rigidly attached to the bracket 82. Instead, they are connected via the force sensor 38, so that when a user holding the housing 84 presses the sensor 36 (in particular the elastomer 46) against a solid object, the force sensor 38, here again in the form of the housing 84, detects and measures the force applied to the handle. For simplicity, in Figure 7 Only a single force sensor 38 is shown in FIG. 1 , but as mentioned above, multiple force sensors may be used to improve accuracy and detect any non-uniformities in the applied pressure.

[0102] Now also refer to Fig.11 , in order to determine the mechanical characteristics of the target 100, it is necessary to obtain data from the front-end accessory 24 and the mobile device 22. An embodiment of the method of data collection is implemented in three main hardware components: the microcontroller 59 of the front-end accessory 24, the mobile device (e.g., smartphone) 22, and the local processing unit 28. The method is responsible for the acquisition of data, the processing of CIS data, and the communication between the hardware.

[0103] Although the above embodiments describe manually applying force, it is also conceivable that Figure 7 As schematically shown in , a motor 85 such as a linear actuator, a screw drive, a pulley or a timing belt can be incorporated into any embodiment to automatically apply the desired force. For example, the motor 85 can be coupled between the housing 84, the bracket 82 or the holder / front end 24 and a fixed or non-moving surface to automatically drive the optical waveguide or compression sensing probe 36 toward the target 100 when activated. This will allow automatic force application rather than manual pressing. In one application, the motor 85 drives the optical waveguide 46 toward the target area 100 with a uniform movement of about 5 to 7 mm in about three seconds to obtain a tactile image.

[0104] In the front attachment 24, the processor 59 obtains the force sensor data and transmits it to the mobile device 22. The processor 59 also controls the brightness of the lighting and communicates with the mobile device 22. As an example, the method can be programmed in C language using the Arduino IDE microprocessor 59. Examples of the functions of each component are as follows Fig.11 shown.

[0105] The mobile device unit 22 collects the CIS image and the applied force data, sends instructions to the front-end processor, and sends the CIS image and related data to the local processing unit 28 and requests the results from it. One possible embodiment of the data acquisition method is an iOS application software. Fig.12 , the software interface on the display screen of the smart phone 22 provides a real-time camera view 102 of the probe 36, a slider 104 to adjust the brightness of the LED 48, a field to add patient information, a display 106 of the force reading, and finally a button for the user to capture a photo. Once the capture button is pressed, the captured photo is displayed on the screen and the user is given the option to either upload the photo or retake the photo. A video image can also be captured. The image is correlated with the applied force information from the force sensor 38.

[0106] The application software 34 in the smartphone 22 captures the CIS image, force data and metadata. Fig.13 In one possible embodiment of the data acquisition process, when the operator presses the capture button, the sequence of data collection begins by sending a command requesting force information via Bluetooth to the front-end microprocessor 59. The front-end microprocessor 59 reads and stores the current force value. The current force value can be a single value, an average of values ​​from two or more sensors, or a list of values ​​from two or more sensors. Fig.12 and Fig.13In the example, the two force sensors 38 report separately. Because the program runs asynchronously, the stored values ​​are sent one at a time to prevent errors from being introduced during transmission. At the same time, the application software on the smartphone 22 continues and collects images using the camera 32. Once the applied force information is returned to the smartphone 22, the captured image and force data are displayed on the smartphone 22. The user is then prompted to either retake the image or upload the image.

[0107] When the user selects the upload button, a connection to the server is initiated. A file containing the image and metadata is prepared. The metadata may contain any or all of the date, time, dimmer value, patient ID and user ID, and force information. This data may be used to form an image name in a format that can be used efficiently to sort the data. Fig.14 An example of data flow when uploading data is shown.

[0108] In the case where it is desired to capture multiple images at different applied forces, the smartphone 22 can record a video while repeatedly polling the front end 24 for force data. The force data is then analyzed to find periods where the rate of change of force over time is relatively uniform, and a portion of the video, or two or more individual frames at the desired force or time intervals, are selected for further analysis. If the smartphone 22 has sufficient processing power, the selection can be performed before uploading, and the selected frames are presented for user approval. If the smartphone 22 does not have sufficient processing power to perform the selection, the unselected data can be uploaded and the selection performed at the local computer 28, but at a greater expense in communication usage.

[0109] Sizing Method for CIS on Mobile Platforms Using 3D Interpolation

[0110] One of the mechanical properties that the mobile platform CIS determines is the size of the target 100. Here, we describe a method to obtain the size of a sub-surface or surface target.

[0111] In order to select the best CIS image, a force versus time graph is plotted. Smooth linear regions are selected for processing the images. These images are used in the mechanical property estimation algorithm. It is desirable to filter out noise from the CIS images. Noise may be due to, for example, stray ambient light entering the system, or to non-imaging light from LED 48 if the position of the LED is not ideal. The maximum intensity and / or the sum of the intensities of the CIS images are used to estimate the target size. In this embodiment, the shape of the inclusion is assumed to be spherical. A 3D interpolation model is provided, which is used to estimate the size of the tumor 100 based on the compression induced image. The 3D interpolation method involves the applied normal force F, the number of pixels N on the compression induced image, and the distance N from the tumor to the image. pand the diameter D that contains the image. We model multiple surfaces based on these three parameters. Once we have the model, we obtain the force F from the mobile platform CIS and the approximate depth from the user. We then use the modeled surface to estimate the size. We develop multiple 3D interpolated surfaces of the form shown in equation (1) for a range of depth layers based on experimental data.

[0112]

[0113] Model coefficient p ij , defines the modeled surface. The indices n and m in equation (1) represent the order of the polynomial used for size estimation. The coefficient p ij It was obtained empirically according to the model described below.

[0114] One possible embodiment is to separate the sizes into two cases. We developed two sets of 3D interpolation methods: one for large inclusions (e.g., we used two diameters of 11.9 mm and 15.5 mm to generate the initial interpolation surface) and the other for small inclusions (e.g., we used diameters of 8.0 mm and 9.9 mm to generate the initial interpolation surface). Therefore, in order to reduce errors, we have two different interpolation surfaces: one for small inclusions and the other for larger inclusions. The interpolation surfaces from each set are determined for three different depth layers (3 mm, 6 mm, and 9 mm in this example). The values ​​of the developed 3D interpolation surface parameters reflect the depth and size changes of the inclusions within the tissue phantom. The physician can qualitatively estimate the depth (shallow, medium, or deep) and size (large or small) of the tumor so that the appropriate surface according to the physician's description can be used.

[0115] Using the developed 3D interpolation surface, we can specify the applied force F and the number of pixels N on the image by p to estimate the size of the inclusion. The size of the inclusion is found using its approximate depth based on force information and the number of pixels. We have multiple interpolation surfaces at different depths. The selection of a specific 3D interpolation surface is based on the approximate depth and size of the inclusion. For the human data experiments, the physician estimated the approximate size and depth of the inclusion. This 3D interpolation method is then used to calculate a more accurate size and depth. An initial interpolation surface can be generated from the developed model, but as more data is collected, a more accurate interpolation surface can be determined from the actual data.

[0116] Method for determining relative softness index

[0117] The compression experiment of the moving platform CIS is similar in some respects to the conventional tensile test to determine the mechanical properties of inclusions. One of the properties we consider is the softness index, which is the degree of softness / hardness of the target. As opposed to the rigid steel compression surface used in conventional hardness measurements, the present method uses a soft silicone probe to compress tissue with soft inclusions. The size and shape of the deformation of the silicone probe element of the moving platform CIS gives information about the tissue with inclusions. However, the difference in the stiffness of the compression surface affects the output of both methods. In a conventional tensile experiment using a rigid probe, the sample deforms, and the deformation of the rigid sample / inclusion is less than that of the soft inclusion. In the current measurement method, what is measured is the deformation of the soft probe, and for harder samples, the deformation of the probe is larger, while for softer samples, the deformation of the probe is smaller. The softness index S obtained CISS It is the reciprocal of the elastic modulus.

[0118] Elasticity describes the ability of a tissue to recover its shape after the applied stress is removed. The response of human skin and soft tissue to compression, as well as the response of tumors, are examples of elastic tissue recovery. Moreover, biological tissues are nonlinear and viscoelastic materials. We used a very small range of indentation (up to 20% of the depth). In all mobile platform CIS experiments, we assumed linear elastic behavior of the tested materials. We used the change in indentation with a soft silicone probe to capture the deformation of the tissue caused by compression. We then estimated the softness index of the tumor area, which is a measure of the relative stretch properties.

[0119] The mobile platform CIS softness index calculation with a tissue phantom is designed to replace conventional elastic modulus measurement techniques using compression.

[0120] In mobile platform CIS applications, the stress σ of each CISS image k z (k) is calculated as follows:

[0121]

[0122] F(k) is the force applied in the z direction, F ref is the force value achieved at the reference point. The reference force corresponds to the first non-empty tactile image during the experiment. A C is the contact area of ​​the mobile platform CIS (e.g. 1134mm 2 ), the contact area includes the solid frame and the elastic silicone probe area. In this example, it is assumed that the entire front face of the probe including the frame is pressed flat on the flat surface of the medium containing the sample 100. If the landing surface is not flat, A C Can be adjusted accordingly.

[0123] Vertical deformation rate d of mobile platform CIS z(k) is expressed as

[0124]

[0125] I(k) represents the sum of the intensities of the kth compressed image, and I ref is the reference force F ref The reference image and force are selected when a relatively large CIS image (occupying at least 25% of the screen) appears on the screen. For example, in the case of breast cancer in a human patient, this may be between 3N and 15N. Due to the fact that the deformation of the probe in the z direction is proportional to the change in the tactile image intensity, we use the sum of the pixel intensity values ​​or the maximum intensity value to estimate the indentation of the probe.

[0126] Softness Index S CISS is a measure of the stiffness of tissue with inclusions. It is calculated as the slope of the moving platform CIS stress-deformation curve calculated for the tumor region:

[0127]

[0128] Due to the mobile platform CIS hardware design, the elastic silicone probe deforms faster when in contact with hard inclusions than with soft inclusions. This results in a softness index that is inversely proportional to the true elastic modulus (Young's modulus) of the imaged inclusion. CISS The larger the value, the softer the tumor. CISS The smaller the size, the harder the tumor.

[0129] Absolute elasticity estimation method using deep learning (Deep Brief Net)

[0130] Here, we propose a method to obtain the absolute elasticity of an object. Here, the example is an application for human tumors, but other applications are possible. To calculate the absolute Young's modulus (elasticity) of a tumor, we use deep learning. We will model the mechanical properties of the object (such as the size of the deformation due to elasticity). The methods are divided into forward methods and inverse methods. See Fig.15 and 16 The forward method provides a stress map of the skin. The reverse method uses deep learning to provide the elasticity of the target, such as Fig.16 shown.

[0131] Modeling and forward methods. The goal of the forward algorithm is to find the relationship between tissue inclusion parameters and CIS data. Three-dimensional analysis leads to a novel method for predicting human tumor characteristics; it can be directly merged with palpation by health care providers. The finite element method (FEM) is used to demonstrate properties such as the shape, size, and elasticity of the tumor. The input parameters are:

[0132] Tissue loading (compression of tissue surface), l

[0133] Indentation diameter, d

[0134] Skin stiffness, E r

[0135] The aim was to study the effects of skin in biological tissue associated with loads applied on the tissue. We used the engineering simulation software package ANSYS (PA). Other modeling software may be used. The finite element model (FEM) included a sensing probe, soft tissue (e.g., representing human breast tissue) and a harder inclusion (e.g., representing a tumor). We modeled the breast and tumor as elastic isotropic elements. The stress distribution was obtained from dynamic probe images. From these models, we obtained the maximum deformation, total deformation and deformation area. The device was compressed against the breast tissue with the tumor. The deformation of the sensing probe was captured. We correlated the size, depth and elasticity of the tumor with the maximum deformation, total deformation and deformation area through simulation.

[0136] The absolute Young's modulus was determined using the inverse method. Although the results of the forward method can show the presence of a tumor, they do not provide detailed information about the characteristics of the tumor. One technique to solve the inverse problem is to perform deep learning on the forward results. The first part is the data acquisition and construction of tactile mapping using FEM. In the second part, we need an inverse model that takes the CIS map as input and produces the size and elasticity of the imaged tumor and tissue as output. The inverse algorithm estimates the elastic modulus, depth, and size of the tumor based on the maximum deformation, total deformation, and deformed area obtained from CIS. For this inverse modeling, we use a deep neural network method called "Deep Brief Net" (DBN). This DBN method improves the accuracy of the inverse algorithm (LeCun et al., 2015). Conventional artificial neural networks (ANNs) suffer from overfitting problems, which leads to poor generalization performance. DBNs can be pre-trained in an unsupervised manner to avoid overfitting, and DBNs with latent variable space in their deep structure can represent complex nonlinear functions that otherwise cannot be efficiently represented by ANNs with shallow structures (i.e., only one or two hidden layers). We use DBNs to improve the accuracy of the inverse algorithm. In this way, we propose to obtain the absolute Young's modulus of the target.

[0137] Risk Index Determination Method

[0138] In our embodiments, we describe a tumor risk index that can be obtained using a mobile platform CIS. We utilize quantified size and softness index information to obtain a tumor risk index. Other mechanical properties, such as fluidity and softness, can be incorporated into the risk index. Here, we describe two different embodiments of a method for determining a risk index.

[0139] Simple weighting method. As a first example, we calculate a risk index with equal weighting between size and softness. We first normalize the size between 1 and 5, where 5 is the largest size, such as 2 cm, and 1 is the smallest size, such as 2 mm. We also normalize the softness to 1 to 5, where 5 is hard and 1 is soft. The softness can range from 0 to 250 × 10 -9 / Pa, divided into 5 levels. Then, we multiplied the size and softness by 0.5 and added them together. Therefore, we weighted the size information and the softness information equally, and the risk index was a number between 1 and 5. The larger the size of the tumor, the more likely it is to be malignant. The softer the tumor, the more likely it is to be benign.

[0140] Data classification using convolutional neural network methods. Another embodiment is to use machine learning methods such as neural networks to classify targets. CIS signatures such as size, activity, and elasticity of tumors will be obtained from the mobile platform CIS. Once the CIS data is obtained, the data must be fused to develop a tumor characterization model. We first create a database of size, depth, activity, and elasticity with malignant tumor information. Then, we use a convolutional neural network (CNN) to classify tumors.

[0141] Even if CIS data were available, characterizing a tumor as malignant or benign is not simple. A computer must fuse the information and provide the patient with the likelihood that the lesion is malignant.

[0142] refer to Fig.17 , we use the Convolutional Neural Network (CNN) approach (Ji et al., 2013) to classify lesions using size, elasticity, and depth information. We will use CNN for tumor classification. A Convolutional Neural Network (CNN) is similar to a neural network but is designed to accept images as input. We present the CNN architecture; convolutional layers, pooling layers, and fully connected layers. The convolutional layer is represented by the following equation:

[0143]

[0144] in, is the value of the neuron at position x of the jth feature map in the i-th layer. Each pooling layer corresponds to the previous convolutional layer. We use max pooling,

[0145]

[0146] where u(n, 1) is the window function of the patch of the convolutional layer, and a j is the maximum value in the neighborhood. These layers are stacked to form the complete CNN architecture. Each layer accepts a 3D volume as input and transforms it into an output 3D volume through a differentiable function. For the application of breast tumors, we obtain CIS images and classify these images along the spatial and temporal dimensions. Then, we develop a CNN to classify benign and malignant tumors as output.

[0147] After correlating the output with the tumor pathology results, we calculated the probability that the tumor is malignant. Once the database is complete, the probability that the tumor is malignant is calculated as a risk index.

[0148] The mobile platform compression induced imaging system (CIS) utilizes only the camera, communication and limited data processing capabilities of the smartphone. Specific embodiments have been described using smartphone current at the time of writing. However, as long as there is a camera in the smartphone and there is communication capability (Bluetooth, Wi-Fi or some successor yet to be developed), the CIS system can work. For new smartphones, the hardware framework of applications and accessories may need to be updated because the physical location of the camera may change. However, the principles of CIS do not depend on the specific functions or design of the smartphone. The application software will change depending on the type of smartphone and the version of the application creation software. For example, the above embodiment uses Xcode 8 and Swift 3.0.2. Similarly, the exact process for synchronization of tactile images and applied force data may depend on the specific smartphone hardware and operating system. However, it will be within the capabilities of ordinary technicians in this field to update this teaching from time to time to keep consistent with the smartphone technology at the time.

[0149] The light source alignment method, damage detection, absolute elasticity calculation method, and data classification method are all independent of smartphone technology changes.

[0150] It would be possible to provide a camera in the front attachment 24, rather than using the camera 32 of the smartphone 22. This would have the advantage of not requiring the bracket to be adjusted or redesigned each time the smartphone designer moved the camera, but would have the disadvantage of having to provide a separate camera (at considerable additional cost since manufacturers of existing devices do not enjoy the huge sales volumes and corresponding economies of scale of the large smartphone manufacturers).

[0151] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0152] The use of the terms "a," "an," and "the," and similar referents in the context of describing the invention (especially in the context of the following claims) should be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "having" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise indicated. The term "connected" should be construed as partially or fully contained within, attached to, or connected together, even if there are intervening things.

[0153] Recitation of ranges of values ​​herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein.

[0154] Unless otherwise noted herein or clearly contradictory to the context, all methods described herein can be performed in any suitable order. Unless otherwise required, the use of any and all examples or exemplary language (e.g., "such as") provided herein is only intended to better illustrate embodiments of the present invention, and does not impose limitations on the scope of the present invention. Various embodiments and elements can be interchanged or combined in any suitable manner as required. Therefore, any feature described in the specification and dependent claims should be understood to be applicable to other embodiments and other claims and can be combined therewith.

[0155] No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

[0156] It will be clear to those skilled in the art that various modifications and variations may be made to the present invention without departing from the spirit and scope of the present invention. It is not intended to limit the present invention to one or more specific forms disclosed, but on the contrary, it is intended to cover all modifications, alternative configurations and equivalents that fall within the spirit and scope of the present invention as defined by the appended claims. Therefore, the present invention is intended to cover modifications and variations of the present invention as long as they fall within the scope of the appended claims and their equivalents.

[0157] Citations

[0158] The following prior publications, some of which are discussed above, are incorporated herein by reference as if expressly set forth herein.

[0159] Dargahi, J. and Najarian, S., “Human tactile perception as a standard for artificial tactile sensing–a review,” International journal of medical robotics and computer assisted surgery, Vol. 1, No. 1, pp. 23–35, 2004.

[0160] Itoh A, Ueno E, Tohno E, Kamma H, Takahashi H, Shiina T, et al., Breast disease: Clinical application of US elastography for diagnosis. Radiology. 2006; Issue 239: pp. 341-50. [PubMed:16484352]

[0161] Ji S., Yang M., and Yu K., “3D Convolutional Neural Networks for Human Action Recognition,” PAMI, vol. 35, no. 1, pp. 221–31, 2013.

[0162] Krouskop, T., Wheeler T., Kallel F., Garra B., (1998) "Elastic Moduli of Breast and Prostate Tissues Under Compression", Ultrasonic Imaging, Vol. 20, pp. 260-274.

[0163] LeCun Y., Bengio Y., and Hinton G., “Deep Learning,” Nature, vol. 521, issue 7553, pp. 436–444, 2015.

[0164] Regini, E., Bagnera S., Tota D., Capanino P., Luparia A., Barisone F., Durando M., Mariscotti G., Gandini G., (2010) "Role of sonoelastography in characterizing breast nodules. Preliminary experience with 120 lesions", Radiol. Med. (La Radiologia Medica.) Springer. February 22, 2010.

[0165] Rivaz, H., Boctor, E, Foroughi, P, Zellars, R, Fichtinger, G, Hager G, (2008) "Ultrasound elastography: a dynamic programming approach", IEEE Trans Med Imaging, Vol. 27, No. 10, pp. 1373 - 1377.

Claims

1. A mobile platform imaging device for generating an image of an object using compression of a target area, comprising: a tactile sensor comprising an optical waveguide having a front surface arranged to contact the target area, the optical waveguide comprising at least a flexible and transparent first layer at the front surface, the tactile sensor further comprising a plurality of light sources configured to direct light into the optical waveguide, the optical waveguide being configured to substantially confine the light within the optical waveguide when the optical waveguide is not deformed, wherein the optical waveguide is configured such that when the first layer is deformed, at least some of the light directed into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by the tactile sensor being pressed against the target area, the tactile sensor comprising a diffuser positioned adjacent to the light sources, the diffuser for diffusing light from the light sources so that the light is evenly distributed within the optical waveguide; a support structure for maintaining the optical waveguide in position when a force is applied to the optical waveguide; a force sensor that detects a force applied to press the tactile sensor against the target area and outputs corresponding force information, the force sensor being mounted to allow calculation of a force applied in a direction perpendicular to the front surface of the optical waveguide; a first communication unit connected to receive force information from the force sensor; and a receptor for holding a mobile device having a second communication unit and an imager, positioned so that the imager can generate image information using at least some of the light scattered from the first layer; wherein the first communication unit is capable of communicating with the second communication unit, and the mobile device is capable of communicating with an external network; wherein a stress value is determined based on the applied force and the stress value is related to image information generated from the applied force; and Therein, a deformation of the first layer is determined based on the image information, and a softness value of the object is determined based on the deformation and the stress value.

2. The mobile platform imaging device of claim 1, the mobile platform imaging device being combined with the mobile device, wherein: The mobile device is a mobile phone.

3. The mobile platform imaging device according to claim 2, wherein: The mobile phone is programmed to generate image information based on at least some of the light scattered from the first layer, receive force information, synchronize the image information with the force information, and transmit the associated information to an external device.

4. The mobile platform imaging device according to claim 3, wherein: The associated information further includes information selected from the group consisting of a current date, a current time, a user ID of a current user of the mobile device, and a target ID of the object.

5. The mobile platform imaging device according to claim 1, wherein: The image information is selected from the group consisting of a single image, a plurality of images, and a video image.

6. The mobile platform imaging device of claim 1, wherein: The mobile device is programmed to exchange a plurality of messages with an external device, generate a hash from content of the plurality of messages, and encrypt synchronized information using an encryption key determined from the hash.

7. The mobile platform imaging device of claim 1, wherein: The receptacle is configured as a frame defining a cradle for selectively removably receiving the mobile device such that an imaging unit of the mobile device is positioned opposite and along a line of sight with the optical waveguide.

8. The mobile platform imaging device of claim 7, wherein: The frame includes a holder having a cutout for mounting the optical waveguide therein.

9. The mobile platform imaging device of claim 8, wherein: A plurality of holes are formed at intervals around the holder for mounting the plurality of light sources to provide selective side illumination of the light guide.

10. The mobile platform imaging device of claim 8, wherein: An opening is formed substantially centrally within the cutout to provide a line of sight between the imaging unit and the optical waveguide.

11. The mobile platform imaging device of claim 7, wherein: The bracket includes raised sides for engaging the mobile device.

12. The mobile platform imaging device of claim 1, wherein: The force sensor is located between the tactile sensor and the receiver for holding the mobile device.

13. The mobile platform imaging device of claim 1, further comprising a handle, wherein: The force sensor is located between the handle and the receptacle for holding the mobile device.

14. The mobile platform imaging device of claim 13, wherein: The handle is defined by a frame of the receptacle.

15. The mobile platform imaging device of claim 13, wherein: The handle is defined by a housing operably mounted to the frame.

16. The mobile platform imaging apparatus of claim 15, further comprising a motor operably mounted on the housing for controlling application of the force sensed by the force sensor to the tactile sensor via the receptor.

17. The mobile platform imaging apparatus of claim 1 further comprising a motor operably mounted on the receptor for controlling application of the force sensed by the force sensor to the tactile sensor through the receptor.

18. The mobile platform imaging device of claim 1, further comprising constant current driver circuitry for powering the plurality of light sources.

19. The mobile platform imaging device of claim 1, wherein: The support structure includes a rigid frame mounted against a surface of the optical waveguide spaced from the front surface.

20. The mobile platform imaging device of claim 1, wherein: The support structure is provided by the structural configuration of the waveguide.

21. The mobile platform imaging device of claim 1, wherein: At least a first layer of the optical waveguide is curved.

22. The mobile platform imaging device of claim 21, wherein: At least a first layer of the optical waveguide is hemispherical.

23. The mobile platform imaging device of claim 21, wherein: At least a first layer of the optical waveguide is semi-cylindrical.

24. The mobile platform imaging device of claim 1, wherein: At least a first layer of the optical waveguide is rectangular.

25. The mobile platform imaging device of claim 1, wherein: At least a first layer of the optical waveguide is an elastomer.

26. The mobile platform imaging device of claim 25, wherein: The elastomer is polydimethylsiloxane (PDMS).

27. A method of determining surface or sub-surface physical properties of an object using the mobile platform imaging device of claim 1, comprising: obtaining image information and corresponding force information under a plurality of applied forces; determining a deformation of the first layer based on the image information at each applied force; estimating the size of the object using 3D interpolation; determining, based on the applied forces, a stress value associated with image information generated from each applied force; as well as A softness value of the object is determined based on the determined deformation and stress values.

28. A method of determining surface or sub-surface dimensions of an object using a mobile platform imaging device that generates an image of the object using compression of a target area, the mobile platform imaging device comprising: a tactile sensor comprising an optical waveguide including at least a flexible and transparent first layer, the tactile sensor further comprising at least one light source configured to direct light into the optical waveguide, wherein the optical waveguide is configured such that when the first layer is deformed, at least some of the light directed into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by the tactile sensor being pressed against the target area; a support structure for maintaining the optical waveguide in position when a force is applied to the optical waveguide; a force sensor that detects a force applied to press the tactile sensor against the target area and outputs corresponding force information; a first communication unit connected to receive force information from the force sensor; and a receptor for holding a mobile device having a second communication unit and an imager, positioned so that the imager can generate image information using at least some of the light scattered from the first layer; wherein the first communication unit is capable of communicating with the second communication unit and the mobile device is capable of communicating with an external network; The method comprises the following steps: obtaining image information and corresponding force information at a plurality of applied forces; and estimating the size of the object using 3D interpolation; The obtaining step includes: obtaining a series of images over a period of time of varying force; and Images are selected from a portion of the time period during which the force varies more smoothly and / or more linearly with time than during other portions of the time period.

29. A method for aligning at least one light source of a mobile platform imaging device that generates an image of an object using compression of a target area, the mobile platform imaging device comprising: a tactile sensor comprising an optical waveguide including at least a flexible and transparent first layer, the tactile sensor further comprising at least one light source configured to direct light into the optical waveguide, wherein the optical waveguide is configured such that when the first layer is deformed, at least some of the light directed into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by the tactile sensor being pressed against the target area; a support structure for maintaining the optical waveguide in position when a force is applied to the optical waveguide; a force sensor that detects a force applied to press the tactile sensor against the target area and outputs corresponding force information; a first communication unit connected to receive force information from the force sensor; and a receptor for holding a mobile device having a second communication unit and an imager, positioned so that the imager can generate image information using at least some of the light scattered from the first layer; wherein the first communication unit is capable of communicating with the second communication unit and the mobile device is capable of communicating with an external network; The method comprises the following steps: applying the tactile sensor to a known object with a preselected force; processing the image information to identify asymmetries in the scattered light; and The at least one light source is adjusted to reduce the asymmetry.

30. A method for detecting cutting or damage of an optical waveguide of a mobile platform imaging device as claimed in claim 1, comprising: applying the tactile sensor to the known object with a preselected force; processing the image information to identify asymmetries in the scattered light; as well as When the asymmetry exceeds a threshold, a cut or damage to the optical waveguide is indicated.

31. A method of determining the softness of an object in a target area using a mobile platform imaging device, the mobile platform imaging device using compression of the target area to generate an image of the object, the mobile platform imaging device comprising: a tactile sensor comprising an optical waveguide including at least a flexible and transparent first layer, the tactile sensor further comprising at least one light source configured to direct light into the optical waveguide, wherein the optical waveguide is configured such that when the first layer is deformed, at least some of the light directed into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by the tactile sensor being pressed against the target area; a support structure for maintaining the optical waveguide in position when a force is applied to the optical waveguide; a force sensor that detects a force applied to press the tactile sensor against the target area and outputs corresponding force information; a first communication unit connected to receive force information from the force sensor; and a receptor for holding a mobile device having a second communication unit and an imager, positioned so that the imager can generate image information using at least some of the light scattered from the first layer; wherein the first communication unit is capable of communicating with the second communication unit and the mobile device is capable of communicating with an external network; The method comprises the following steps: determining a contact area based on image information; determining stress based on contact area and force information from the tactile sensor; determining a strain of the optical waveguide based on the image information and the determined stress; and The strain of the object is determined based on the strain of the optical waveguide, the contact area, and the stress.

32. The method of claim 31, wherein: The step of determining the strain of the optical waveguide uses at least one of a sum of intensities of pixels of scattered light and an intensity of a maximum intensity pixel of scattered light.

33. A mobile platform imaging device for generating an image of an object using compression of a target area, comprising: a tactile sensor comprising an optical waveguide having a front surface arranged to contact the target area, the optical waveguide comprising at least a flexible and transparent first layer at the front surface, the tactile sensor further comprising at least one light source configured to direct light into the optical waveguide, the optical waveguide being configured to substantially confine the light within the optical waveguide when the optical waveguide is not deformed, wherein the optical waveguide is configured such that when the first layer is deformed, at least some of the light directed into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by the tactile sensor being pressed against the target area, the tactile sensor comprising a diffuser positioned adjacent to the at least one light source, the diffuser being configured to diffuse light from the at least one light source so that the light is evenly distributed within the optical waveguide; a support structure for maintaining the optical waveguide in position when a force is applied to the optical waveguide, the support structure comprising a rigid support for resisting deformation of a rear surface of the optical waveguide opposite the front surface, the rigid support being configured so as not to impede capture of scattered light from the optical waveguide; a force sensor that detects a force applied to press the tactile sensor against the target area and outputs corresponding force information associated with the applied force; a first communication unit connected to receive force information from the force sensor; and a receptor for holding a mobile device having a second communication unit and an imager, positioned so that the imager can generate image information using at least some of the light scattered from the first layer; The first communication unit is capable of communicating with the second communication unit, and the mobile device is capable of communicating with an external network.

34. A mobile platform imaging device for generating an image of an object using compression of a target area, comprising: a tactile sensor comprising an optical waveguide having a front surface arranged to contact the target area, the optical waveguide comprising at least a flexible and transparent first layer at the front surface, the tactile sensor further comprising at least one light source configured to direct light into the optical waveguide, the optical waveguide being configured to substantially confine the light within the optical waveguide when the optical waveguide is not deformed, wherein the optical waveguide is configured such that when the first layer is deformed, at least some of the light directed into the optical waveguide is scattered out of the first layer, and wherein the first layer is deformed by the tactile sensor being pressed against the target area, the tactile sensor comprising a diffuser positioned adjacent to the at least one light source, the diffuser being configured to diffuse light from the light source so that the light is evenly distributed within the optical waveguide; a support structure for maintaining the optical waveguide in position when a force is applied to the optical waveguide; a plurality of sensors, each sensor configured to provide a signal for determining a force applied by the tactile sensor on the target area, the signal from the sensor providing information for calculating the force in a direction perpendicular to the front surface of the optical waveguide; a first communication unit connected to receive output signals from the plurality of sensors and store the output signals; and a receptor for holding a mobile device having a second communication unit and an imager, positioned so that the imager can generate image information using at least some of the light scattered from the first layer; The first communication unit is capable of communicating with the second communication unit, and the mobile device is capable of communicating with an external network.

Images