Texture detection apparatus, system and method for analysis

Abstract

Embodiments of the present disclosure relate to projecting light on a surface and analyzing scattered light to obtain spatial information of the surface and generate a three-dimensional model of the surface. The three-dimensional model can then be analyzed to calculate one or more surface characteristics, such as roughness. The surface characteristics can then be analyzed to provide a result, such as a diagnosis or product recommendation. In some examples, a mobile device is used to analyze the surface.

Description

Technical Field

[0001] This invention relates to apparatus, systems, and methods for collecting and analyzing surface texture data. Background Technology

[0002] Surface texture analysis (e.g., roughness, variability) is used across various industries. For example, surface texture can be analyzed to confirm adequate polishing of bearing surfaces during manufacturing to reduce friction. In another instance, surface texture can be analyzed to confirm the proper application of coatings to surfaces (e.g., paint). Data on surface texture can be acquired using a variety of methods, including interferometry, capacitance, high-resolution imaging, confocal microscopy, and electron microscopy. Acquiring data to calculate various properties of the surface texture may require specialized equipment. This can reduce availability and / or increase the cost of analyzing surface textures. Therefore, techniques for analyzing surface textures using less expensive and / or more readily available equipment may be needed. Summary of the Invention

[0003] According to examples of this disclosure, a mobile device can be used to acquire surface texture data and calculate one or more surface properties from the data. In some instances, the one or more surface properties can be used to generate diagnostics or recommendations.

[0004] According to at least one embodiment of this disclosure, an apparatus may include: an infrared light source configured to project a plurality of infrared light spots onto a surface; a sensor mounted in the same substrate or housing as the infrared light source, the sensor being configured to detect a signal in response to the infrared light scattered away from the surface; and a processor mounted in the same substrate or housing as the infrared light source and the sensor, the processor being configured to generate a three-dimensional (3D) model of the surface based at least in part on the signal detected by the sensor, calculate one or more properties of the 3D model, including at least one of the following: surface roughness, maximum peak height, particle size, lesion size, number of lesions, or volatility, or any combination thereof, identify a dataset associated with the 3D model based at least in part on the following: surface roughness, maximum peak height, particle size, lesion size, number of lesions, or volatility, and output data from the dataset via a graphical user interface (GUI) of the apparatus.

[0005] According to at least one embodiment of the present disclosure, a method may include: illuminating a surface with an infrared light source of a mobile device; receiving a signal in response to infrared light scattered on the surface from the infrared light source; generating a three-dimensional (3D) model of the surface based at least in part on the signal; generating values ​​of at least one property of the surface from the 3D model; and providing a dataset based at least in part on the values.

[0006] According to at least one embodiment of the present disclosure, an apparatus may include: an infrared light source configured to emit infrared light on a surface; a sensor configured to detect the infrared light scattered on the surface and generate a signal based at least in part on the detected infrared light; and a processor configured to generate a three-dimensional (3D) model of the surface based at least in part on the signal, calculate a plurality of surface properties based at least in part on the 3D model, identify a dataset associated with the plurality of surface properties, wherein the dataset includes at least one of diagnostics or product recommendations, and provides data from the dataset via a graphical user interface, wherein the infrared light source, sensor, and processor are contained on the same substrate or within the same housing.

[0007] According to at least one embodiment of this disclosure, a method may include: projecting a plurality of infrared light spots onto a surface; detecting infrared light in response to scattering of the plurality of infrared light spots on the surface; generating a signal based at least in part on the detected infrared light; acquiring an image of the surface with a camera; generating a three-dimensional (3D) model of the surface based at least in part on the signal; calculating a plurality of surface properties based at least in part on the 3D model; identifying a dataset associated with the plurality of surface properties, wherein the dataset includes diagnostics, product recommendations, or a combination thereof; and providing data from the dataset via a graphical user interface. Attached Figure Description

[0008] Figure 1 This is a schematic illustration of a computing device 100 arranged according to an example of the present disclosure.

[0009] Figure 2 This invention describes a technique for obtaining spatial information about a surface, based on examples of this disclosure.

[0010] Figure 3 An example of a machine learning model based on the examples of this disclosure.

[0011] Figure 4 A flowchart of a method according to an example of this disclosure.

[0012] Figure 5 A flowchart of a method according to an example of this disclosure.

[0013] Figure 6 A graphical representation of an application of an example according to this disclosure. Detailed Implementation

[0014] The following description of certain embodiments is exemplary in nature and is in no way intended to limit the scope of this disclosure or its application or use. In the following detailed description of embodiments of the apparatus, systems, and methods of the present invention, reference is made to the accompanying drawings, which form part of the invention and illustrate specific embodiments in which the described apparatus, systems, and methods may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the apparatus, systems, and methods disclosed herein, and it should be understood that other embodiments may be utilized and structural and logical changes may be made without departing from the spirit and scope of this disclosure. Furthermore, for clarity, detailed descriptions of certain features will not be elaborated where they would be obvious to those skilled in the art, so as not to obscure the description of embodiments of this disclosure. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of this disclosure is defined only by the appended claims.

[0015] An emerging area of ​​focus in surface texture analysis is the analysis of surface properties of biological tissues, such as skin. Surface roughness, variability, peak height, particle size, lesion size (e.g., acne, insect bites, scars, birthmarks, cancer), number of lesions, and / or other surface properties can provide information about tissue health. For example, the roughness and / or particle size can differ between normal skin and cancerous skin (e.g., melanoma, squamous cell carcinoma). However, similar to non-biological applications for analyzing surface texture, techniques for analyzing the surface texture of biological tissues may require specialized equipment, which can limit the ability to acquire data on the surface texture of biological tissues.

[0016] According to examples of this disclosure, a mobile device can be used to acquire surface texture data and calculate one or more surface properties from said data. In some instances, the mobile device can further analyze the surface properties to provide product recommendations and / or diagnoses. In the case of biological tissues, product recommendations may include cosmetics (e.g., moisturizers, toners, cleansers), and diagnoses may include diseases and / or conditions (e.g., cancer, infection, allergic reactions). In non-biological tissues, product recommendations may include processes or settings (e.g., extra time in a polishing machine, use of specific grit sandpaper), and diagnoses may include the state of the product (e.g., defective, acceptable). In some instances, product recommendations and / or diagnoses may be generated by a machine learning model based on surface properties.

[0017] Figure 1This is a schematic illustration of a computing device 100 arranged according to an example of the present disclosure. The computing device 100 may include processor(s) 102, computer-readable media(s) 104, memory controller(s) 110, memory 112, and interfaces(s) 114. In some examples, the computing device 100 may include and / or be communicatively coupled to a display 116, an infrared light source 118, and / or a sensor 120. In some examples, the computing device 100 may optionally include and / or be communicatively coupled to a camera 122 and / or a flash 124. In some examples, the computing device 100 may be included in a mobile device 150, such as a smartphone, a gaming device (e.g., a Nintendo Switch), or a tablet computer. Figure 1 As shown, in some instances, computing device 100, sensor 120, infrared light source 118, and display 116 may all be contained within a single device / housing (e.g., mobile device 150). In some instances, some or all of the components (e.g., processor 102 and sensor 120) may be contained on the same substrate. In some instances, computing device 100 may be implemented entirely or partially using a computer, server, television, or laptop computer.

[0018] Computer-readable medium 104 is accessible to processor 102. Computer-readable medium 104 may be encoded with executable instructions 108. Executable instructions 108 are executable by processor 102. In some instances, executable instructions 108 may cause processor 102 to provide commands to infrared light source 118, sensor 120, camera 122, and / or flash 124 to acquire signals and / or images of the surface. In some instances, executable instructions 108 may cause processor 102 to generate, for example, a three-dimensional (3D) model of the surface based on signals received from sensor 120. In some instances, executable instructions 108 may cause processor 102 to analyze the 3D model to calculate values ​​of surface properties. In some instances, executable instructions 108 may cause processor 102 to implement a machine learning application that includes one or more machine learning models. The machine learning application may implement various functions, such as producing results (e.g., making inferences) based at least in part on the values ​​of surface properties. Alternatively or additionally, in some instances, the machine learning application or a portion thereof may be implemented in hardware contained within the computer-readable medium 104 and / or processor 102, such as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA).

[0019] Computer-readable media 104 may store data 106. In some instances, data 106 may include signals received from sensor 120, images received from camera 122, 3D models generated by processor 102, values ​​of surface characteristics calculated by processor 102, one or more databases of surface characteristic values, and / or results generated by machine learning applications. Computer-readable media 104 may be implemented using any media containing non-transitory computer-readable media. Examples include memory, random access memory (RAM), read-only memory (ROM), volatile or non-volatile memory, hard disk drive, solid-state drive, or other storage devices. Although Figure 1 A single media may be displayed, but multiple media may be used to implement computer-readable media 104.

[0020] In some instances, processor 102 may be implemented using one or more central processing units (CPUs), graphics processing units (GPUs), ASICs, FPGAs, or other processor circuitry systems. In some instances, processor 102 may execute some or all of the executable instructions 108. In some instances, processor 102 may communicate with memory 112 via memory controller 110. In some instances, memory 112 may be volatile memory, such as dynamic random access memory (DRAM). In some instances, memory 112 may provide information to and / or receive information from processor 102 and / or computer-readable medium 104 via memory controller 110. Although a single memory 112 and a single memory controller 110 are shown, any number may be used. In some instances, memory controller 110 may be integrated with processor 102.

[0021] In some instances, interface 114 may provide a communication interface to another device (e.g., infrared light source 118), a user, and / or a network (e.g., LAN, WAN, Internet). Interface 114 may be implemented using wired and / or wireless interfaces (e.g., Wi-Fi, Bluetooth, HDMI, USB, etc.). In some instances, interface 114 may include user interface components that can receive input from a user. User input may include, but is not limited to, a desired resolution, selection of one or more regions of interest, and desired output (e.g., diagnostics, product recommendations). Examples of user interface components include a keyboard, mouse, touchpad, touchscreen, and microphone. For example, a user may view an image of a subject's face on display 116 (in some instances, this may be a touchscreen) and circle the region of interest on the image. In some instances, interface 114 may communicate information between an external device (e.g., camera 122) and one or more components of computing device 100 (e.g., processor 102 and computer-readable media 104), which may include user input, data 106, signals, images, and / or commands.

[0022] In some instances, computing device 100 may communicate with display 116 as a separate component (e.g., using a wired and / or wireless connection), or display 116 may be integrated with computing device. In some instances, display 116 may display data 106, such as outputs (e.g., results) generated by one or more machine learning models implemented by computing device 100, 3D models generated by processor 102, and / or images acquired by camera 122. In some instances, processor 102 may execute commands to implement a graphical user interface (GUI) provided on display 116. Any number or variety of displays may be present, including one or more LED, LCD, plasma, or other display devices. In some instances, display 116 may be a touchscreen, allowing the user to provide user input (e.g., selecting an area of ​​interest on a surface) via the GUI.

[0023] In some instances, the infrared light source 118 may include an infrared light-emitting diode (LED). In some instances, the infrared light source 118 may include a point projector. That is, the infrared light source 118 can project infrared light onto a surface as a plurality of discrete points. In some instances, the point projector may include diffraction optics (e.g., a diffraction grating) that generates points from the infrared light source 118. The points may have the same initial diameter or different diameters. The initial diameter means the diameter of the points at the infrared light source 118 as the distance between the infrared light source 118 and the surface increases. The points may be circular, elliptical, square, and / or other shapes. The points may be projected in various patterns (e.g., grids, concentric circles, and / or spirals). The points may be uniformly spaced or have varying densities (e.g., some areas to which the points are projected may have points that are more closely spaced than other areas). In some instances, the infrared light source 118 may project dozens, hundreds, thousands, or tens of thousands of points of infrared light. In some instances, the infrared light source 118 can project multiple patterns of dots that are then stitched together. For example, a surface can be illuminated with multiple grids of dots, and these grids can be stitched together by the computing device 100 to provide data about the entire surface. Multiple grids can be projected simultaneously or in series. For example, when the surface is larger than the size of the dot pattern that can be projected at the desired resolution, the infrared light source 118 can illuminate different portions of the surface at different times. In some instances, the overall shape of the surface or the stitching of the dot pattern can be based at least in part on changes in aberrations in the low-frequency range of spatial frequencies.

[0024] In other instances, infrared source 118 may illuminate (e.g., irradiate) a surface with infrared light in a non-discrete manner (e.g., with an unfocused or semi-focused beam). In some instances, infrared source 118 may be a coherent source. Optionally, in some instances, infrared source 118 may project polarized light onto a surface. In some instances, infrared source 118 may include polarizer 126 (e.g., polarization filter, polarization converter) to generate polarized light. In some instances, infrared source 118 may project infrared light in response to one or more commands provided by processor 102.

[0025] In some instances, sensor 120 can detect infrared light. In some instances, sensor 120 can detect visible light. In some instances, sensor 120 can detect a wide range of wavelengths (e.g., from infrared to ultraviolet wavelengths). In some instances, sensor 120 may include multiple sensors that detect light of different wavelengths (e.g., a sensor for detecting infrared light, a sensor for detecting visible light, and / or a sensor for detecting ultraviolet light). Sensor 120 may include a photoresistor, a photodiode, a phototransistor, a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) sensor, and / or other optical sensors. Sensor 120 may generate an electrical signal in response to infrared light, such as infrared light emitted from infrared light source 118 and scattered backward toward sensor 120 from its surface. Optionally, in some instances, instead of polarizing light being included in infrared light source 118, polarizer 126 may be included in sensor 120 such that the scattered light is polarized before being detected by sensor 120. In some applications, when the polarizer 126 is in the path of the sensor 120 rather than the infrared light source 118, the polarizer may be referred to as an analyzer.

[0026] In some instances, sensor 120 may be included in camera 122. Sensor 120 may generate signals in response to ambient light, infrared light from infrared light source 118, and / or flash 124 incident on sensor 120. The signals may be used to generate images, such as images of surfaces. In some instances, sensor 120 may detect light and / or provide electrical signals to computing device 100 in response to one or more commands provided by processor 102.

[0027] Optionally, flash 124 may emit light for acquiring an image via camera 122. For example, flash 124 may be used when ambient light is too low to acquire an image with an acceptable signal-to-noise ratio. Flash 124 may include a visible light source, an infrared light source, an ultraviolet light source, and / or a broadband light source (e.g., including two or more of visible, infrared, and ultraviolet light). In some instances, flash 124 may include an LED, a xenon flash tube, and / or other light sources. In some instances, flash 124 may be included within camera 122. In some instances, flash 124 may emit light for a pre-configured duration in response to commands provided by camera 122 and / or one or more commands provided by processor 102.

[0028] In some instances, executable instructions 108 may cause processor 102 to provide commands (e.g., via interface 114) to infrared light source 118 and / or sensor 120. The commands may cause infrared light source 118 to project infrared light onto a surface, and the sensor to detect a signal generated by the infrared light scattered (e.g., reflected) away from the surface. The signal may be provided from sensor 120 to processor 102 (e.g., via interface 114) and / or computer-readable medium 104. The signal generated in response to infrared light scattered and / or reflected back to sensor 120 from the surface may provide spatial information (e.g., profile) about the surface. In some instances, the signal may provide additional information about the surface (e.g., oiliness versus dryness).

[0029] Executable instructions 108 may cause processor 102 to generate a 3D model of the surface based on received signals (e.g., spatial information provided by the signals). In some instances, the 3D model may resemble a topographic map of the surface. In instances where an image of the surface is acquired, processor 102 may analyze the image to generate the 3D model. For example, processor 102 may utilize image processing techniques to detect bright and dark areas that may indicate the contours of the surface (e.g., bright areas may indicate peaks, and dark areas may indicate valleys). In some instances, processor 102 may analyze the image and signals from scattered infrared light to generate the 3D model. In some instances, signals from infrared light may be used to generate the 3D model, and an image from a camera may be superimposed on the surface of the 3D model. In some instances, the 3D model may be provided to computer-readable medium 104 for storage. In some instances, the 3D model may be provided to processor 102.

[0030] Executable instructions 108 may cause processor 102 to analyze a 3D model to calculate one or more values ​​for one or more properties of the surface (e.g., surface characteristics). Examples of surface characteristics include roughness, maximum peak height, particle size, lesion size, number of lesions, spatial frequency (e.g., what spatial frequencies are present), and volatility. In an instance where an image of the surface is acquired, processor 102 may analyze the image to calculate one or more values ​​for one or more surface characteristics.

[0031] Surface roughness can be calculated in several ways. For example, roughness can be calculated using the arithmetic mean deviation of a 3D model of the surface, as shown in Equation 1 below:

[0032] (1)

[0033] Where Ra is the surface roughness, n is the number of equally spaced sample points along the surface defined by the 3D model, and y i The average line is the vertical distance from the average line to the i-th data point. The average line of the 3D model is the average "height" of the surface represented by the 3D model. In some instances, all n are in a line (e.g., a trace) along the 3D model, and the average line is the average height of the 3D model along said line. In some instances, n are in a grid (e.g., multiple traces) along the 3D model. In these instances, there may be multiple average lines calculated for each row of the grid for the sample point n, or the average line may actually be a two-dimensional (2D) plane calculated based on all or part of the 3D model. Regardless of whether the average line is a line or a plane, for each sample point n on the surface, the vertical distance (y) between said point and the average line is measured. Each y indicates how much the height of the sample point n deviates from the average height of the surface. For example, if the sample point n of the surface is in a "valley" below the average height of the surface, then y is the vertical distance between the sample point in the valley and the average height line. Therefore, Ra provides the average of the degree to which the sample points n of the surface deviate from the average height of the surface as a measure of surface roughness. Larger Ra indicates deeper valleys and / or higher peaks compared to smaller Ra.

[0034] In another example, the root mean square (RMS) technique can be used to calculate the roughness based on a 3D model of the surface, as shown in Equation 2 below:

[0035] (2)

[0036] Where Rq is the surface roughness, and n, i and y iThis has the same meaning as in Equation 1. In Equation 2, instead of the average deviation of the height of the sample points n used as a measure of surface roughness, the square of each deviation y from the average height is taken before summing and averaging. The square root of the sum of averages is then taken to obtain Rq. Ra and Rq are provided only as examples, and other techniques for calculating surface roughness can also be used, such as the maximum valley depth (e.g., the lowest value of y for all n), the maximum peak height (e.g., the highest value of y for all n), skewness (e.g., a measure of the asymmetry of the distribution of height deviations), and / or kurtosis (e.g., a measure of outliers in the height deviations). In some instances, multiple surface roughness values ​​may be calculated, each based on a different technique.

[0037] In some instances, volatility can be calculated by comparing 3D models and / or images with standard images and / or standard samples. Examples include International Organization for Standardization (ISO) Standard 10110, ISO Standard 4287, and / or American Society of Mechanical Engineers (ASME) B46.1. In other instances, volatility can be calculated by analogy to surface roughness with widely spaced sample points.

[0038] In some instances, watershed algorithms for 3D models and / or images can be used to calculate particle size, lesion size, and / or lesion number. Watershed algorithms use various techniques to discover lines that define the top of a "ridge" and the region of a "basin" between or surrounded by ridges, based on the height of points in the 3D model and / or the brightness of pixels in the image. Examples of watershed algorithms that can be used include topological watersheds, priority flooding, and watershed cuts.

[0039] In some instances, processor 102 can analyze both the 3D model and the image to calculate values ​​for surface properties. For example, processor 102 can apply image segmentation techniques to the image to locate lesions or other areas of interest (e.g., the nose) and / or exclude certain areas from the analysis (e.g., eyebrows). Processor 102 can then analyze the 3D model based on the image analysis at lesion sites, areas of interest, and / or unexcluded areas. Examples of suitable image segmentation techniques include, but are not limited to, random forests, U-net, YOLO v103, and / or combinations thereof.

[0040] In some instances, processor 102 may preprocess the 3D model and / or image before calculating surface properties. For example, the 3D model may be flattened, scaled, and / or denoised. Similarly, the image may be processed for contrast enhancement, scaling, denoising, and / or other filtering. In some instances, processor 102 may analyze the 3D model and / or image to subtract the overall shape from the 3D model before calculating surface properties. The overall shape refers to the general shape of the surface, which may not provide meaningful information about surface properties. For example, when the surface is a personal face and the entire face or most of the face is illuminated, the overall shape of the face or a part of the face (e.g., the contour of the cheek or jaw) may not provide meaningful information about the surface properties of the skin on the personal face. Therefore, the overall shape (typically a low-frequency spatial signal) is subtracted from the 3D model before calculating surface properties.

[0041] In some instances, executable instructions 108 may cause processor 102 to implement a machine learning application that analyzes one or more values ​​of surface characteristics calculated by processor 102 and makes one or more inferences based on the one or more values ​​of surface characteristics to produce a result. In some instances, the result may be, or be contained in, a dataset associated with a 3D model and / or the calculated surface characteristics. The machine learning application may implement one or more machine learning models. Examples of machine learning models include, but are not limited to, neural networks, decision trees, and support vector machines. In some instances, the result may contain a diagnosis. For example, the machine learning application may infer that the values ​​of surface characteristics of tissue (e.g., skin) indicate that the tissue is cancerous. In some instances, the result may contain product recommendations. For example, the result may provide a recommendation for a moisturizer based at least in part on the values ​​of surface characteristics. In some instances, the result may contain both a diagnosis and a product recommendation. For example, based on one or more values ​​of surface characteristics of a subject's skin, the machine learning application may infer that the subject has psoriasis and provide a diagnosis and medication as product recommendations to treat the condition.

[0042] In some instances, instead of or supplementing machine learning applications, results can be generated by comparing one or more values ​​of a surface characteristic to a database of surface characteristic values. For example, one or more values ​​of a surface characteristic can be compared to values ​​in a database. Processor 102 can provide results associated with the closest match to a value in the database. The database of surface characteristic values ​​can be static, or it can be updated, for example, through user input (e.g., feedback). Examples of user input include product recommendations, doctor diagnoses, and reviews of new product specifications.

[0043] Optionally, in some instances, machine learning models can be used to calculate one or more values ​​of one or more surface properties, at least in part, based on 3D models and / or images.

[0044] Optionally, in some instances, computing device 100 may communicate with computing system 140. Computing system 140 may include one or more computing devices. In some instances, computing system 140 may be a cloud computing system. In these instances, computing device 100 may provide signals, images, and / or surface characteristics to computing system 140. Computing system 140 may generate 3D models, compute surface characteristics, and / or analyze surface characteristics using machine learning models. Computing system 140 may then provide the 3D models, surface characteristics, and / or results from the machine learning models back to computing device 100. In some instances, computing system 140 may contain a database of surface characteristics. This arrangement may be desirable in some applications, for example, where computing device 100 may have limited computing power, for example, if computing device 100 is contained within a compact mobile device 150. This arrangement may be more convenient when dynamically training machine learning models and / or dynamically updating the database.

[0045] Optionally, computing device 100 and / or computing system 140 may receive subsurface information. For example, subsurface information may be provided via optical coherence tomography (OCT) or elastic scattering spectroscopy (ESS). This data may be provided by optional light sources and sensors coupled to computing device 100 and / or from separate equipment that provides subsurface information to computing device 100 and / or computing system 140 for analysis. Subsurface information may be analyzed in analogous to data from the irradiated surface to discover surface characteristics of one or more layers beneath the surface. For example, subsurface information may be used to search for characteristics of one or more skin layers (e.g., stratum corneum, stratum lucidum, stratum spinosum).

[0046] Figure 2 This invention describes a technique for acquiring spatial information about a surface according to embodiments of the present disclosure. In some embodiments, light 202, such as infrared light, can be emitted from a light source 218 of the mobile device 200 onto the surface 204. In some embodiments, the light source 218 may include... Figure 1 The light source 118 is shown in the illustration. In the illustrated example, surface 204 is the skin of subject 206. In some examples, mobile device 200 may include computing device 100. In some examples, mobile device 200 may include mobile device 150.

[0047] In some instances, light 202 may be projected onto surface 204 as a point (e.g., a light spot). Two example point projection patterns 208 and 210 are shown. Point projection pattern 208 is a grid, and point projection pattern 210 is a spiral. However, other point projection patterns may be used in other instances. Although projection is referred to as a point here, it should be understood that light may be emitted to produce a point of any shape. In some instances, the light 202 may be polarized (e.g., by polarizer 126) before it is projected onto surface 204. Light 202 may be scattered (e.g., reflected) from surface 204. Scattered light 212 may be received by sensor 220. In some instances, sensor 220 may include Figure 1 The sensor 120 is shown in the image. In some instances, the scattered light 212 may be polarized by a polarizer (e.g., polarizer 126) before being received by the sensor.

[0048] In some instances, signals received from sensors can indicate the intensity of light scattered from multiple points projected onto surface 204. The varying intensities of the points (independently and / or related to each other) can provide spatial information about the surface. For example, a low intensity of scattered light at a particular point can indicate that those points are farther from the light source and / or device 200 than points associated with high intensities of scattered light. In some instances, the phase difference of the scattered light between points can provide spatial information about the surface. In some instances, the polarization state of the scattered light 212 and / or the points can provide spatial information about the surface and / or enhance the spatial information provided by the intensity and phase information.

[0049] In some instances, as illustrated in box 214, the difference in size of the dots relative to each other and / or the difference in spacing between dots reflected back to device 200 can provide spatial information about surface 204. For example, a larger dot may indicate that a portion of surface 204 is farther away than a portion of surface 204 associated with a smaller dot, because the light 202 associated with the larger dot has a greater diffusion distance. In some instances, differences in the shape of the dots can provide spatial information about surface 204 (e.g., the contour of surface 204 reflects light 202 to make the dot appear "warped" at the sensor).

[0050] In some instances, light 202 does not need to be projected as a point. More precisely, light 202 can be emitted as a loosely focused beam onto surface 204, or it can be emitted as a more tightly focused beam that scans (e.g., is projected and translated) across surface 204. Differences in intensity and / or phase at different locations on surface 204 can be analyzed, and / or other specular reflection analysis techniques can be used to extract spatial information about surface 204.

[0051] Based on spatial information obtained from the signal in response to light 202 scattered from the surface 204, the mobile device 200 can generate a 3D model 216 of the surface 204.

[0052] The distance between device 200 and surface 204 can be at least partially based on information desired by the user. For example, skin firmness is related to low-frequency errors in spatial frequencies. The less low-frequency error, the firmer the skin. To obtain data on low frequencies of surface 204, data can be obtained from a large portion of the surface (e.g., one or more cheeks, most of the forehead). Therefore, the distance between device 200 and surface 204 can be greater than the distance when analyzing a small area to allow light 202 to be projected across a larger area. In contrast, if the analysis of a lesion (e.g., a suspected birthmark) is being performed, then a smaller area can be focused on, and device 200 can be placed closer to surface 204.

[0053] Figure 3 This is an example of a machine learning model based on the examples of this disclosure. Figure 3 In the examples shown, the machine learning model includes a neural network 300. In some instances, the neural network may be implemented by a computing device 100 and / or a mobile device 200. In some instances, the neural network 300 may be a convolutional network with three-dimensional layers. The neural network 300 may include input nodes 302. In some instances, the input nodes 302 may be organized in layers of the neural network 300. The input nodes 302 may be coupled to one or more layers 308 of hidden units 306 according to weights 304. In some instances, hidden units 306 may perform operations on one or more inputs x from input nodes 302 at least partially based on associated weights 304. In some instances, hidden units 306 may be coupled to one or more layers 314 of hidden units 312 according to weights 310. Hidden units 312 may perform operations on one or more outputs from hidden units 306 at least partially based on weights 310. The output of hidden units 312 may be provided to output nodes 316 to provide a result y.

[0054] In some instances, the input x may contain one or more values ​​for one or more surface properties (e.g., roughness, volatility). In some instances, the result y may contain one or more diagnostics and / or product recommendations. In some instances, the result may be contained in a dataset associated with the 3D model and / or surface properties calculated from the 3D model.

[0055] In some instances, the neural network 300 can be trained by providing one or more training datasets. In some instances, the neural network 300 can be trained by a computing device used for inference using the neural network (e.g., computing device 100, computing system 140, and / or mobile device 200). In some instances, the neural network 300 can be trained by another computing device to determine weights and / or node arrangement or other neural network configuration information, and said weights and / or other neural network configuration information is provided to the computing device used for inference.

[0056] In some instances, supervised learning techniques can be used to train the neural network 300. In some instances, the training data may contain a set of inputs x, each input x associated with a desired outcome y (e.g., labeled). Each input x may contain one or more values ​​for one or more surface properties. For example, an input x may contain a roughness value and a value for the particle size associated with the outcome y, where the outcome y is a diagnosis of basal cell carcinoma. Based on the training dataset, the neural network 300 may adjust the number of one or more weights 304, 310, hidden units 306, 312, and / or the number of layers 308, 314 of the neural network 300. The trained neural network 300 can then be used to infer the outcome y based on the input x (which is not associated with the desired outcome).

[0057] In some instances, the neural network 300 can be trained dynamically. That is, the neural network 300 can continue to adjust the number of one or more weights 304, 310, hidden units 306, 312, and / or the number of layers 308, 314 based on new data. For example, user intervention (e.g., a doctor's input of a diagnosis of a lesion with specific surface characteristics) can cause the neural network 300 to adjust. Furthermore, in some instances, semi-supervised and / or unsupervised techniques can be used to train the machine learning model. In these instances, the dataset may not contain the desired outcome associated with each input.

[0058] Figure 3 The machine learning models shown are provided as examples only, and this disclosure is not limited to neural network 300. For example, a machine learning model may include multiple neural networks 300 and / or other machine learning models (e.g., support vector machines). In some instances, a machine learning model may include different machine learning models for different applications. For example, there may be a machine learning model for a cosmetic application (e.g., determining cosmetic recommendations) and a separate machine learning model for a medical application (e.g., determining the diagnosis of a lesion).

[0059] Figure 4 This is a flowchart of method 400 according to an example of the present disclosure. In some examples, method 400 may be performed by mobile device 150 and / or mobile device 200.

[0060] At box 402, a "surface scan" can be performed. In some instances, the scan can be performed using an infrared light source of the mobile device (e.g., infrared light source 118). The scan may involve irradiating the surface with a point, loosely focused beam, and / or a tightly focused beam. In some instances, the surface may be tissue, such as skin. However, in other instances, the surface may be inanimate, such as the surface of a diamond or a paint coating.

[0061] At block 404, the action of "receiving a signal in response to infrared light" can be performed. This signal may be generated by a sensor of the mobile device (e.g., sensor 120) in response to infrared light scattering on a surface and incident on the sensor. In some instances, the signal may be provided to a processor of the mobile device (e.g., processor 102).

[0062] At box 406, "Generate a 3D model of the surface" can be performed. The 3D model can be generated by the processor based at least in part on the signal. At box 408, "Analyze the 3D model to generate values" can be performed. In some instances, the analysis can be performed by the processor. In some instances, the values ​​can be values ​​of at least one characteristic of the surface. In some instances, the surface characteristics can include surface roughness, maximum peak height, grain size, lesion size, number of lesions, and / or volatility. In some instances, only a portion of the 3D model can be analyzed to generate the values. For example, only the region of interest indicated by user input (e.g., via interface 114) can be analyzed to generate the values. In another instance, only the region of the 3D model corresponding to a segment in an image (e.g., a lesion segmented from an image of the face) can be analyzed to generate the values.

[0063] At box 410, the "Provide Results" action can be performed. In some instances, the results may be at least partially based on the values. The results may be part of a dataset, and in some instances, they may be associated with a 3D model and / or calculated surface properties. In some instances, the results may include product recommendations and / or diagnostics. Product recommendations and / or diagnostics may be data contained in the dataset. In some instances, the results can be produced by comparing the value of at least one property of the surface with values ​​in a database. In some instances, the results can be produced by analyzing the value of at least one property of the surface using a machine learning model. In some instances, the results can be provided on a display of a mobile device (e.g., display 116).

[0064] Optionally, at box 412, "acquiring an image of the surface" can be performed. In some instances, the image can be acquired using a camera of the mobile device (e.g., camera 122). In some instances, box 412 is performed simultaneously with box 402. In some instances, box 412 is performed before box 402. In some instances, a 3D model of the surface can be at least partially based on the image. In some instances, the image can be analyzed to produce a value for at least one property of the surface and / or another value for at least one property of the surface. In some instances, the image can be acquired using a flash of the mobile device (e.g., flash 124).

[0065] Optionally, at box 414, "Polarization of infrared light" can be performed. In some instances, the infrared light may be polarized by a polarizer (e.g., polarizer 126). In some instances, box 414 is performed before box 402. In some instances, box 414 is performed after box 402 and before box 404. Optionally, at box 416, "Rendering of a 3D model" can be performed. The rendering can be displayed on a monitor. In some instances, box 416 is performed before box 410. In some instances, box 416 and box 410 are performed simultaneously.

[0066] Figure 5 This is a flowchart of method 500 according to an example of the present disclosure. In some instances, some or all of method 500 may be performed by computing device 100, mobile device 150 and / or mobile device 200.

[0067] At box 502, the action "projecting multiple infrared dots onto a surface" can be performed. In some instances, the dots can be projected by an infrared light source (e.g., 118). The dots can be projected in a grid, spiral, and / or other pattern. The dots can be uniformly or non-uniformly spaced. In some instances, the infrared light is polarized.

[0068] At block 504, the action of "detecting infrared light and generating a signal in response to the scattering of multiple infrared light spots on a surface" can be performed. In some instances, the detection and generation can be performed by a sensor (e.g., sensor 120). In some instances, the signal can be an electrical signal that can be provided to a processor (e.g., processor 102) via an interface (e.g., interface 114).

[0069] At box 506, the action "Acquire image of surface" can be performed. In some instances, an image can be acquired using a camera (e.g., camera 122). In some instances, the image can be provided to the processor via an interface.

[0070] At box 508, the action "Generate a 3D model of the surface" can be performed. The 3D model may be at least partially based on the signal. In some instances, the 3D model may be generated by a processor.

[0071] At box 510, the function "Analyze the 3D model and image to calculate multiple surface properties" can be performed. In some instances, surface properties may include at least one of surface roughness, maximum peak height, particle size, lesion size, number of lesions, or variability. In some instances, particle size, lesion size, or number of lesions is calculated at least in part based on a watershed algorithm. In some instances, the analysis may be performed by a processor. In some instances, only a portion of the 3D model may be analyzed. For example, only the region of interest indicated by user input (e.g., via interface 114) may be analyzed. In another instance, only the region of the 3D model corresponding to a segment in the image (e.g., a lesion segmented from an image of a face) may be analyzed.

[0072] At box 512, the function "Analyze multiple surface properties to produce results" can be performed. In some instances, the results may include diagnostics and / or product recommendations. In some instances, the analysis may be performed by a processor. In some instances, the processor implements a neural network to analyze surface properties. In some instances, the neural network is a convolutional neural network.

[0073] At box 514, the action "Provide results on display" can be performed. The display (e.g., display 116) may be contained within a computing device (e.g., computing device 100) and / or a mobile device (e.g., mobile device 150, mobile device 200).

[0074] Figure 6 A graphical representation 600 of an application according to an example of this disclosure. For example... Figure 6 As shown, mobile device 602 can be used to scan a surface with light (e.g., infrared light) and / or acquire images of the surface. As previously described, in some instances, scanning may involve projecting dots onto a surface. In some applications, the surface may be the skin or other tissue of a subject 604 (e.g., a patient). In some applications, the surface may be the surface of object 606, such as the paint coating on a car or the surface of a diamond. In some instances, mobile device 602 may include computing device 100. In some instances, mobile device 602 may include mobile device 150 and / or mobile device 200.

[0075] Mobile device 602 can analyze signals generated by light scattered back to mobile device 602 from a surface, and / or analyze acquired images to generate a 3D model 608 of the surface. In some instances, the 3D model 608 can be provided on a display 610 of mobile device 602. Optionally, in some instances, mobile device 602 can provide signals and / or images to a remote computing device, such as cloud computing device 618. In these instances, cloud computing device 618 can analyze signals and / or images to generate the 3D model 608 and provide the 3D model to mobile device 602 for display.

[0076] Mobile device 602 can analyze 3D model 608 and calculate values ​​for surface properties (e.g., roughness, grain size, fluctuation, maximum peak height). In some instances, mobile device 602 can calculate multiple values ​​for the same surface property and / or values ​​for multiple surface properties. In some instances, surface property values ​​612 can be provided via a GUI provided on display 610. Optionally, in some instances, mobile device 602 can provide the 3D model to another computing device (e.g., cloud computing device 618). In these instances, other computing devices (e.g., cloud computing device 618) can analyze 3D model 608 and provide surface property values ​​612 to mobile device 602.

[0077] Mobile device 602 can analyze surface characteristics 612 to produce results 615. In some instances, surface characteristics 612 are compared against a database of surface characteristics to produce results 615. In some instances, surface characteristic values ​​612 are analyzed by one or more machine learning models to produce results 615. In some instances, results 615 are provided via a GUI on the display 610 of mobile device 602. Optionally, in some instances, mobile device 602 can provide surface characteristic values ​​612 to cloud computing device 618. Cloud computing device 618 can analyze surface characteristic values ​​612 and provide results to mobile device 602.

[0078] In some instances, result 615 may include diagnosis 614 and / or product recommendation 616. In applications where the surface is the tissue of subject 604, diagnosis 614 may include an indication of a disease condition (e.g., benign vs. cancerous) or other condition (e.g., dry skin). Product recommendation 616 may include a medicine and / or treatment regimen for treating the condition indicated by diagnosis 614. In applications where the surface is the surface of object 606, diagnosis 614 may include an indication of whether the product meets or does not meet specifications (e.g., roughness is within acceptable limits). Product recommendation 616 may include product specifications (e.g., abrasive content for polishing) and / or process specifications (e.g., polishing time).

[0079] like Figure 6 As described herein, in both biological and non-biological applications, mobile devices are capable of providing information about surface properties and / or results based on surface analysis. Therefore, the devices, systems, methods, and apparatuses of this disclosure allow users to acquire and analyze data about surface properties without requiring expensive and / or specialized equipment.

[0080] Of course, it should be understood that any of the examples, embodiments, or processes described herein may be combined with or separated from one or more other examples, embodiments, and / or processes and / or performed in a separate device or device portion of a system, apparatus, or method according to the present invention.

[0081] Finally, the foregoing discussion is intended to be illustrative only and should not be construed as limiting the appended claims to any particular embodiment or group of embodiments. Therefore, while various embodiments of this disclosure have been described in particular detail, it should be understood that numerous modifications and alternative embodiments can be devised by those skilled in the art without departing from the broader and contemporaneous spirit and scope of this disclosure as set forth in the appended claims. Consequently, the specification and drawings should be viewed in an illustrative manner and are not intended to limit the scope of the appended claims.

Claims

1. An apparatus for analyzing surface texture, comprising: an infrared light source configured to project a plurality of infrared light points onto a surface; a sensor mounted within the same substrate or the same housing as the infrared light source, the sensor configured to detect a signal in response to the infrared light scattering off the surface; and a processor mounted within the same substrate or the same housing as the infrared light source and the sensor, the processor configured to: generate a three-dimensional (3D) model of the surface based at least in part on the signal detected by the sensor; calculate a general shape of the surface and subtract the general shape of the surface from the 3D model; calculate one or more characteristics of the 3D model, including at least one of a roughness, a maximum peak height, a grain size, a lesion size, a number of lesions, or a waviness of the surface, or any combination thereof, after subtracting the general shape of the surface from the 3D model; identify a data set associated with the 3D model based at least in part on at least one of the roughness, the maximum peak height, the grain size, the lesion size, the number of lesions, or the waviness; and output data from the data set via a graphical user interface (GUI) of the apparatus.

2. The apparatus of claim 1, further comprising at least one light emitting diode (LED) configured to emit light for a preconfigured duration of time in response to a user input or a command from an application on the apparatus, wherein the sensor is configured to detect an additional signal in response to light from the at least one LED scattering from the surface, wherein at least one of the roughness, the maximum peak height, the grain size, the lesion size, the number of lesions, or the waviness is calculated based at least in part on the additional signal.

3. The apparatus of claim 1, further comprising: at least one light emitting diode (LED) configured to emit light for a preconfigured duration of time in response to a user input or a command from an application on the apparatus; and a complementary metal-oxide-semiconductor (CMOS) image sensor configured to acquire an image of the surface with the at least one LED, wherein at least one of the roughness, the maximum peak height, the grain size, the lesion size, the number of lesions, or the waviness is calculated based at least in part on the image.

4. The apparatus of claim 1, further comprising a camera, wherein the camera is configured to acquire an image of the surface, wherein at least one of the roughness, the maximum peak height, the grain size, the lesion size, the number of lesions, or the waviness is calculated based at least in part on the image.

5. The apparatus of claim 1, wherein the surface roughness is calculated using an arithmetic mean deviation, a root mean square, a maximum valley depth.

6. The apparatus of claim 1, wherein at least one of the grain size, the lesion size, or the number of lesions is calculated using a watershed algorithm. ​ 7. The device of claim 1, wherein the processor implements a machine learning model to analyze the roughness, the maximum peak height, the grain size, the lesion size, the number of lesions, or the waviness in order to provide the data from the data set.

8. The device of claim 7, wherein the machine learning model is a neural network.

9. The device of claim 1, wherein the surface comprises skin cells.

10. The device of claim 1, further comprising a display, wherein the GUI provides a rendering of the 3D model, an image of the surface, a product recommendation, a diagnosis, or a combination thereof to the display, wherein the product recommendation, the diagnosis, or a combination thereof is included in the data set.

11. The device of claim 1, wherein the device is further configured to provide the at least one of the roughness, the maximum peak height, the grain size, the lesion size, the number of lesions, or the waviness of the surface to a cloud computing device for analysis, wherein the cloud computing device is configured to provide the data set associated with the 3D model to the processor.

12. The device of claim 1, further comprising at least one of a computer, a tablet, a smartphone, or a wearable computing device.

13. The device of claim 1, further comprising a polarizer configured to polarize the infrared light or an analyzer configured to polarize the signal.

14. A method for analyzing a surface texture, comprising: illuminating a surface with an infrared light source of a mobile device; receiving a signal in response to infrared light from the infrared light source that is scattered on the surface; generating a three-dimensional (3D) model of the surface based at least in part on the signal; computing a general shape of the surface; subtracting the general shape of the surface from the 3D model; after the subtraction, generating a value of at least one property of the surface from the 3D model; and providing a data set based at least in part on the value.

15. The method of claim 14, wherein the at least one property of the surface comprises at least one of a roughness, a maximum peak height, a grain size, a lesion size, a number of lesions, or a waviness of the surface.

16. The method of claim 14, further comprising comparing the value of the at least one property of the surface to values in a database to generate the data set.

17. The method of claim 14, further comprising analyzing the value of the at least one property of the surface with a machine learning model to provide the data set.

18. The method of claim 14, further comprising: acquiring an image of the surface with a camera of the mobile device; and generating the 3D model of the surface based at least in part on the image.

19. The method of claim 14, further comprising: acquiring an image of the surface with a camera of the mobile device; and ​ ​ generating the value of the at least one characteristic of the surface based at least in part on the image.

20. The method of claim 19, wherein the image is acquired with a light emitting diode.

21. The method of claim 14, further comprising polarizing the infrared light.

22. The method of claim 14, wherein the data set includes a product recommendation, a diagnosis, or a combination thereof.

23. The method of claim 14, further comprising displaying a rendering of the 3D model.

24. An apparatus for analyzing surface texture, comprising: an infrared light source configured to emit infrared light on a surface; a sensor configured to detect the infrared light scattered on the surface and generate a signal based at least in part on the detected infrared light; and a processor configured to: generate a three-dimensional (3D) model of the surface based at least in part on the signal; calculate a global shape of the surface and subtract the global shape of the surface from the 3D model prior to analyzing the 3D model; calculate a plurality of surface characteristics based at least in part on the 3D model after subtracting the global shape of the surface from the 3D model; identify a data set associated with the plurality of surface characteristics, wherein the data set includes at least one of a diagnosis or a product recommendation; and provide data from the data set via a graphical user interface, wherein the infrared light source, sensor, and processor are included on a same substrate or within a same housing.

25. The apparatus of claim 24, wherein the infrared light source is a point projector.

26. The apparatus of claim 24, further comprising a camera configured to acquire an image of the surface, and the processor is further configured to analyze the image to calculate the plurality of surface characteristics, wherein the camera comprises a CMOS image sensor, a charge-coupled device, or a combination thereof.

27. The apparatus of claim 24, wherein the infrared light source is a polarized light source.

28. The apparatus of claim 24, wherein the processor is configured to implement a machine learning model to analyze the plurality of surface characteristics to identify the data set.

29. The apparatus of claim 24, wherein the processor is configured to compare the plurality of surface characteristics to a database of surface characteristics to identify the data set.

30. The apparatus of claim 24, further comprising a display included on a same substrate or within a same housing, the display configured to display the graphical user interface, wherein the graphical user interface provides data of the data set, a rendering of the 3D model, or at least one of the plurality of surface characteristics on the display.

31. A method for analyzing surface texture, comprising: projecting a plurality of infrared light points on a surface; detecting infrared light in response to scattering of the plurality of infrared light points on the surface; generating a signal based at least in part on the detected infrared light; acquiring an image of the surface with a camera; generating a three-dimensional (3D) model of the surface based at least in part on the signals; computing a general shape of the surface based at least in part on at least one of the signals and the images; subtracting the general shape of the surface from the 3D model; computing a plurality of surface characteristics based at least in part on the 3D model after subtracting the general shape of the surface from the 3D model; identifying a data set associated with the plurality of surface characteristics, wherein the data set includes a diagnosis, a product recommendation, or a combination thereof; and providing data from the data set via a graphical user interface.

32. The method of claim 31, wherein the infrared light is polarized.

33. The method of claim 31, wherein the identifying of the data set associated with the plurality of surface characteristics is performed by a neural network.

34. The method of claim 33, wherein the neural network is a convolutional neural network.

35. The method of claim 31, wherein the plurality of surface characteristics includes at least one of a roughness, a maximum peak height, a grain size, a lesion size, a number of lesions, or a waviness of the surface.

36. The method of claim 35, wherein at least one of the grain size, the lesion size, or the number of lesions is computed based at least in part on a watershed algorithm.

37. The method of claim 31, wherein the identifying of the data set associated with the plurality of surface characteristics is performed by a cloud computing device.

Images