Method for performing skin analysis by using artificial intelligence algorithm

Abstract

The invention discloses a method for performing skin analysis by using an artificial intelligence algorithm, and relates to the technical field of optical imaging, and the method comprises the following steps: S1, selecting an equipment working mode; s2, turning on a light source; s3, acquiring an image; and S4, processing and analyzing the RCM image and the OCT image by adopting an artificial intelligence algorithm. According to the invention, the imaging capability of RCM and OCT technologies is maintained, the respective advantages of the two technologies are combined, the numerical aperture NA of the imaging objective lens is fully utilized in an RCM mode, and the NA is not fully utilized in an OCT mode, so that high resolution of submicron scale can be provided, and deeper imaging depth can be realized; only one imaging probe is used for RCM and OCT modes, and RCM light and OCT light can share at least a part of an imaging path; by means of artificial intelligence algorithm analysis, diagnosis and identification of dermatological common diseases can be more reliably achieved, and real-time, non-invasive and three-dimensional quantitative evaluation of biological tissues can also be achieved.

Description

[0001] This application is a divisional application. The application number of the original application is 202410173752.1, the application date is February 7, 2024, and the name of the invention is "A high-resolution dual-modal skin measurement and analysis device and method based on artificial intelligence." Technical Field

[0002] The present invention relates to the field of optical imaging technology, and in particular to a method for performing skin analysis using an artificial intelligence algorithm. Background Art

[0003] Imaging of biological tissues is helpful for the diagnosis of skin cancer, oral cancer, and skin or oral burns. In recent years, optical imaging techniques, such as reflectance confocal microscopy (RCM) and optical coherence tomography (OCT), have benefited skin lesions, especially noninvasive assessment.

[0004] RCM can show the details of the epidermis of biological tissues and the blood flow of microcapillaries in the epidermis of biological tissues. RCM imaging provides an optical section of about 3 microns and a lateral resolution of 0.5 to 1.0 microns, and the skin imaging depth can reach 300 microns. RCM can achieve non-invasive screening and diagnosis of cancer or burns while minimizing the need for biopsy.

[0005] OCT is an imaging method that can provide cross-sectional images that can show deep tissue morphology and can be used to visualize the epidermis and dermis in human skin. Each layer has different birefringence properties and shows differences in polarization-sensitive OCT (PS-OCT) images. PS-OCT images can help assess the integrity and tissue morphology of collagen in the dermis. OCT images can help determine the deep spread of cancer or the depth of burn injuries. OCT imaging depth is at least 1.5 mm, but OCT has a coarse scale resolution, with an optical sectioning resolution of 5 to 10 microns, which generally does not provide enough resolution to resolve subcellular details and diagnose early cancers.

[0006] In summary, RCM can provide submicron resolution, which enables the identification of morphological changes in the skin, but its disadvantage is that the penetration depth is limited. It is difficult to assess the spread depth of cancer or burns using only RCM, while the benefit of OCT imaging of the same lesion is that its invasion depth can be better distinguished. RCM and OCT complement each other and can help improve the diagnosis of BCC when used with the same instrument.

[0007] At the same time, with the help of artificial intelligence algorithm analysis, many other functions can also be achieved, such as the diagnosis and differentiation of common dermatological diseases: such as differentiation of seborrheic keratosis, flat warts and syringoma; differentiation of benign and malignant skin tumors: such as pigmented nevus and melanoma; definition of skin lesion boundaries: guiding surgical excision of skin lesions; monitoring the occurrence and development of the disease: such as monitoring changes in skin microcirculation during the development of psoriasis; detecting treatment effects: such as monitoring pigment production during the treatment of vitiligo; monitoring of skin physiological state: such as measuring the thickness of the skin epidermis and stratum corneum, detecting hair growth, etc.; monitoring of drug absorption: monitoring the distribution of topical drugs in the skin, etc.

[0008] Therefore, how to provide a method for skin analysis using an artificial intelligence algorithm is an urgent problem that needs to be solved by those skilled in the art. Summary of the invention

[0009] In view of this, the present invention provides a method for skin analysis using an artificial intelligence algorithm, which helps to solve the problems existing in the prior art.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A high-resolution dual-modality skin measurement and analysis device based on artificial intelligence, comprising an imaging probe, an RCM imaging device, an OCT imaging device and a system control and data processing device; the imaging probe is optically connected to the RCM imaging device and the OCT imaging device respectively, and the RCM imaging device and the OCT imaging device are electrically connected to the system control and data processing device respectively;

[0012] The imaging probe is used to detect the sample to be tested;

[0013] The RCM imaging device is used to provide a first incident light beam to the imaging probe, and receive a first return light beam returned from the imaging probe to perform RCM imaging;

[0014] The OCT imaging device is used to provide a second incident light beam to the imaging probe and receive a second return light beam returned from the imaging probe to perform OCT imaging;

[0015] The system control and data processing device is used for receiving, analyzing, processing and displaying RCM images and OCT images.

[0016] In the above device, optionally, in the imaging probe, a sample to be measured, a first optical element, and a third optical element are sequentially arranged from top to bottom; the third optical element is arranged in parallel with the second optical element;

[0017] The first optical element is used to adjust the focus of the first incident light beam or the second incident light beam, direct the first incident light beam or the second incident light beam toward the sample to be measured, and return the first return light beam or the second return light beam to the first surface of the third optical element;

[0018] The first surface of the third optical element faces the sample to be measured, and is used to reflect the first incident light beam generated by the RCM imaging device; the second surface of the third optical element faces the OCT imaging device, and is used to allow the second incident light beam generated by the OCT imaging device to pass through the third optical element;

[0019] The first surface of the third optical element is also used to reflect the first return light beam returned from the sample to be tested, and guide the first return light beam returned from the sample to be tested to the first surface of the second optical element; the first surface of the third optical element is also used to allow the second return light beam returned from the sample to be tested to pass through, and guide the second return light beam to the OCT imaging device;

[0020] The first surface of the second optical element is used to guide the first incident light beam generated by the RCM imaging device to the first surface of the third optical element; the first surface of the second optical element is also used to guide the first return light beam returned from the first surface of the third optical element to the RCM imaging device.

[0021] In the above device, optionally, the RCM imaging device is provided with an RCM light source, a collimating lens, a first polarizer, a beam splitter and a first optical scanner in order from bottom to top; a first lens, a second lens and a second optical scanner are provided in order on one side of the first optical scanner; a third lens and a fourth lens are provided in order above the second optical scanner; a second polarizer, a fifth lens and a detector are provided in order on a side of the beam splitter corresponding to the first lens;

[0022] The RCM light source is used to emit a first incident light beam to the collimating lens;

[0023] The collimating lens is used for collimating the first incident light beam and directing the collimated light beam to the first polarizer;

[0024] A first polarizer is used to direct a linear polarization component of the first incident light beam to the beam splitter;

[0025] A beam splitter is used to split the first incident light beam so that a portion of the light beam is directed to the first optical scanner;

[0026] The first optical scanner is used to scan the first incident light beam, and guide the first incident light beam to the second optical scanner through a path including the first lens and the second lens;

[0027] The first lens and the second lens are used to optically couple the first optical scanner with the second optical scanner;

[0028] The second optical scanner is used to scan the first incident light beam into a direction perpendicular to the linear scanning; and guide the first incident light beam to the imaging probe through a path including a third lens and a fourth lens;

[0029] The third lens and the fourth lens are used to couple the second optical scanner with the second optical element in the imaging probe;

[0030] The second optical scanner is also used to receive the first return light beam passing through the fourth lens and the third lens, and to descan the first return light beam, and to guide the first return light beam to the first optical scanner through the second lens and the first lens;

[0031] The first optical scanner is also used to descan the first return beam and direct the first return beam to the beam splitter;

[0032] The beam splitter is further configured to direct at least a portion of the first return beam to a second polarizer;

[0033] The second polarizer is used to pass light of a certain polarization state through the fifth lens;

[0034] The fifth lens is used to propagate the first return light beam to the detector;

[0035] The detector is used to output first electrical signals corresponding to the return light at multiple positions on the detector, representing the RCM image of the sample to be tested. The first electrical signal is transmitted to the first frame grabber of the system control and data processing device through the first electrical connection.

[0036] The above device, optionally, the OCT imaging device is provided with an OCT light source, a fiber optic interferometer, a dual spectrometer and an OCT scanner;

[0037] The OCT light source is used to generate a second incident light beam;

[0038] The fiber optic interferometer is used to receive and guide the second incident light beam;

[0039] The OCT scanner is used to direct a second incident light beam to a second surface of a third optical element in the imaging probe;

[0040] The dual spectrometer is used to receive a second return light beam in an orthogonal polarization state from the sample to be tested, and output an OCT image indicating the sample to be tested, which is recorded as a second electrical signal. The second electrical signal is transmitted to a real-time digital signal processing board of the system control and data processing device through a second electrical connection.

[0041] In the above device, optionally, the fiber optic interferometer includes an illumination optical path, a sample optical path, a reference optical path, and a detection optical path;

[0042] The illumination optical path includes a first waveguide, a phase modulator, a first polarization controller, a second waveguide and a 10 / 90 splitter connected in sequence;

[0043] The sample optical path includes a third waveguide, a first circulator and a fourth waveguide connected in sequence; the first circulator and the 10 / 90 splitter are connected via the third waveguide;

[0044] The reference optical path includes a fifth waveguide, a second circulator, a sixth waveguide, a second polarization controller, a sixth lens and a reflector connected in sequence; the second circulator is connected to the 10 / 90 splitter through the fifth waveguide; the reflector is located on the translation stage;

[0045] The detection optical path includes a seventh waveguide, a 50 / 50 polarization-sensitive optical fiber combiner, and an eighth waveguide connected in sequence; the first circulator is connected to the 50 / 50 polarization-sensitive optical fiber combiner through the seventh waveguide, and the second circulator is connected to the 50 / 50 polarization-sensitive optical fiber combiner through the eighth waveguide; the detection optical path also includes a ninth waveguide and a tenth waveguide; the ninth waveguide and the tenth waveguide are connected in parallel to the 50 / 50 polarization-sensitive optical fiber combiner and the dual spectrometer.

[0046] The above device, optionally, the system control and data processing device includes a real-time digital signal processing board and a computer in communication connection and a computer and a display in communication connection;

[0047] The computer includes a first frame grabber, a second frame grabber and a data acquisition board which are arranged in sequence;

[0048] The first frame grabber is used to receive the first electrical signal output by the RCM imaging device and digitally generate an RCM image of the sample to be tested;

[0049] The real-time digital signal processing board is used to receive and process the second electrical signal from the OCT imaging device, and output it to the second frame grabber through an electrical connection;

[0050] The second frame grabber is used to digitize the second electrical signal to generate an OCT image of the sample to be tested.

[0051] Optionally, the above-mentioned device has an artificial intelligence algorithm integrated into the system control and data processing device for further processing and analyzing the RCM images and OCT images, including denoising, segmentation, and 3D segmentation.

[0052] A high-resolution dual-modality skin measurement and analysis method based on artificial intelligence, applied to any of the above-mentioned high-resolution dual-modality skin measurement and analysis devices based on artificial intelligence, comprises the following steps:

[0053] S1. Select the device working mode;

[0054] S2, turn on the light source;

[0055] S3, collecting images;

[0056] S4. Artificial intelligence analyzes images.

[0057] In the above method, optionally, the device working modes include using only RCM, using only OCT, and using both RCM and OCT.

[0058] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a high-resolution dual-modality skin measurement and analysis device and method based on artificial intelligence, which has the following beneficial effects: the present invention maintains the imaging capabilities of RCM and OCT technologies, combines the respective advantages of the two technologies, and fully utilizes the numerical aperture NA of the imaging objective in the RCM mode, while underutilizing the NA in the OCT mode, thereby providing both submicron-scale high resolution and deeper imaging depth; only one imaging probe is used for the RCM and OCT modes; RCM light and OCT light can share at least a portion of the imaging path; the imaging objective can be imaged at two different wavelengths (for example, at 830nm and 1310nm); RCM and OCT images can be displayed simultaneously in real time; with the help of artificial intelligence algorithm analysis, the diagnosis and identification of common dermatological diseases can be more reliably achieved, and real-time, non-invasive and three-dimensional quantitative evaluation of biological tissues can also be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 This is a structural diagram of a high-resolution dual-modality skin measurement and analysis device based on artificial intelligence disclosed in the present invention;

[0061] Figure 2 This is a flow chart of a high-resolution dual-modality skin measurement and analysis method based on artificial intelligence disclosed in the present invention;

[0062] Figure 3 It is a DeblurGAN network framework diagram disclosed in an embodiment of the present invention;

[0063] Figure 4 It is a structural diagram of the Segnet deep neural network disclosed in an embodiment of the present invention;

[0064] Figure 5 is a schematic diagram of pooling index and upsampling disclosed in an embodiment of the present invention;

[0065] Figure 6It is a network structure diagram based on the improved 3D U-Net disclosed in an embodiment of the present invention;

[0066] Figure 7 is a diagram of the structure of a four-level 3D U-Net of Net1 disclosed in an embodiment of the present invention;

[0067] Figure 8 is a network structure diagram of Net2 disclosed in an embodiment of the present invention;

[0068] Fig. 9 is a residual structure diagram of the encoding and decoding path disclosed in an embodiment of the present invention;

[0069] Fig.10 is a bottleneck structure diagram disclosed in an embodiment of the present invention;

[0070] Among them, 1-sample to be tested, 2-first optical element, 3-second optical element, 4-third optical element, 3A-first surface of the second optical element, 4A-first surface of the third optical element, 4B-second surface of the third optical element, 8-fourth lens, 9-third lens, 10-second optical scanner, 11-second lens, 12-first lens, 13-first optical scanner, 14-beam splitter, 15-first polarizer, 16-collimating lens, 17-RCM light source, 18-second polarizer, 19-fifth lens, 20-detector, 21-first electrical connection, 22-OCT light source, 23-phase modulator, 24-first polarization controller, 25-10 / 90 splitter, 26-first circulator, 27-50 / 50 polarization sensitive fiber combiner, 28-second circulator, 29-second polarization controller, 30-translation stage, 31-sixth lens, 32-reflector, 33-dual spectrometer, 34-OCT scanner, 35-second electrical connection, 36-computer, 37-display, 38-real-time digital signal processing board, 39-first frame grabber, 40-second frame grabber, 41-data acquisition board, 42-RCM imaging device, 43-OCT imaging device, 44-imaging probe, 45-fifth waveguide, 46-third waveguide, 47-seventh waveguide, 48-eighth waveguide, 49-fourth waveguide, 50-sixth waveguide, 51-ninth waveguide, 52-tenth waveguide, 53-first waveguide, 54-second waveguide, 55-third electrical connection, 56-system control and data processing device, 57-fiber interferometer. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0073] Reference Figure 1 As shown, the present invention discloses a high-resolution dual-modality skin measurement and analysis device based on artificial intelligence, comprising an imaging probe 44, an RCM imaging device 42, an OCT imaging device 43 and a system control and data processing device 56; the imaging probe 44 is optically connected to the RCM imaging device 42 and the OCT imaging device 43 respectively, and the RCM imaging device 42 and the OCT imaging device 43 are electrically connected to the system control and data processing device 56 respectively;

[0074] The imaging probe 44 is used to detect the sample 1 to be tested;

[0075] The RCM imaging device 42 is used to provide a first incident light beam to the imaging probe 44 and receive a first return light beam returned from the imaging probe 44 to perform RCM imaging;

[0076] The OCT imaging device 43 is used to provide a second incident light beam to the imaging probe 44 and receive a second return light beam returned from the imaging probe 44 to perform OCT imaging;

[0077] The system control and data processing device 56 is used for data reception, analysis, processing and display.

[0078] Furthermore, in the imaging probe 44, a sample to be tested 1, a first optical element 2, and a third optical element 4 are sequentially arranged from top to bottom; the third optical element 4 is arranged parallel to the second optical element 3;

[0079] The first optical element 2 is used to adjust the focus of the first incident light beam or the second incident light beam, direct the first incident light beam or the second incident light beam toward the sample 1 to be tested, and return the first return light beam or the second return light beam to the first surface 4A of the third optical element;

[0080] The third optical element 4 is a dichroic mirror, and a first surface 4A of the third optical element faces the sample 1 to be measured, and is used to reflect the first incident light beam generated by the RCM imaging device 42; a second surface 4B of the third optical element faces the OCT imaging device 43, and is used to make the second incident light beam generated by the OCT imaging device 43 share at least a part of the imaging path with the first incident light beam through the third optical element;

[0081] The first surface 4A of the third optical element is also used to reflect the first return light beam returned from the sample 1 to be tested, and guide the first return light beam returned from the sample 1 to be tested to the first surface 3A of the second optical element; the first surface 4A of the third optical element is also used to allow the second return light beam returned from the sample 1 to pass through, and guide the second return light beam to the OCT imaging device 43;

[0082] The second optical element 3 is a dichroic mirror, and the first surface 3A of the second optical element is used to guide the first incident light beam generated by the RCM imaging device 42 to the first surface 4A of the third optical element; the first surface 3A of the second optical element is also used to guide the first return light beam returned from the first surface 4A of the third optical element to the RCM imaging device 42.

[0083] Specifically, the RCM imaging device 42 provides a first beam of light to the imaging probe 44 to perform RCM imaging. The first beam of light is irradiated onto the first surface 3A of the second optical element. The second optical element 3 reflects the first beam of light onto the first surface 4A of the third optical element. The third optical element 4 reflects the first beam of light to the first optical element 2, which adjusts the focus of the incident RCM light. The first beam of light passes through the first optical element 2 and is emitted to the sample 1 to be imaged (for example, biological tissue). The RCM imaging device 42 scans the first beam of light along the sample 1 to be imaged through the second optical element 3 and the third optical element 4. The sample 1 to be imaged reflects, absorbs, and backscatters the first beam of light. Part of the reflected and backscattered light returns to the RCM imaging device 42. The first return light passes through the first optical element 2 and is emitted to the first surface 4A of the third optical element. The third optical element 4 reflects the first return light onto the first surface 3A of the second optical element. The second optical element 3 reflects the first return light to the RCM imaging device 42. The RCM imaging device 42 descans the first return light and detects the image of the sample 1 to be imaged. The RCM imaging device 42 outputs a first signal indicating the RCM image through the first electrical connection 21 for the system control and data processing device 56 to perform data acquisition, processing and display.

[0084] Further, the RCM imaging device 42 is provided with an RCM light source 17, a collimating lens 16, a first polarizer 15, a beam splitter 14 and a first optical scanner 13 in sequence from bottom to top; a first lens 12, a second lens 11 and a second optical scanner 10 are provided in sequence on one side of the first optical scanner 13; a third lens 9 and a fourth lens 8 are provided in sequence above the second optical scanner 10; a second polarizer 18, a fifth lens 19 and a detector 20 are provided in sequence on one side of the beam splitter 14;

[0085] The RCM light source 17 is used to emit a first incident light beam to the collimating lens 16;

[0086] The collimating lens 16 is used to collimate the first incident light beam and guide it to the first polarizer 15;

[0087] The first polarizer 15 is used to direct the linear polarization component of the first incident light beam to the beam splitter 14;

[0088] The beam splitter 14 is used to split the first incident light beam so that at least a part of the light beam is directed to the first optical scanner 13;

[0089] The first optical scanner 13 is used to scan the first incident light beam and guide the first incident light beam to the second optical scanner 10 through a path including the first lens 12 and the second lens 11;

[0090] The first lens 12 and the second lens 11 are used to optically couple the first optical scanner 13 with the second optical scanner 10;

[0091] The second optical scanner 10 is used to scan the first incident light beam into a direction perpendicular to the linear scanning; the first incident light beam is guided to the imaging probe 44 through a path including a third lens 9 and a fourth lens 8;

[0092] The third lens 9 and the fourth lens 8 are used to couple the second optical scanner 10 with the second optical element 3 in the imaging probe 44;

[0093] The second optical scanner 10 is also used to receive the first return light beam passing through the fourth lens 8 and the third lens 9, and to descan the first return light beam, and to guide the first return light beam to the first optical scanner 13 through the second lens 11 and the first lens 12;

[0094] The first optical scanner 13 is also used to descan the first return beam and guide the first return beam to the beam splitter 14;

[0095] The beam splitter 14 is also used to direct at least a portion of the first return beam to the second polarizer 18;

[0096] The second polarizer 18 is used to pass light of a certain polarization state through the fifth lens 19;

[0097] The fifth lens 19 is used to propagate the first return light beam to the detector 20;

[0098] The detector 20 is used to output a first electrical signal corresponding to the return light at multiple positions on the detector 20, representing the RCM image of the sample 1 to be tested. The first electrical signal is transmitted to the first frame grabber 39 of the system control and data processing device 56 through the first electrical connection 21.

[0099] Further, the OCT imaging device 43 is provided with an OCT light source 22, a fiber interferometer 57, a dual spectrometer 33 and an OCT scanner 34;

[0100] The OCT light source 22 is used to generate a second incident light beam;

[0101] The fiber optic interferometer 57 is used to receive and guide the second incident light beam;

[0102] The OCT scanner 34 is used to direct the second incident light beam to the second surface 4B of the third optical element in the imaging probe 44;

[0103] The dual spectrometer 33 is used to receive a second return light beam in an orthogonal polarization state from the sample to be tested 1, and output an OCT image indicating the sample to be tested 1, which is recorded as a second electrical signal. The second electrical signal is transmitted to the real-time digital signal processing board 38 of the system control and data processing device 56 through the second electrical connection 35.

[0104] Specifically, the OCT imaging device 43 provides a second incident light beam to the imaging probe 44. The second incident light beam is irradiated onto the second surface 4B of the third optical element, and passes through the first optical element 2 through the third optical element 4. The second incident light beam propagates toward the sample 1 to be tested. The second incident light beam hits the sample 1 to be tested. The second incident light beam may share part of the imaging path with the first incident light beam. The sample 1 to be tested reflects, absorbs, and backscatters the second incident light beam. Part of the reflected and backscattered light returns to the OCT imaging device 43. The second return light passes through the first optical element 2 to the first surface 4A of the third optical element. The second return light passes through the third optical element 4. The second return light re-enters the OCT imaging device 43. The OCT imaging device 43 detects the image of the sample 1 to be tested. The OCT imaging device 43 outputs a second signal indicating the OCT image to the system control and data processing device 56 through the second electrical connection 35 for data acquisition, processing, and display. The OCT light may share part of the imaging path (including the first optical element 2 and the third optical element 4) with the RCM light. In some embodiments, the second optical element 3 is a mirror. In some embodiments, the third optical element 4 is a dichroic mirror. In some embodiments, the first optical element 2 is an imaging objective lens that adjusts the focus of incident light thereon. In some embodiments, the first optical element 2 is a set of lenses. In some embodiments, the RCM imaging device 42 is a standard confocal microscope that operates in a reflection mode. In some embodiments, the OCT imaging device 43 is a PS-OCT imaging device. In some embodiments, the OCT imaging device 43 is a standard spectrometer or an OCT instrument based on a swept source.

[0105] Further, the fiber optic interferometer 57 includes an illumination optical path, a sample optical path, a reference optical path, and a detection optical path;

[0106] The illumination optical path includes a first waveguide 53, a phase modulator 23, a first polarization controller 24, a second waveguide 54 and a 10 / 90 splitter 25 connected in sequence;

[0107] The sample optical path includes a third waveguide 46, a first circulator 26 and a fourth waveguide 49 connected in sequence; the first circulator 26 and the 10 / 90 splitter 25 are connected via the third waveguide 46;

[0108] The reference optical path includes a fifth waveguide 45, a second circulator 28, a sixth waveguide 50, a second polarization controller 29, a sixth lens 31 and a reflector 32 connected in sequence; the second circulator 28 and the 10 / 90 splitter 25 are connected via the fifth waveguide 45; the reflector 32 is located on the translation stage 30;

[0109] The detection optical path includes a seventh waveguide 47, a 50 / 50 polarization-sensitive optical fiber combiner 27, and an eighth waveguide 48 connected in sequence; the first circulator 26 is connected to the 50 / 50 polarization-sensitive optical fiber combiner 27 through the seventh waveguide 47, and the second circulator 28 is connected to the 50 / 50 polarization-sensitive optical fiber combiner 27 through the eighth waveguide 48; the detection optical path also includes a ninth waveguide 51 and a tenth waveguide 52; the ninth waveguide 51 and the tenth waveguide 52 are connected in parallel to the 50 / 50 polarization-sensitive optical fiber combiner 27 and the dual spectrometer 33.

[0110] Specifically, in some embodiments, the imaging probe 44 avoids using a dichroic mirror for reflection in the OCT mode, because the dichroic mirror may cause severe scattering of light and reduce the quality of the OCT image. In some embodiments, the first optical element 2 further includes a coating that allows a light beam with a wavelength between about 800 nanometers and 1400 nanometers to pass through. In some embodiments, the numerical aperture (NA) of the first optical element 2 is fully utilized in the RCM mode to achieve high-resolution imaging. In some embodiments, the numerical aperture of the first optical element 2 is not fully utilized in the OCT mode to achieve an imaging depth greater than about one millimeter. In some embodiments, the numerical aperture of the first optical element 2 is approximately 0.8-1.0.

[0111] Specifically, the RCM imaging device 42 includes an RCM light source 17, which provides a beam of light to a collimating lens 16. The collimating lens 16 collimates the light beam and guides it to a first polarizer 15. The first polarizer 15 guides the linear polarization component of the light beam to a beam splitter 14. The beam splitter 14 splits the light beam so that at least a portion of the light beam is directed to a first optical scanner 13. The first optical scanner 13 scans the light beam (for example, generates a high-speed line scan). The first optical scanner 13 guides the light beam to a second optical scanner 10 through a path including a first lens 12 and a second lens 11. The first lens 12 and the second lens 11 optically couple the first optical scanner 13 with the second optical scanner 10. The second optical scanner 10 scans the light beam in a direction perpendicular to the linear scan (for example, generates a low-speed raster scan). The second optical scanner 10 guides the light beam to an imaging probe 44 through a path including a third lens 9 and a fourth lens 8, and the third lens 9 and the fourth lens 8 optically couple the second optical scanner 10 with a second optical element 3 in the imaging probe 44.

[0112] Specifically, the light returned from the imaging probe 44 propagates toward the RCM imaging device 42. The returned light propagates to the second optical scanner 10 through the fourth lens 8 and the third lens 9. The second optical scanner 10 performs descanning on the returned light. The second optical scanner 10 directs the returned light to the first optical scanner 13 through the second lens 11 and the first lens 12. The first optical scanner 13 performs descanning on the returned light. The first optical scanner 13 directs the returned light to the beam splitter 14. The beam splitter 14 directs at least a portion of the returned light to the second polarizer 18, and the second polarizer 18 allows light of a certain polarization state to pass through the fifth lens 19. The returned light propagates through the fifth lens 19 and is incident on the detector 20. The detector 20 outputs a first electrical signal corresponding to the returned light at multiple positions on the detector 20. The first electrical signal can represent an RCM image of the sample. The first electrical signal is transmitted to the system control and data processing device 56 through the first electrical connection 21.

[0113] In some embodiments, the RCM light source 17 is an 830 nm laser source. In some embodiments, the first optical scanner 13 is a line scanner. In some embodiments, the second optical scanner 10 is a low speed raster scanner. In some embodiments, the detector 20 is an avalanche photodetector.

[0114] Specifically, the OCT light source 22 guides a beam of light to the phase modulator 23 through the first waveguide 53. The phase modulator 23 modulates the phase of the light beam and guides the light beam to the first polarization controller 24. The first polarization controller 24 adjusts the polarization state of the light beam and guides the light beam to the 10 / 90 splitter 25 through the second waveguide 54. The 10 / 90 splitter 25 splits the light beam and guides a portion of the light beam to the sample light path and another portion of the light beam to the reference light path. The light entering the sample light path is transmitted through the third waveguide 46 and the first circulator 26. The light is guided from the first circulator 26 to the OCT scanner 34 through the fourth waveguide 49. The OCT scanner 34 guides the light to the imaging probe 44. The light returned from the imaging probe 44 is guided to the first circulator 26 through the fourth waveguide 49. The first circulator 26 guides the returned light to the 50 / 50 polarization-sensitive fiber combiner 27 through the seventh waveguide 47. The light entering the reference light path is guided to the second circulator 28 through the fifth waveguide 45. The second circulator 28 guides the light to the second polarization controller 29 through the sixth waveguide 50. The second polarization controller 29 polarizes the light and guides the light to the sixth lens 31. The light is projected to the reflector 32 through the sixth lens 31. The reflector 32 is located on the translation stage 30. The translation stage 30 can adjust the position of the reflector 32 so that the length of the reference optical path matches the length of the sample optical path. A portion of the light guided to the reference optical path is reflected by the reflector 32. The reflected light passes through the sixth lens 31 and the second polarization controller 29. The second polarization controller 29 adjusts the polarization state of the light to match the polarization state of the light from the sample optical path. The light passes through the sixth waveguide 50 and the second circulator 28. The second circulator 28 guides the light to the 50 / 50 polarization-sensitive fiber combiner 27 through the eighth waveguide 48. The light entering the detection optical path is guided to the 50 / 50 polarization-sensitive fiber combiner 27 through the seventh waveguide 47 and the eighth waveguide 48. The light is guided to the dual spectrometer 33 through the ninth waveguide 51 and the tenth waveguide 52. Each spectrometer of the dual spectrometer 33 receives light of orthogonal polarization states from the sample in order to monitor changes in the polarization state of the light caused by the sample. The dual spectrometer 33 provides a signal indicative of an OCT image of the sample. The signal is transmitted to the system control and data processing device 56 via the second electrical connection 35.

[0115] Specifically, the dual spectrometer 33 may include a diffraction grating, a spectrometer lens system, or a digital camera. In some embodiments, the digital camera may be an InGaAs camera. In some embodiments, the dual spectrometer 33 includes two InGaAs cameras. In some embodiments, each InGaAs camera has 1024 elements and an effective area of ​​25 mm. The InGaAs camera can be used to collect light in two orthogonal polarization states from the 50 / 50 polarization-sensitive fiber combiner 27. The dual spectrometer 33 can use a specially designed diffraction grating and lens system to accommodate a spectral bandwidth of more than 120 nanometers.

[0116] In some embodiments, the dual spectrometer 33 can be replaced with a balanced detection scheme. In some embodiments, the OCT light source 22 can be replaced with a wavelength scanning light source (e.g., a swept frequency source). In some embodiments, the wavelength scanning light source can be a block optical polarization combiner. In some embodiments, all fiber optic components can use polarization-maintaining fibers. In some embodiments, the phase modulator 23 can be replaced with an in-line fiber polarizer.

[0117] In some embodiments, the OCT imaging device 43 is a PS-OCT imaging device. In some embodiments, the OCT light source 22 is a broadband superluminescent diode. The OCT light source 22 can have a central wavelength of about 1310 nanometers and a bandwidth of about 135 nanometers. In some embodiments, the axial resolution of a broadband source (e.g., a broadband superluminescent diode) can be determined as follows:

[0118]

[0119] Where λ0 is the central wavelength of the light source, Δλ is the spectral width of the light source, and n is the refractive index of the sample. For biological tissue, n is approximately 1.34. For air, n is approximately 1.00. In some embodiments, when λ0 is equal to 1310 nanometers and Δλ is equal to 135 nanometers, the axial resolution of the OCT light source 22 in air is l z The axial resolution in biological tissue is about 5.6 microns. z About 4.2 microns.

[0120] Further, the system control and data processing device 56 includes a real-time digital signal processing board 38 and a computer 36 in communication connection and a computer 36 and a display 37 in communication connection;

[0121] The computer 36 includes a first frame grabber 39, a second frame grabber 40 and a data acquisition board 41 which are arranged in sequence;

[0122] The first frame grabber 39 is used to receive the first electrical signal output by the RCM imaging device 42, and digitally generate an RCM image of the sample 1 to be tested;

[0123] The real-time digital signal processing board 38 is used to receive and process the second electrical signal from the OCT imaging device 43 and output it to the second frame grabber 40 through the third electrical connection 55;

[0124] The second frame grabber 40 is used to digitize the second electrical signal to generate an OCT image of the sample 1 to be tested.

[0125] Specifically, the first frame grabber 39 communicates with the RCM imaging device 42. The first frame grabber 39 receives imaging output from the RCM imaging device 42 via the first electrical connection 21, so that the system control and data processing device 56 can generate and display the RCM image. The second frame grabber 40 communicates with the real-time digital signal processing board 38 (RT-DSP board). The real-time digital signal processing board 38 (RT-DSP board) receives output from the OCT imaging device 43 via the second electrical connection 35. The real-time digital signal processing board 38 provides the OCT imaging output to the second frame grabber 40 via the third electrical connection 55. The first frame grabber 39 and the second frame grabber 40 each digitize the output they receive. The first frame grabber 39 generates an RCM image of the sample 1 to be tested. The second frame grabber 40 generates an OCT image of the sample 1 to be tested. Each image is provided to the display 37 connected to the computer 36.

[0126] In some embodiments, the real-time digital signal processing board 38 (RT-DSP board) is based on field programmable gate array (FPGA) hardware and can perform OCT data processing and display at a high frame rate. In some embodiments, the real-time digital signal processing board 38 (RT-DSP board) operates as a standalone device and communicates with the computer 36 via a PCI bus. In some embodiments, the real-time digital signal processing board 38 (RT-DSP board) is replaced by a commercially available graphics processing unit (GPU).

[0127] Furthermore, artificial intelligence algorithms are integrated into the system control and data processing device 56 for further processing and analyzing RCM images and OCT images, including denoising, segmentation, and 3D segmentation.

[0128] Specifically, (1) denoising

[0129] The image denoising algorithm based on deep learning DeblurGAN has a network architecture consisting of two parts: the generator and the discriminator. The loss function value of the target image is reduced by continuously generating confrontation between the generator and the discriminator, thereby restoring a clear image. As a denoising deep learning algorithm, the features extracted by the generator of the DeblurGAN network include the noise features of the image. From the following formula, we can see that based on this formula of DeblurGAN, a clear image is restored by simulating the distribution of degraded pixels.

[0130] I B =k(M)*I S +A

[0131] Among them I B represents the blurred image, k(M) represents the unknown blur kernel determined by M, and I SRepresents the latent image, A represents additive noise. From the above formula, we can see that DeblurGAN can restore a clear image and also has the function of eliminating additive noise.

[0132] During the training phase, a discriminator needs to be defined. The architecture of the discriminator network is the same as that of PatchGAN. Except for the last convolutional layer, the rest of the instance regularization and LeakyReLU are all α=0.2. The overall framework is as follows: Figure 3 .

[0133] DeblurGAN is a conditional generative adversarial network that generates noise images. B As input, it is restored to a clear image I S , the CNN network trained with denoising is used as the generator At the same time, the discriminator Construct a generative adversarial network and generate adversarial loss. In DeblurGAN, the loss function is defined as the weighted sum of multiple loss functions, expressed as:

[0134] L=L GAN +λ·L X

[0135] Where λ = 100, because there is no need to penalize the mismatch between input and output, so L GAN Some of them use WGAN-GP as the loss function, which has been proven to be stable for training on a variety of GAN structures and requires almost no hyperparameter adjustment. The specific expression is as follows:

[0136]

[0137] The other part of the loss function L X Defined as content loss, its two typical choices are L1 (MAE) loss or L2 (MSE) loss defined on the original image. Using these functions as the only optimization target may lead to the generated image texture being too smooth due to pixel-level averaging problems in pixel space. Therefore, a perceptual loss method is proposed on this basis. It is defined based on the difference in CNN feature maps between the generated and target images, and its specific expression is as follows:

[0138]

[0139] in is the feature map obtained by the jth convolution (after activation) before the i-th max pooling layer in the VGG19 network, W i,j and H i,jis the dimension of the feature map. The activation function of the VGG convolutional layer is used here. The deeper activation function represents the more abstract features. The perceptual loss function focuses on restoring the generated content, while the adversarial loss focuses on restoring the texture details.

[0140] (2) Division

[0141] The Segnet semantic segmentation network adopts an encoder and decoder structure (Encoder-Decoder) as a whole, and is mainly used for pixel-level image segmentation. The Segnet network structure has a total of 62 layers, excluding the final activation layer and classifier. In the encoder part, the feature extraction of the data is achieved through the combination of multiple convolutional layers and pooling layers. In the decoder part, Segnet upsamples the feature map extracted in the encoder to increase the data size, and uses the position information of the maximum value pooling in the corresponding encoder when upsampling. This method retains the important detail features in the original image and discards unimportant features and noise. Without any post-processing algorithm, Segnet automatically achieves edge smoothing of the result image. The Segnet network is mainly composed of three parts. The overall framework is as follows Figure 4 As shown in the figure, in the order of data transmission, there are encoder network, decoder network and classifier. The first two parts are symmetrical in structure, and there are 13 convolutional layers in each network. Finally, the calculation result of the network is output as a prediction value through the classifier at the end.

[0142] Each encoder in the encoder network generates a feature map, i.e. a set of feature map values, through the convolution operation of the convolution layer. The convolution is performed in the same way, and the size of the features before and after the convolution operation is the same. Next, the feature map is batch normalized, and the ReLU activation function is applied to the batch normalized result. Finally, the feature map is max-pooled using a 2*2 window with a window movement step of 2.

[0143] Max pooling acts on the input image and uses the translation invariance of features in a small space to retain the main features of the feature map. Similarly, in the feature map generated by downsampling, each pixel retains the local features of the input image (according to the sampling window setting). Both pooling and downsampling can achieve a certain degree of feature invariance, and even if some information is lost, the robustness of the classifier can be maintained on low-resolution features. However, as the network level continues to deepen, the data information loss caused by pooling is too much, so that most of the image boundary information will be lost, which greatly affects the image segmentation effect. Therefore, unlike the pooling process of previous convolutional networks, in the Segnet neural network, the boundary information in the image is obtained and saved. After the encoder generates the feature map and before downsampling, the position index of the maximum pooling is obtained and stored, that is, in the calculation process of each encoder, the position information of the maximum value in the pooling window is recorded.

[0144] In the decoder network, each decoder will upsample the input feature map. The upsampling process is as follows Figure 5 As shown. In the corresponding encoder, the position information generated by the pooling process will be recorded, and then the input data will be placed according to the pooling position index information to achieve upsampling. Next, batch normalization is performed on each feature map. In this network, the size and number of channels of the feature map generated by the decoder are the same as the encoder input at the corresponding position. There is a special case where the number of channels of the encoder that accepts image input is determined by the input sample image, which may be inconsistent with the number of channels of the feature map generated by the corresponding decoder at the end.

[0145] At the end of the Segnet network structure, a Softmax classifier is generally used. The high-dimensional feature map output by the decoder network is used to classify all pixels through the classifier. For a classification task with K categories, the output result of the classifier is a probability image with K channels. Among them, each pixel has a corresponding probability value in each channel, that is, the probability that the pixel belongs to the category of the channel. The channel with the highest probability value among all channels is the final predicted category result of the pixel. Batch Normalization (BN) was proposed to solve the problem of internal covariate shift. In experiments, covariates refer to factors that are not subject to human intervention during the experiment and will affect the experimental results. In the network model, they can be regarded as input data. In deep neural networks, it is generally assumed that the distribution of data in the training set and the test set is similar, which is also a prerequisite for the effectiveness of deep neural network training and prediction models. However, in the training process of traditional deep learning networks, the data distribution of the middle layer will change greatly with the update of parameters. Generally, the deeper the middle layer in the neural network, the more obvious this phenomenon is. Slight changes in shallow network parameters will be amplified by multiple layers of the network and have a significant impact on the subsequent data distribution in deeper layers. Neural network training will continuously adjust parameters to adapt to changes in data distribution, making it difficult for model training to converge. This phenomenon, in which the distribution of data in the middle layer of the network changes due to changes in network parameters during training, is called internal covariate shift. Internal covariate shift can easily cause network training to fall into the gradient saturation zone, slow model training, and preprocessing of the original data alone is ineffective. To solve this problem, some scholars have proposed batch normalization operations. The batch normalization layer, referred to as the BN layer, is set after the convolution operation to normalize the feature data and then input the normalized data into the activation function. The processing method of the BN layer is to readjust the feature data output by the convolution operation to a normal distribution. "Batch" means that during training, the mean and variance of a portion of the data in the current layer are used to standardize the input of each layer. The steps are as follows: (1) Calculate the mean μ of the batch data of the current layer B , as shown in the following formula:

[0146]

[0147] Among them, m represents the data capacity in the current layer, x i is a specific data value;

[0148] Calculate batch data variance As shown in the following formula:

[0149]

[0150] Data Normalization As shown in the following formula:

[0151]

[0152] Among them, ε is a very small positive number, and ε is added to prevent the denominator from being 0;

[0153] For the normalized data, appropriate scaling and translation are required to retain the data features learned during training, as shown in the following formula:

[0154]

[0155] Among them, γ and β are learnable parameters involved in training. γ represents scaling the size of the result data, and β represents translating the distribution of the result data. The purpose of adding these two parameters to the BN layer is to allow the data after batch normalization to still maintain the overall characteristics of the original data and retain the characteristics obtained through network training, rather than forcibly restricting the data distribution to a standard normal distribution. By scaling and translating the data, the output data of the BN layer has a mean of β and a variance of γ 2 And, when the parameters γ and β are equal to σ B and μ B When , the output data of the BN layer is restored to the input value. The batch normalization data processing method can suppress the problem of gradient failure in the model, prevent overfitting of the training results, and accelerate network convergence.

[0156] (3) 3D segmentation

[0157] Medical images have fuzzy boundaries and complex gradients, which require more high-resolution information. However, the distribution of segmentation targets in human images is very regular, the internal structure of the human body is relatively fixed, and the semantics are simple and clear. Low-resolution information can provide this information for the recognition of target objects. Therefore, both high-level semantic information and low-level features are very important in medical image segmentation tasks. U-Net combines low-resolution information (providing a basis for object category recognition) and high-resolution information (providing a basis for accurate segmentation and positioning), which is very suitable for medical image segmentation. The bottom-level information is low-resolution information after multiple downsampling, which can provide contextual semantic information of the segmentation target in the entire image, and can be understood as a feature that reflects the relationship between the target and its environment. This feature helps to judge the category of the object. The high-level information is high-resolution information that is directly transferred from the encoding path to the layer corresponding to the decoding path at the same height through the concat operation. It can provide more refined features for segmentation, such as gradients. Based on the 3DU-Net structure, this paper proposes a two-stage 3D U-Net network structure 3D UV-Net. Make full use of high-level information and low-level information to identify and accurately locate the target structure. The two-stage method is mainly used to eliminate the noise generated by other similar tissue structures, thereby improving the segmentation accuracy of the target tissue. The preprocessed data is input into Net1, which is a 4-level 3D U-net structure for rough segmentation of the target structure; Net2 is a 4-level 3D V-Net structure, which adds a residual module to improve convergence and perform fine segmentation. The input of Net2 is composed of the input of Net1 and the output of Net1 after filtering and removing noise. Figure 6 shown.

[0158] Net1 is designed based on the classic 3D U-Net structure, such as Figure 7As shown. It is divided into two parts: the encoding path and the decoding path. In the encoding path, each level contains two 3*3*3 convolutional layers, and then uses normalized batch processing to speed up convergence, followed by a linear correction unit (i.e., ReLU function), and then a 2*2*2 maximum pooling layer with a step size of 2. In the decoding path, each level contains a 2*2*2 deconvolution layer with a step size of 2, followed by two 3*3*3 convolutional layers, each of which is followed by a BN layer and a ReLU layer. Finally, a 1*1*1 convolution is used to reduce the number of channels to the required output number of channels for the output of the segmentation result. In the encoding path, layers of the same resolution are passed to the decoding path through jump connections to provide it with original high-resolution features. When splicing and fusing the same-level features of encoding and decoding, their spatial resolutions are required to be strictly consistent. Therefore, the selection of the size value of the input image is strictly restricted. Therefore, the padding method of padding = 1 is adopted, so that the output after 3*3*3 convolution is consistent with the spatial resolution of the input, and the same-level features can be directly spliced.

[0159] Net2 is an improved 3D convolutional neural network, such as Figure 8 As shown in the figure, the size of each level of output feature map is marked. This network is based on the V-Net network structure, and a bottleneck structure is added to the V-Net network structure. Similarly, Net2 also has two paths, namely the encoding path and the decoding path. In the encoding path, it contains a bottleneck block, followed by a 2*2*2 convolution layer with a stride of 2, which is used to replace the pooling operation, thereby reducing memory usage and extracting global image features; in the decoding path, it contains a 2*2*2 deconvolution layer with a stride of 2, followed by a bottleneck block. Finally, a 1*1*1 convolution layer is used to reduce the number of output channels to 1.

[0160] In each level, a residual structure is added using a method similar to ResNet, such as Fig. 9 As shown in the figure, the input of each level has two flows, one is through the bottleneck structure, and the other is directly added to the output of the last convolutional layer of the level. The feature fusion adopts the Element-wise Product method, that is, directly multiplying the values ​​of corresponding positions. Compared with the Element-wiseSum method, this method can highlight or suppress the feature information of certain specific areas, which is conducive to the detection of small targets; compared with the feature map splicing in Net1, this method can reduce memory usage.

[0161] The bottleneck structure is as follows Fig.10As shown in the figure, it consists of three layers of convolution. The first layer is a 1*1*1 convolution layer, which is used to reduce the number of channels. The feature map is then sent to a 3*3*3 convolution layer, and finally a 1*1*1 convolution layer is used to restore the number of channels of the feature map. By reducing and restoring the number of feature channels, cross-channel features can be more effectively fused, and the model size and running time are reduced, making it more suitable for processing large 3D images.

[0162] The loss function is used to measure the inconsistency between the model's predicted value f(x) and the true value Y. It is a non-negative real-valued function, usually represented by L(Y, f(x)). The smaller the loss function, the better the robustness of the model. The loss function is the core part of the empirical risk function and an important part of the structural risk function. In the task of medical image segmentation, the set similarity metric function Dice coefficient is often used as the loss function. This function is usually used to calculate the similarity of two samples, and the value range is [0,1]:

[0163]

[0164] Here, |X∩Y| represents the intersection between X and Y, and |X| and |Y| represent the number of elements in X and Y, respectively. The coefficient of the numerator is multiplied by 2 because the common elements in X and Y are repeatedly counted in the denominator. In this segmentation task, x represents the groundtruth segmentation image, and Y represents the predicted segmentation image.

[0165] As can be seen from the following formula, the closer the Dice coefficient is to 1, the better the prediction result performance is. However, the loss function needs to be as small as possible, so simply subtracting 1 can get Diceloss (Dice coefficient loss function):

[0166]

[0167] There are many functions for optimizing the Dice function, such as the mean Sorensen Dice loss function, the Sorensen Dice loss function of the direct optimization term, etc. This paper proposes an improved joint loss function, which is composed of the cross entropy loss function and the mean Sorensen Dice loss function:

[0168] L total =L S_Dice +L CE

[0169] The cross entropy loss function is the most commonly used loss function in image segmentation tasks. In semantic segmentation, each pixel needs to be assigned a unique category. The cross entropy loss function needs to evaluate the class prediction of each pixel and average it over all pixels. The cross entropy loss function is shown in the following formula:

[0170]

[0171] Where p(x) represents the predicted value and q(x) represents the true value. The mean Sorensen Dice loss function is shown in the following formula:

[0172]

[0173] For smoothing purposes, an eps factor of 1 is added to the denominator.

[0174] Reference Figure 2 As shown, a high-resolution dual-modality skin measurement and analysis method based on artificial intelligence is applied to any of the above-mentioned high-resolution dual-modality skin measurement and analysis devices based on artificial intelligence, comprising the following steps:

[0175] S1. Select the device working mode;

[0176] S2, turn on the light source;

[0177] S3, collecting images;

[0178] S4. Artificial intelligence analyzes images.

[0179] Furthermore, the device working modes include using only RCM, using only OCT, and using both RCM and OCT.

[0180] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0181] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for skin analysis using an artificial intelligence algorithm, characterized in that: The following steps are involved: S1. Select the device working mode; S2, turn on the light source; S3, collecting images; S4. Use artificial intelligence algorithms to process and analyze RCM images and OCT images.

2. The method for skin analysis using an artificial intelligence algorithm according to claim 1, characterized in that: The device in S1 is a high-resolution dual-modality skin measurement and analysis device, including an imaging probe, an RCM imaging device, an OCT imaging device, and a system control and data processing device; the imaging probe is optically connected to the RCM imaging device and the OCT imaging device, respectively, and the RCM imaging device and the OCT imaging device are electrically connected to the system control and data processing device, respectively; An imaging probe, used for detecting a sample to be tested; An RCM imaging device, used for providing a first incident light beam to an imaging probe and receiving a first return light beam returned from the imaging probe for RCM imaging; An OCT imaging device, used for providing a second incident light beam to an imaging probe and receiving a second return light beam returned from the imaging probe for OCT imaging; System control and data processing device, used for receiving, analyzing, processing and displaying RCM images and OCT images.

3. The method for skin analysis using an artificial intelligence algorithm according to claim 2, characterized in that: In S1, the device working modes include using only the RCM imaging device, using only the OCT imaging device, and using both the RCM imaging device and the OCT imaging device.

4. The method for skin analysis using an artificial intelligence algorithm according to claim 1, characterized in that: In S2, the light source is an RCM light source and / or an OCT light source.

5. The method for skin analysis using an artificial intelligence algorithm according to claim 1, characterized in that: In S4, the artificial intelligence algorithms for processing and analyzing RCM images and OCT images include denoising, segmentation, and 3D segmentation.

6. The method for skin analysis using an artificial intelligence algorithm according to claim 5, characterized in that: The denoising method uses the image denoising algorithm based on deep learning DeblurGAN. The network architecture consists of two parts: the generator and the discriminator. I B =k(M)*I S +A Where: I B represents the blurred image, k(M) represents the unknown blur kernel determined by M, and I S represents the latent image, A represents the additive noise; In DeblurGAN, the loss function is defined as the weighted sum of multiple loss functions, expressed as: L=L GAN +λ·L X Where: λ = 100; Since there is no need to penalize the mismatch between input and output, L GAN Some of them use WGAN-GP as the loss function, which is expressed as follows: The other part of the loss function L X Defined as content loss, the two typical choices are L1 loss or L2 loss defined on the original image; using these functions as the only optimization target is defined based on the difference between the CNN feature maps between the generated and target images, and its specific expression is as follows: in is the feature map obtained by the jth convolution before the i-th max pooling layer in the VGG19 network; W i,j and H i,j is the dimension of the feature map, using the activation function of the VGG convolutional layer.

7. The method for skin analysis using an artificial intelligence algorithm according to claim 5, characterized in that: The segmentation uses the Segnet semantic segmentation network, which adopts an encoder and decoder structure for pixel-level image segmentation. Calculate the mean μ of the batch data in the current layer B , as shown in the following formula: Where: m represents the data capacity in the current layer, x i is a specific data value; Calculate batch data variance As shown in the following formula: Data Normalization As shown in the following formula: Where: ε is a very small positive number, and ε is added to prevent the denominator from being 0; The normalized data is appropriately scaled and translated to retain the data features learned during training, as shown in the following formula: Where: γ and β are learnable parameters involved in training, γ represents scaling the size of the result data, and β represents translating the distribution of the result data; the BN layer adds these two parameters, and by scaling and translating the data, the output data of the BN layer obeys a mean of β and a variance of γ 2 Normal distribution; and, when the parameters γ and β are equal to σ B and μ B When , the output data of the BN layer is restored to the input value.

8. The method for skin analysis using an artificial intelligence algorithm according to claim 5, characterized in that: For 3D segmentation, a two-stage 3D U-Net network structure 3DUV-Net is designed based on the 3DU-Net structure. The 3D UV-Net includes Net1 and Net2. Net1 is a 4-level 3D U-net structure used for rough segmentation of the target structure. Net2 is a 4-level 3D V-Net structure that adds a residual module to improve convergence and perform fine segmentation; The input of Net2 is composed of the input of Net1 and the output of Net1 after filtering and removing noise points; The Dice coefficient, a set similarity measurement function, is used as the loss function. This function is used to calculate the similarity between two samples, and its value range is [0,1]: Where |X∩Y| represents the intersection between X and Y, |X| and |Y| represent the number of elements in X and Y respectively; X represents the ground truth segmentation image, and Y represents the predicted segmentation image; The expression of the Dice coefficient loss function is: The Dice function is optimized by an improved joint loss function, where the improved joint loss function is composed of the cross entropy loss function and the mean Sorensen Dice loss function: L total =L S_Dice +L CE The cross entropy loss function is shown in the following formula: Where: p(x) represents the predicted value, q(x) represents the true value; The mean Sorensen Dice loss function is shown in the following formula: Where: The eps factor is 1.

9. The method for skin analysis using an artificial intelligence algorithm according to claim 8, characterized in that: Net1 is designed based on the 3D U-Net structure and is divided into two parts: the encoding path and the decoding path. In the encoding path, each level contains two 3*3*3 convolutional layers, followed by normalized batch processing, followed by a linear correction unit, and then a 2*2*2 maximum pooling layer with a stride of 2. In the decoding path, each level contains a 2*2*2 deconvolution layer with a stride of 2, followed by two 3*3*3 convolutional layers, each of which is followed by a BN layer and a ReLU layer. Finally, a 1*1*1 convolution is used to reduce the number of channels to the required output number of channels and output the segmentation results. In the encoding path, layers of the same resolution are passed to the decoding path through jump connections to provide original high-resolution features. When splicing and fusing the same-level features of encoding and decoding, the spatial resolution is required to be strictly consistent.

10. The method for skin analysis using an artificial intelligence algorithm according to claim 8, characterized in that: Net2 is an improved 3D convolutional neural network. It is based on the V-Net network structure and adds a bottleneck structure to the V-Net network structure. Net2 has an encoding path and a decoding path. In the encoding path, it contains a bottleneck block, followed by a 2*2*2 convolution layer with a stride of 2, which is used to replace the pooling operation, thereby reducing memory usage and extracting global image features; in the decoding path, it contains a 2*2*2 deconvolution layer with a stride of 2, followed by a bottleneck block, and finally a 1*1*1 convolution layer to reduce the number of output channels to 1; a residual structure is added to each level, that is, the input of each level has two flows, one is through the bottleneck structure, and the other is directly added to the output of the last convolution layer of the level; feature fusion adopts the Element-wise Product method, that is, the values ​​of corresponding positions are directly multiplied; The bottleneck structure consists of three layers of convolution. The first layer is a 1*1*1 convolution layer, which is used to reduce the number of channels. The feature map is then sent to a 3*3*3 convolution layer, and finally a 1*1*1 convolution layer is used to restore the number of channels of the feature map.

Images