A method for reconstructing internal and external fingerprints from OCT based on neural network structure search

Abstract

The present invention discloses a method for reconstructing internal and external fingerprints from optical coherence tomography (OCT) based on a neural network structure search. The method includes collecting a fingerprint OCT image dataset and performing preprocessing. The method utilizes a neural architecture search network, which is updated and adjusted based on the training of the entire internal and external fingerprint extraction network. After the internal and external fingerprint extraction network is trained, a determined neural architecture search network is obtained, enabling more accurate extraction of the stratum corneum contour and the viable epidermal connection layer contour, thereby reconstructing internal and external fingerprints with more texture features. Furthermore, the method utilizes a contour regression module that incorporates the classic multi-head attention mechanism in the Transformer and utilizes the temporal characteristics of contours to enable the internal and external fingerprint extraction network to focus on contours of different regions, capture long-range contour dependencies, and output more accurate contour information.

Description

Technical Field

[0001] The present invention belongs to the technical field of fingerprint recognition, and in particular relates to an OCT internal and external fingerprint reconstruction method based on neural network structure search. Background Art

[0002] Due to the uniqueness and permanence of fingerprints, fingerprint recognition technology has long played a vital role in financial management and public security. Conventional automatic fingerprint recognition systems capture 2D fingerprint texture information from the outside of the fingertip to identify individuals. However, fingerprint information from the skin's surface is affected by factors such as age and environment. When the finger surface is permanently damaged by aging, wear, and tear, the quality of the fingerprint on the finger surface deteriorates, affecting the performance of the fingerprint recognition system. Furthermore, the ridges and valleys of the finger surface can be easily counterfeited by fingerprint films made of materials such as silicone, thereby deceiving security systems.

[0003] Studies have shown that the epidermal fingerprint of a finger originates from the living epidermal connection layer at the junction of the epidermis and dermis of the finger. The ridge and valley information of the finger surface can be obtained from the fluctuations of the living epidermal connection layer, and the surface fingerprint can be reconstructed. The fingerprint obtained based on the contour of the living epidermal connection layer is called the internal fingerprint, which is the source of the epidermal fingerprint. When the finger is subjected to trauma such as immersion or cutting that does not involve the dermis, the internal fingerprint can be reconstructed to obtain relatively complete fingerprint information. When an individual's fingerprint is blurred due to aging and wear, the internal fingerprint can serve as a supplement to the fingerprint information. In order to obtain internal information of the finger, optical coherence tomography (OCT), a non-invasive imaging technology, obtains depth information of 1 to 3 mm under the skin of the finger by capturing a kind of 3D fingertip volume data, providing a new direction for obtaining internal fingerprints.

[0004] Existing OCT fingerprint reconstruction methods are divided into manual methods and deep learning methods. Manual methods mainly find the contours of the stratum corneum and the living epidermis connecting layer through grayscale changes or clustering, and then reconstruct the inner and outer fingerprints. More accurate contours mean that the reconstructed inner and outer fingerprints have more texture features. However, due to differences between individuals, the method parameters are difficult to adapt to the differences between individuals, and manual intervention is required, making it difficult to improve efficiency. Deep learning methods are easily affected by noise and have difficulty extracting the precise contours of the stratum corneum and the living epidermis connecting layer. In addition, the manually designed network structure depends on the designer's own understanding of OCT images and neural networks, and requires manual parameter adjustment, which has certain limitations. OCT fingerprint reconstruction requires a method that is robust to noise and can minimize the limitations of manual design. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems raised in the background technology and propose an OCT internal and external fingerprint reconstruction method based on neural network structure search.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] The present invention proposes an OCT internal and external fingerprint reconstruction method based on neural network structure search, which includes collecting fingerprint OCT image data sets and performing preprocessing;

[0008] Constructing an internal and external fingerprint extraction network, wherein the internal and external fingerprint extraction network includes a neural architecture search network and a contour regression module;

[0009] The internal and external fingerprint extraction network is trained using a preprocessed fingerprint OCT image dataset. Before each round of training, the structure of the neural architecture search network is first determined. Then, the entire internal and external fingerprint extraction network is trained iteratively until the loss function converges. The trained internal and external fingerprint extraction network is obtained. The multiple outputs of the determined neural architecture search network are passed through the contour regression module, and the outputs of the contour regression module are fused to obtain the output of the internal and external fingerprint extraction network.

[0010] Obtain multiple preprocessed OCT images of the fingerprint to be detected, and input them into the trained internal and external fingerprint extraction network respectively to obtain their respective stratum corneum contours and living epidermal connection layer contours. Based on the multiple stratum corneum and living epidermal connection layer contours, calculate and obtain the internal and external fingerprints corresponding to the OCT image to be detected.

[0011] Preferably, before determining the structure of the neural architecture search network during the training process, the neural architecture search network includes a first common sampling unit, a second common sampling unit, and a third common sampling unit connected in sequence, and a fourth common sampling unit, a fifth common sampling unit, and a sixth common sampling unit connected in sequence, and a first upsampling unit, a second upsampling unit, and a third upsampling unit connected in sequence, and a fourth upsampling unit, a fifth upsampling unit, and a sixth upsampling unit connected in sequence, and a first downsampling unit, a second downsampling unit, and a third downsampling unit connected in sequence, and a fourth downsampling unit, a fifth downsampling unit, and a sixth downsampling unit connected in sequence;

[0012] Each normal sampling unit includes multiple candidate operations, which are: normal convolution, void convolution, dynamic convolution, depth-separable convolution and skip connection; each upsampling unit includes multiple candidate operations, which are: upsampling convolution, upsampling dynamic convolution, upsampling depth-separable convolution, upsampling void convolution and bilinear interpolation; each downsampling unit includes multiple candidate operations, which are: maximum pooling, average pooling, downsampling convolution, downsampling dynamic convolution, downsampling void convolution and downsampling depth-separable convolution;

[0013] Before the first round of training, for each sampling unit, the parameters of each candidate operation are randomly initialized and weights Each sampling unit is represented by a directed acyclic graph, and the directed acyclic graph contains two input features and four intermediate features, as well as multiple pointing edges. Before the sampling unit is determined, each pointing edge contains multiple candidate operations based on the parameters of each candidate operation. The determined sampling unit is obtained by determining the pointing edges between the two input features and the four intermediate features in the directed acyclic graph and the specific candidate operations on the pointing edges by a preset method, wherein for each intermediate feature, two pointing edges pointing to the intermediate feature are included; for each intermediate feature in the ordinary sampling unit, the candidate operations on the two pointing edges pointing to the intermediate features are all the candidate operations of the ordinary sampling unit; for each intermediate feature in the upsampling unit, one of the candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the upsampling unit, and the other candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the ordinary sampling unit; for each intermediate feature in the downsampling unit, one of the candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the downsampling unit, and the other candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the ordinary sampling unit;

[0014] Finally, the result of adding the four intermediate features of the determined sampling unit is used as the output of each sampling unit;

[0015] The entire internal and external fingerprint extraction network is trained for one round, and the loss function is calculated. The parameters of the internal and external fingerprint extraction network are updated based on the gradient descent method, and the parameters of each candidate operation are updated. and weights The parameters of the follower's internal and external fingerprint extraction networks are updated together;

[0016] Iterate back and forth until the loss function converges, and the final neural architecture search network and the trained internal and external fingerprint extraction network are obtained.

[0017] Preferably, in the neural architecture search network, the input of the neural architecture search network is passed through the first convolutional layer to obtain a first feature, the first feature and the input of the neural architecture search network are used as two input features of the first common sampling unit to obtain the output feature of the first common sampling unit, and the first feature and the input of the neural architecture search network are also used as two input features of the first downsampling unit to obtain the output feature of the first downsampling unit, the input of the neural architecture search network and the output feature of the first downsampling unit are used as two input features of the sixth upsampling unit to obtain the output feature of the sixth upsampling unit, the output feature of the sixth upsampling unit is weightedly summed with the output feature of the first common sampling unit to obtain the second feature, and the second feature is added to the neural architecture search network. The input of the architecture search network is used as the two input features of the second common sampling unit to obtain the output feature of the second common sampling unit. The second feature and the output feature of the first downsampling unit are used as the two input features of the fourth downsampling unit to obtain the output of the fourth downsampling unit. The output feature of the first downsampling unit is passed through the first convolutional layer to obtain the fifth feature. The fifth feature and the output feature of the first downsampling unit are used as the two input features of the fourth common sampling unit to obtain the output feature of the fourth common sampling unit. The fifth feature and the output feature of the first downsampling unit are used as the two input features of the second downsampling unit to obtain the output feature of the second downsampling unit. The output feature of the second downsampling unit and the output feature of the first downsampling unit are used as The two input features of the fourth upsampling unit are used to obtain the output feature of the fourth upsampling unit. The output feature of the fourth upsampling unit, the output feature of the fourth common sampling unit and the output feature of the fourth downsampling unit are weighted and summed to obtain the sixth feature. The sixth feature and the second feature are used as the two input features of the fifth upsampling unit to obtain the output feature of the fifth upsampling unit. The output feature of the second common sampling unit and the output feature of the fifth upsampling unit are weighted and summed to obtain the third feature. The sixth feature and the output feature of the first downsampling unit are used as the two input features of the fifth common sampling unit to obtain the output feature of the fifth common sampling unit. The third feature and the sixth feature are used as the two input features of the sixth downsampling unit to obtain the sixth feature. The output features of the downsampling unit, the sixth feature and the output feature of the second downsampling unit are used as the two input features of the fifth downsampling unit to obtain the output features of the fifth downsampling unit, the output features of the second downsampling unit are passed through the first convolution layer to obtain the eighth feature, the eighth feature and the output feature of the second downsampling unit are used as the two input features of the sixth common sampling unit to obtain the output features of the sixth common sampling unit, the eighth feature and the output feature of the second downsampling unit are used as the two input features of the third downsampling unit to obtain the output features of the third downsampling unit, the output features of the third downsampling unit and the output features of the second downsampling unit are used as the two input features of the first upsampling unit to obtain the output features of the first upsampling unit,Performing a weighted summation on the output feature of the first upsampling unit, the output feature of the sixth normal sampling unit, and the output feature of the fifth downsampling unit to obtain a ninth feature; using the ninth feature and the sixth feature as two input features of the second upsampling unit to obtain the output feature of the second upsampling unit; performing a weighted summation on the output feature of the second upsampling unit, the output feature of the sixth downsampling unit, and the output feature of the fifth normal sampling unit to obtain a seventh feature; using the seventh feature and the third feature as two output features of the third upsampling unit to obtain the output feature of the third upsampling unit; using the third feature and the second feature as two input features of the third normal sampling unit to obtain the output feature of the third normal sampling unit; performing a weighted summation on the output feature of the third normal sampling unit and the output feature of the third upsampling unit to obtain a fourth feature;

[0018] The weight in the weighted sum is the weight of each determined sampling unit .

[0019] Preferably, the features obtained by downsampling the input of the internal and external fingerprint extraction network are used as the input of the neural architecture search network;

[0020] In the contour regression module, first, the input of the neural architecture search network is added to the fourth feature to obtain the 23rd feature, the output feature of the first downsampling unit is added to the seventh feature to obtain the 24th feature, and the output feature of the second downsampling unit is added to the ninth feature to obtain the 25th feature;

[0021] Then, the input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature, and the twenty-fifth feature are sequentially passed through the second convolutional layer and the multi-head attention mechanism group to obtain the tenth feature, eleventh feature, twelfth feature, thirteenth feature, fourteenth feature, and fifteenth feature corresponding to the input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature, and the fifteenth feature, respectively;

[0022] The tenth feature is added to the thirteenth feature to obtain the sixteenth feature, the eleventh feature is added to the fourteenth feature to obtain the seventeenth feature, and the twelfth feature is added to the fifteenth feature to obtain the eighteenth feature. The sixteenth feature, the seventeenth feature, and the eighteenth feature are all passed through the fully connected layer and the activation layer in sequence to obtain the nineteenth feature, the twentieth feature, and the twenty-first feature corresponding to the sixteenth feature, the seventeenth feature, and the eighteenth feature in sequence, respectively. The twenty-first feature is added to the twentieth feature after passing through the upsampling layer to obtain a first addition result, and then the first addition result is multiplied by a preset value to obtain the twenty-second feature. The twenty-second feature is added to the nineteenth feature after passing through the upsampling layer to obtain a second addition result, and then the second addition result is multiplied by the preset value to obtain the output of the internal and external fingerprint extraction network, and the output of the internal and external fingerprint extraction network is the stratum corneum contour and the active epidermal connection layer contour, wherein the stratum corneum contour includes the stratum corneum upper surface contour and the stratum corneum lower surface contour, and each pixel point of each contour has a depth value.

[0023] Preferably, the multi-head attention mechanism group includes four multi-head attention mechanism modules connected in sequence, and each of the multi-head attention mechanism modules includes a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a linear layer connected in sequence, wherein the input of the multi-head attention mechanism module is added to the output of the multi-head attention mechanism layer to obtain an added feature, and the added feature is used as the input of the second normalization layer, and the output of the linear layer is added to the added feature to obtain the output of the multi-head attention mechanism module.

[0024] Preferably, during the pre-processing process, the OCT image is intercepted using a preset rectangular frame to capture the stratum corneum region and the viable epidermal junction region;

[0025] During the training process, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum, and the contour of the living epidermal junction layer of the OCT image are annotated, as well as the area of ​​the stratum corneum and the area of ​​the living epidermal junction layer;

[0026] After each round of training, the fourth feature in the neural architecture search network is passed through the third convolutional layer and the upsampling layer to obtain a predicted image of the area including the stratum corneum and the area of ​​the living epidermal connection layer;

[0027] Loss function sampled during training The calculation formula is as follows:

[0028] ;

[0029] in,

[0030] ;

[0031] ;

[0032] in, represents the calculated loss of both the stratum corneum area and the viable epidermal junction area, express The weight of It represents the loss calculated from the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal junction layer. express The weight of Indicates all the pixels on an OCT image in the fingerprint OCT image dataset. represents the area of ​​the stratum corneum, the area of ​​the living epidermal junction layer, or the background area on the predicted image, and When is the area of ​​the stratum corneum, When represents the area of ​​the living epidermal junction layer, When , it represents the background area. express On the pixels, and Do not indicate the type In the region The predicted pixel value and the true pixel value of each pixel point are Indicates the number of pixels on the predicted stratum corneum upper surface contour, predicted stratum corneum lower surface contour, or predicted living epidermal junction layer contour. indicates a predicted upper stratum corneum surface contour, a predicted lower stratum corneum surface contour, or a predicted live epidermal junctional layer contour, and When , it indicates the predicted surface contour of the stratum corneum. When , it indicates the predicted surface contour under the stratum corneum. When is the predicted contour of the living epidermal junction layer, and Respectively expressed in the type On the outline of The predicted contour depth value and the actual contour depth value of each pixel.

[0033] Preferably, in the process of calculating the inner and outer fingerprints corresponding to the OCT image to be detected based on the contours of the multiple stratum corneum and the living epidermis connection layer, the inner fingerprint includes a deep inner fingerprint and a grayscale inner fingerprint, and the outer fingerprint includes a deep outer fingerprint and a grayscale outer fingerprint;

[0034] The OCT image of the fingerprint to be detected is Each OCT image has a size of , Indicates width, After each OCT image is trained through the internal and external fingerprint extraction network, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal connection layer are obtained. Each pixel point of each contour has a depth value, and the width of each contour is , that is, there are Depth values ​​are obtained and each contour is mapped to the corresponding OCT image of the fingerprint to be detected;

[0035] For each OCT image of the fingerprint to be detected, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum, and the contour of the living epidermis-epidermal junction layer are correspondingly subtracted from each other the depth values ​​of the pixel points at the corresponding positions of the lower surface contour of the stratum corneum and the upper surface contour to obtain a first result, and then the depth values ​​of the pixel points at the corresponding positions of the living epidermis-epidermal junction layer contour and the upper surface contour of the stratum corneum are correspondingly subtracted to obtain a second result;

[0036] for The upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal junction layer corresponding to the OCT image of the fingerprint to be detected will be The first results are stitched together and saved as an image to obtain the deep external fingerprint. The second results are stitched together and saved as an image to obtain the deep inner fingerprint;

[0037] For each mapped stratum corneum upper surface contour on the OCT image of the fingerprint to be detected, the grayscale values ​​of all pixels on the stratum corneum upper surface contour are obtained, and the grayscale values ​​of all pixels are added to the average first result of the preset multiple to obtain a third result, where the average first result is The average of the first results;

[0038] For the mapped The OCT image of the fingerprint to be detected is obtained A third result, and The three results are stitched together and saved as an image to obtain the grayscale external fingerprint;

[0039] For the contour of the live epidermis-epidermal connection layer on each mapped OCT image of the fingerprint to be detected, obtain the peak position pixel point and the trough position pixel point closest to each pixel point on the contour of the live epidermis-epidermal connection layer, take the average of the depth values ​​corresponding to the peak position pixel point and the trough position pixel point, and obtain the grayscale value of the pixel point where the average value is located, referred to as the first grayscale value. The first grayscale values ​​corresponding to all pixels on the contour of the live epidermis-epidermal connection layer on each mapped OCT image of the fingerprint to be detected are used as the fourth result;

[0040] For the mapped The OCT image of the fingerprint to be detected is obtained The fourth result will be The fourth result is spliced ​​and saved as an image to obtain the grayscale inner fingerprint.

[0041] Preferably, the upsampling convolution includes ordinary convolution and upsampling operations connected in sequence, the upsampling dynamic convolution includes dynamic convolution and upsampling operations connected in sequence, the upsampling depth-separable convolution includes depth-separable convolution and upsampling operations connected in sequence, the upsampling hole convolution includes hole convolution and upsampling operations connected in sequence, the downsampling convolution includes ordinary convolution and downsampling operations connected in sequence, the downsampling dynamic convolution includes dynamic convolution and downsampling operations connected in sequence, the downsampling hole convolution includes hole convolution and downsampling operations connected in sequence, and the downsampling depth-separable convolution includes depth-separable convolution and downsampling operations connected in sequence.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This OCT internal and external fingerprint reconstruction method based on neural network structure search uses a neural architecture search network, which is updated and adjusted based on the training of the entire internal and external fingerprint extraction network. Once the internal and external fingerprint extraction network is trained, a determined neural architecture search network is obtained, which allows for more accurate extraction of the stratum corneum contour and the living epidermal connection layer contour, thereby reconstructing the internal and external fingerprint with more texture features.

[0044] 2. This OCT internal and external fingerprint reconstruction method based on neural network structure search uses a contour regression module, and the contour regression module integrates the classic multi-head attention mechanism in Transformer. It utilizes the temporal characteristics of the contour to enable the internal and external fingerprint extraction network to focus on the contours of different regions, capture the long-distance dependence of the contour, and output more accurate contour information. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the process of the OCT internal and external fingerprint reconstruction method based on neural network structure search of the present invention;

[0046] Figure 2 This is a structural diagram of the internal and external fingerprint extraction network of the present invention;

[0047] Figure 3 This is a structural diagram of the neural architecture search network of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of the multi-head attention mechanism module of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of all common sampling units in the trained internal and external fingerprint extraction network of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of all upsampling units in the trained internal and external fingerprint extraction network of the present invention;

[0051] Figure 7 This is a schematic diagram of the structure of all downsampling units in the trained internal and external fingerprint extraction network of the present invention;

[0052] Figure 8 A plurality of OCT images of fingerprints to be detected that have been preprocessed by the present invention;

[0053] Figure 9 Schematic diagram of the depth fingerprint of the present invention;

[0054] Figure 10 Schematic diagram of the fingerprint within the depth of the present invention;

[0055] Figure 11 Schematic diagram of the grayscale outer fingerprint of the present invention;

[0056] Figure 12 Schematic diagram of the grayscale fingerprint of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] like Figures 1-12 As shown, a method for reconstructing internal and external fingerprints of OCT based on neural network structure search is provided, including:

[0059] Step 1: Collect fingerprint OCT image dataset and perform preprocessing;

[0060] The OCT image is a vertical section of the fingerprint (the vertical section is perpendicular to the palm surface). During the preprocessing process, a preset rectangular frame (e.g., a rectangular frame of 128256 dimensions) is used to cut out the stratum corneum and the viable epidermal junction layer of the OCT image (thus removing invalid background information from the OCT image).

[0061] At the same time, after collecting and preprocessing the fingerprint OCT image dataset, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal connection layer of the OCT image are annotated, as well as the area of ​​the stratum corneum and the area of ​​the living epidermal connection layer are annotated for subsequent calculation of the loss function.

[0062] Step 2: Construct an internal and external fingerprint extraction network, which includes a neural architecture search network and a contour regression module.

[0063] Step 3: Use the preprocessed fingerprint OCT image dataset to train the internal and external fingerprint extraction network. Before each round of training, first determine the structure of the neural architecture search network, then train the entire internal and external fingerprint extraction network, and iterate back and forth until the loss function converges to obtain a trained internal and external fingerprint extraction network. The multiple outputs of the determined neural architecture search network are all passed through the contour regression module, and the output of the contour regression module is fused to obtain the output of the internal and external fingerprint extraction network;

[0064] Wherein, before the structure of the neural architecture search network is determined during the training process, the neural architecture search network includes a first normal sampling unit, a second normal sampling unit, and a third normal sampling unit connected in sequence (from data input to output direction), and a fourth normal sampling unit, a fifth normal sampling unit, and a sixth normal sampling unit connected in sequence (from data input to output direction), and a first upsampling unit, a second upsampling unit, and a third upsampling unit connected in sequence (from data input to output direction), and a fourth upsampling unit, a fifth upsampling unit, and a sixth upsampling unit connected in sequence (from data input to output direction), and a first downsampling unit, a second downsampling unit, and a third downsampling unit connected in sequence (from data input to output direction), and a fourth downsampling unit, a fifth downsampling unit, and a sixth downsampling unit connected in sequence (from data input to output direction);

[0065] Each ordinary sampling unit includes multiple candidate operations, which are: ordinary convolution, void convolution, dynamic convolution, depth-separable convolution and jump connection. Each upsampling unit includes multiple candidate operations, which are: upsampling convolution, upsampling dynamic convolution, upsampling depth-separable convolution, upsampling void convolution and bilinear interpolation. Each downsampling unit includes multiple candidate operations, which are: maximum pooling, average pooling, downsampling convolution, downsampling dynamic convolution, downsampling void convolution and downsampling depth-separable convolution. Among them, the upsampling convolution includes ordinary convolution and upsampling operations connected in sequence from data input to output, and the upsampling dynamic convolution includes ordinary convolution and upsampling operations connected in sequence from data input to output. Dynamic convolution and upsampling operations, upsampling depthwise separable convolution includes depthwise separable convolution and upsampling operations sequentially connected from data input to output, upsampling hole convolution includes hole convolution and upsampling operations sequentially connected from data input to output, downsampling convolution includes ordinary convolution and downsampling operations sequentially connected from data input to output, downsampling dynamic convolution includes dynamic convolution and downsampling operations sequentially connected from data input to output, downsampling hole convolution includes hole convolution and downsampling operations sequentially connected from data input to output, downsampling depthwise separable convolution includes depthwise separable convolution and downsampling operations sequentially connected from data input to output;

[0066] When determining the structure of the neural architecture search network, it is necessary to determine a fixed directed acyclic graph for each sampling unit, and all directed edges in the fixed directed acyclic graph correspond to a fixed candidate operation;

[0067] Before the first round of training, for each sampling unit, the parameters of each candidate operation are randomly initialized and weights , each sampling unit is represented by a directed acyclic graph, and the directed acyclic graph contains two input features and four intermediate features (the result of adding the four intermediate features is used as the output of the corresponding sampling unit), as well as multiple pointing edges. Before the sampling unit is determined, each pointing edge contains multiple candidate operations (after the sampling unit is determined, each pointing edge has only one determined candidate operation). Based on the parameters of each candidate operation The predetermined method is used to determine the directed edges (i.e., the connection mode of the directed edges) between the two input features and the four intermediate features in the directed acyclic graph, as well as the specific candidate operations on the directed edges, to obtain the determined sampling units (the process of obtaining the determined sampling units based on the predetermined method belongs to the prior art and will not be described in detail in this solution, such as the existing literature Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search[J]. arXiv preprint arXiv:1806.09055, 2018. The process is described in detail), where for each intermediate feature, two pointing edges are included pointing to the intermediate feature (wherein for the first intermediate feature, the two input features of the directed acyclic graph are respectively pointed to the first intermediate feature through pointing edges, and the connection method of the remaining pointing edges can be determined by a preset method), for each intermediate feature in the ordinary sampling unit, the candidate operations on the two pointing edges pointing to the intermediate feature are all the candidate operations of the ordinary sampling unit, for each intermediate feature in the upsampling unit, one of the candidate operations on the pointing edge pointing to the intermediate feature is all the candidate operations of the upsampling unit, and the other candidate operations on the pointing edge pointing to the intermediate feature is all the candidate operations of the ordinary sampling unit, for each intermediate feature in the downsampling unit, one of the candidate operations on the pointing edge pointing to the intermediate feature is all the candidate operations of the downsampling unit, and the other candidate operations on the pointing edge pointing to the intermediate feature is all the candidate operations of the ordinary sampling unit;

[0068] Finally, the result of adding the four intermediate features of the determined sampling unit is used as the output of each sampling unit;

[0069] The entire internal and external fingerprint extraction network is trained for one round, and the loss function is calculated. The parameters of the internal and external fingerprint extraction network are updated based on the gradient descent method, and the parameters of each candidate operation are updated. and weights The parameters of the follower's internal and external fingerprint extraction networks are updated together (that is, they are also updated based on the gradient descent method);

[0070] Repeated iteration (multiple training) until the loss function converges to obtain the final neural architecture search network (the structure of each common sampling unit in the final neural architecture search network is as follows Figure 5 As shown (i.e. Figure 5 Extract the structure of all common sampling units in the trained internal and external fingerprint network (directed acyclic graph), and finally determine the structure of each upsampling unit in the neural architecture search network as follows Figure 6 As shown (i.e. Figure 6 Extract the structure of all upsampling units in the trained internal and external fingerprint network (directed acyclic graph), and finally determine the structure of each downsampling unit in the neural architecture search network as follows Figure 7 As shown (i.e. Figure 7 Extract the structure (directed acyclic graph) of all downsampling units in the trained internal and external fingerprint extraction network, where A and B represent the two input features of each directed acyclic graph, C represents the two output features of each directed acyclic graph, and 0, 1, 2, and 3 represent the four intermediate features of each directed acyclic graph, where 0 is the first intermediate feature, and so on in order; Figure 2 All common sampling units in ), and obtain the trained internal and external fingerprint extraction network. It should be noted that, in this embodiment, the labeled data is divided into three equal parts and a three-fold cross validation is performed.

[0071] During the training process, after each round of training, the fourth feature in the neural architecture search network is passed through the third convolutional layer and the upsampling layer to obtain a predicted image of the area including the stratum corneum and the area of ​​the living epidermal connection layer;

[0072] Loss function sampled during training The calculation formula is as follows:

[0073] ;

[0074] in,

[0075] ;

[0076] ;

[0077] in, represents the calculated loss of both the stratum corneum area and the viable epidermal junction area, express The weight of It represents the loss calculated from the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal junction layer. express The weight of Indicates all the pixels on an OCT image in the fingerprint OCT image dataset. represents the area of ​​the stratum corneum, the area of ​​the living epidermal junction layer, or the background area on the predicted image, and When is the area of ​​the stratum corneum, When represents the area of ​​the living epidermal junction layer, When , it represents the background area. express On the pixels, and Do not indicate the type In the region The predicted pixel value and the true pixel value of each pixel point are Indicates the number of pixels on the predicted stratum corneum upper surface contour, predicted stratum corneum lower surface contour, or predicted living epidermal junction layer contour (consistent with the width of the input image), indicates a predicted upper stratum corneum surface contour, a predicted lower stratum corneum surface contour, or a predicted live epidermal junctional layer contour, and When , it indicates the predicted surface contour of the stratum corneum. When , it indicates the predicted surface contour under the stratum corneum. When is the predicted contour of the living epidermal junction layer, and Respectively expressed in the type On the outline of The predicted contour depth value and the actual contour depth value of each pixel.

[0078] Step 4: Obtain multiple OCT images of the fingerprint to be detected after preprocessing (the size of the multiple OCT images of the fingerprint to be detected after preprocessing in this embodiment is ,in is the width of the OCT image, is the height of the OCT image, is the number of OCT images, and in this embodiment , , ,like Figure 8 As shown in the figure, the inner and outer fingerprints corresponding to the OCT image to be detected are obtained by calculating based on the multiple stratum corneum and living epidermal connection layer contours.

[0079] Among them, in the internal and external fingerprint extraction network, the input of the internal and external fingerprint extraction network is subjected to the downsampling layer to obtain the features as the input of the neural architecture search network (i.e. Figure 2 and Figure 3 Stem1 in ();

[0080] like Figure 3As shown, in the neural architecture search network, the input of the neural architecture search network is passed through the first convolutional layer to obtain the first feature, the first feature and the input of the neural architecture search network are used as the two input features of the first common sampling unit to obtain the output feature of the first common sampling unit, and the first feature and the input of the neural architecture search network are also used as the two input features of the first downsampling unit to obtain the output feature of the first downsampling unit, the input of the neural architecture search network and the output feature of the first downsampling unit are used as the two input features of the sixth upsampling unit to obtain the output feature of the sixth upsampling unit, the output feature of the sixth upsampling unit and the output feature of the first common sampling unit are weightedly summed to obtain the second feature, and the second feature is added to the neural architecture search network. The input of the structure search network is used as the two input features of the second common sampling unit to obtain the output feature of the second common sampling unit. The second feature and the output feature of the first downsampling unit are used as the two input features of the fourth downsampling unit to obtain the output of the fourth downsampling unit. The output feature of the first downsampling unit is passed through the first convolutional layer to obtain the fifth feature. The fifth feature and the output feature of the first downsampling unit are used as the two input features of the fourth common sampling unit to obtain the output feature of the fourth common sampling unit. The fifth feature and the output feature of the first downsampling unit are used as the two input features of the second downsampling unit to obtain the output feature of the second downsampling unit. The output feature of the second downsampling unit and the output feature of the first downsampling unit are used as the second downsampling unit. The two input features of the four upsampling units are used to obtain the output feature of the fourth upsampling unit. The output feature of the fourth upsampling unit, the output feature of the fourth common sampling unit and the output feature of the fourth downsampling unit are weighted and summed to obtain the sixth feature. The sixth feature and the second feature are used as the two input features of the fifth upsampling unit to obtain the output feature of the fifth upsampling unit. The output feature of the second common sampling unit and the output feature of the fifth upsampling unit are weighted and summed to obtain the third feature. The sixth feature and the output feature of the first downsampling unit are used as the two input features of the fifth common sampling unit to obtain the output feature of the fifth common sampling unit. The third feature and the sixth feature are used as the two input features of the sixth downsampling unit to obtain the sixth feature. The output features of the downsampling unit, the sixth feature and the output feature of the second downsampling unit are used as the two input features of the fifth downsampling unit to obtain the output features of the fifth downsampling unit, the output features of the second downsampling unit are passed through the first convolution layer to obtain the eighth feature, the eighth feature and the output feature of the second downsampling unit are used as the two input features of the sixth common sampling unit to obtain the output features of the sixth common sampling unit, the eighth feature and the output feature of the second downsampling unit are used as the two input features of the third downsampling unit to obtain the output features of the third downsampling unit, the output features of the third downsampling unit and the output features of the second downsampling unit are used as the two input features of the first upsampling unit to obtain the output features of the first upsampling unit,Performing a weighted summation on the output feature of the first upsampling unit, the output feature of the sixth normal sampling unit, and the output feature of the fifth downsampling unit to obtain a ninth feature; using the ninth feature and the sixth feature as two input features of the second upsampling unit to obtain the output feature of the second upsampling unit; performing a weighted summation on the output feature of the second upsampling unit, the output feature of the sixth downsampling unit, and the output feature of the fifth normal sampling unit to obtain a seventh feature; using the seventh feature and the third feature as two output features of the third upsampling unit to obtain the output feature of the third upsampling unit; using the third feature and the second feature as two input features of the third normal sampling unit to obtain the output feature of the third normal sampling unit; performing a weighted summation on the output feature of the third normal sampling unit and the output feature of the third upsampling unit to obtain a fourth feature;

[0081] The weight in the weighted sum is the weight of each determined sampling unit .

[0082] like Figure 2 As shown, in the contour regression module, first, the input of the neural architecture search network is added to the fourth feature to obtain the twenty-third feature, the output feature of the first downsampling unit is added to the seventh feature to obtain the twenty-fourth feature, and the output feature of the second downsampling unit is added to the ninth feature to obtain the twenty-fifth feature;

[0083] Then the input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature and the twenty-fifth feature are sequentially passed through the second convolutional layer (1 1 convolution) and a multi-head attention mechanism group, obtaining the tenth feature, eleventh feature, twelfth feature, thirteenth feature, fourteenth feature, and fifteenth feature corresponding to the input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature, and the twenty-fifth feature, respectively;

[0084] The tenth feature and the thirteenth feature are added to obtain the sixteenth feature, the eleventh feature and the fourteenth feature are added to obtain the seventeenth feature, the twelfth feature and the fifteenth feature are added to obtain the eighteenth feature, the sixteenth feature, the seventeenth feature and the eighteenth feature are all passed through the full connection layer and the activation layer (using the sigmoid activation function) in sequence to obtain the nineteenth feature, the twentieth feature and the twenty-first feature corresponding to the sixteenth feature, the seventeenth feature and the eighteenth feature respectively. The twenty-first feature is added to the twentieth feature after passing through the upsampling layer to obtain the first addition result, and then the first addition result is added to the preset value (wherein the preset value The twenty-second feature is obtained by multiplying the two features by a predetermined value (which is empirically set to 0.5) and the two features. After the twenty-second feature passes through the upsampling layer, the two features are added to the nineteenth feature to obtain a second addition result. The second addition result is then multiplied by a preset value (wherein the preset value is empirically set to 0.5) to obtain the output of the internal and external fingerprint extraction network. The output of the internal and external fingerprint extraction network is a stratum corneum contour with a depth value and a living epidermal connection layer contour with a depth value, wherein the stratum corneum contour includes an upper surface contour of the stratum corneum and a lower surface contour of the stratum corneum, and both the upper surface contour of the stratum corneum and the lower surface contour of the stratum corneum have depth values ​​(i.e., the output of the internal and external fingerprint extraction network is three contours with depth values. Figure 2 In order to facilitate the display of the three output contours, the three contours are placed on the corresponding areas (i.e., the area of ​​the stratum corneum and the area of ​​the living epidermal junction layer) for display).

[0085] Among them, the multi-head attention mechanism group includes four multi-head attention mechanism modules connected in sequence, and each multi-head attention mechanism module includes (from data input to output direction) a first normalization layer, a multi-head attention mechanism layer (the multi-head attention mechanism layer in Transformer), a second normalization layer and a linear layer connected in sequence, wherein the input of the multi-head attention mechanism module is added to the output of the multi-head attention mechanism layer to obtain an added feature, and the added feature is used as the input of the second normalization layer, and the output of the linear layer is added to the added feature to obtain the output of the multi-head attention mechanism module (such as Figure 4 shown).

[0086] Among them, in the process of calculating the inner and outer fingerprints corresponding to the OCT image to be detected based on the contours of multiple stratum corneum and living epidermal connection layers, the inner fingerprint includes a deep inner fingerprint and a grayscale inner fingerprint, and the outer fingerprint includes a deep outer fingerprint and a grayscale outer fingerprint;

[0087] The OCT image of the fingerprint to be detected is Each OCT image has a size of , Indicates width, After each OCT image is trained through the internal and external fingerprint extraction network, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal connection layer are obtained. Each pixel point of each contour has a depth value, and the width of each contour is (The width of the contour is the width along the OCT image), that is, there are Depth values ​​are obtained and each contour is mapped to the corresponding OCT image of the fingerprint to be detected;

[0088] For each OCT image of the fingerprint to be detected, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum, and the contour of the living epidermis-epidermal junction layer are correspondingly subtracted from each other the depth values ​​of the pixel points at the corresponding positions of the lower surface contour of the stratum corneum and the upper surface contour to obtain a first result, and then the depth values ​​of the pixel points at the corresponding positions of the living epidermis-epidermal junction layer contour and the upper surface contour of the stratum corneum are correspondingly subtracted to obtain a second result;

[0089] for The upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal junction layer corresponding to the OCT image of the fingerprint to be detected will be The first results are stitched together and saved as an image to obtain the deep external fingerprint (such as Figure 9 as shown), The second result is stitched together and saved as an image to obtain the deep inner fingerprint (such as Figure 10 shown);

[0090] For each mapped stratum corneum upper surface contour on the OCT image of the fingerprint to be detected, the grayscale values ​​of all pixels on the stratum corneum upper surface contour are obtained, and the grayscale values ​​of all pixels are added to the average first result of the preset multiple (the preset multiple is 0.8 in this embodiment) to obtain the third result, where the average first result is The average of the first results;

[0091] For the mapped The OCT image of the fingerprint to be detected is obtained A third result, and The third result is stitched together and saved as an image to obtain the grayscale external fingerprint (such as Figure 11 shown);

[0092] For the contour of the live epidermal connection layer on each mapped OCT image of the fingerprint to be detected, obtain the peak position pixel and the trough position pixel closest to each pixel point on the contour of the live epidermal connection layer (the closest horizontal distance, that is, the closest along the width direction) (where the contour of the live epidermal connection layer is a wavy curve), take the average of the depth values ​​corresponding to the peak position pixel and the trough position pixel point, and obtain the grayscale value of the pixel point where the average value occurs, referred to as the first grayscale value. The first grayscale values ​​corresponding to all pixels on the contour of the live epidermal connection layer on each mapped OCT image of the fingerprint to be detected are used as the fourth result;

[0093] For the mapped The OCT image of the fingerprint to be detected is obtained The fourth result will be The fourth result is spliced ​​and saved as an image to obtain the grayscale inner fingerprint (such as Figure 12 shown).

[0094] This OCT internal and external fingerprint reconstruction method based on neural network structure search uses a neural architecture search network, and the neural architecture search network is updated and adjusted based on the training of the entire internal and external fingerprint extraction network. When the internal and external fingerprint extraction network is trained, a determined neural architecture search network is obtained, which enables more accurate extraction of the stratum corneum contour and the living epidermal connection layer contour, thereby reconstructing the internal and external fingerprints with more texture features; this OCT internal and external fingerprint reconstruction method based on neural network structure search uses a contour regression module, and the contour regression module integrates the classic multi-head attention mechanism in Transformer, and utilizes the temporal characteristics of the contour to enable the internal and external fingerprint extraction network to pay attention to the contours of different areas, capture the long-distance dependence of the contour, and output more accurate contour information.

[0095] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for reconstructing internal and external fingerprints of OCT images based on neural network structure search, characterized by: The OCT internal and external fingerprint reconstruction method based on neural network structure search includes: Collect fingerprint OCT image dataset and perform preprocessing; Constructing an internal and external fingerprint extraction network, wherein the internal and external fingerprint extraction network includes a neural architecture search network and a contour regression module; The internal and external fingerprint extraction network is trained using a preprocessed fingerprint OCT image dataset. Before each round of training, the structure of the neural architecture search network is first determined. Then, the entire internal and external fingerprint extraction network is trained iteratively until the loss function converges. The trained internal and external fingerprint extraction network is obtained. The multiple outputs of the determined neural architecture search network are passed through the contour regression module, and the outputs of the contour regression module are fused to obtain the output of the internal and external fingerprint extraction network. Obtain multiple pre-processed OCT images of the fingerprint to be detected, input them into the trained internal and external fingerprint extraction network respectively, obtain the contours of the stratum corneum and the living epidermis connecting layer, and calculate the internal and external fingerprints corresponding to the OCT image to be detected based on the multiple contours of the stratum corneum and the living epidermis connecting layer; The neural architecture search network includes a first common sampling unit, a second common sampling unit, and a third common sampling unit connected in sequence, a fourth common sampling unit, a fifth common sampling unit, and a sixth common sampling unit connected in sequence, a first upsampling unit, a second upsampling unit, and a third upsampling unit connected in sequence, a fourth upsampling unit, a fifth upsampling unit, and a sixth upsampling unit connected in sequence, a first downsampling unit, a second downsampling unit, and a third downsampling unit connected in sequence, a fourth downsampling unit, a fifth downsampling unit, and a sixth downsampling unit connected in sequence; The fourth feature is obtained by weighted summing the output features of the third normal sampling unit and the third upsampling unit, the seventh feature is obtained by weighted summing the output features of the second upsampling unit, the sixth downsampling unit, and the fifth normal sampling unit, and the ninth feature is obtained by weighted summing the output features of the first upsampling unit, the sixth normal sampling unit, and the fifth downsampling unit. In the contour regression module, the input of the neural architecture search network is added to the fourth feature to obtain the twenty-third feature, the output feature of the first downsampling unit is added to the seventh feature to obtain the twenty-fourth feature, and the output feature of the second downsampling unit is added to the ninth feature to obtain the twenty-fifth feature. The input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature and the twenty-fifth feature are all processed in turn by the second convolutional layer and the multi-head attention mechanism group.

2. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 1, characterized in that: Before determining the structure of the neural architecture search network during training, each common sampling unit includes multiple candidate operations, namely: common convolution, dilated convolution, dynamic convolution, depth-wise separable convolution, and skip connection; each upsampling unit includes multiple candidate operations, namely: upsampling convolution, upsampling dynamic convolution, upsampling depth-wise separable convolution, upsampling dilated convolution, and bilinear interpolation; each downsampling unit includes multiple candidate operations, namely: maximum pooling, average pooling, downsampling convolution, downsampling dynamic convolution, downsampling dilated convolution, and downsampling depth-wise separable convolution; Before the first round of training, for each sampling unit, the parameters of each candidate operation are randomly initialized and weights Each sampling unit is represented by a directed acyclic graph, and the directed acyclic graph contains two input features and four intermediate features, as well as multiple pointing edges. Before the sampling unit is determined, each pointing edge contains multiple candidate operations based on the parameters of each candidate operation. The determined sampling unit is obtained by determining the pointing edges between the two input features and the four intermediate features in the directed acyclic graph and the specific candidate operations on the pointing edges by a preset method, wherein for each intermediate feature, two pointing edges pointing to the intermediate feature are included; for each intermediate feature in the ordinary sampling unit, the candidate operations on the two pointing edges pointing to the intermediate features are all the candidate operations of the ordinary sampling unit; for each intermediate feature in the upsampling unit, one of the candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the upsampling unit, and the other candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the ordinary sampling unit; for each intermediate feature in the downsampling unit, one of the candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the downsampling unit, and the other candidate operations on the pointing edges pointing to the intermediate feature is all the candidate operations of the ordinary sampling unit; Finally, the result of adding the four intermediate features of the determined sampling unit is used as the output of each sampling unit; The entire internal and external fingerprint extraction network is trained for one round, and the loss function is calculated. The parameters of the internal and external fingerprint extraction network are updated based on the gradient descent method, and the parameters of each candidate operation are updated. and weights The parameters of the follower's internal and external fingerprint extraction networks are updated together; Iterate back and forth until the loss function converges, and the final neural architecture search network and the trained internal and external fingerprint extraction network are obtained.

3. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 2, characterized in that: In the neural architecture search network, the input of the neural architecture search network is passed through the first convolutional layer to obtain the first feature, the first feature and the input of the neural architecture search network are used as the two input features of the first common sampling unit to obtain the output feature of the first common sampling unit, and the first feature and the input of the neural architecture search network are also used as the two input features of the first downsampling unit to obtain the output feature of the first downsampling unit, the input of the neural architecture search network and the output feature of the first downsampling unit are used as the two input features of the sixth upsampling unit to obtain the output feature of the sixth upsampling unit, the output feature of the sixth upsampling unit and the output feature of the first common sampling unit are weightedly summed to obtain the second feature, and the second feature is added to the neural architecture search network. The input of the search network is used as the two input features of the second ordinary sampling unit to obtain the output feature of the second ordinary sampling unit. The second feature and the output feature of the first downsampling unit are used as the two input features of the fourth downsampling unit to obtain the output of the fourth downsampling unit. The output feature of the first downsampling unit is passed through the first convolutional layer to obtain the fifth feature. The fifth feature and the output feature of the first downsampling unit are used as the two input features of the fourth ordinary sampling unit to obtain the output feature of the fourth ordinary sampling unit. The fifth feature and the output feature of the first downsampling unit are used as the two input features of the second downsampling unit to obtain the output feature of the second downsampling unit. The output feature of the second downsampling unit and the output feature of the first downsampling unit are used as the fourth downsampling unit. The two input features of the upsampling unit are used to obtain the output feature of the fourth upsampling unit, the output feature of the fourth upsampling unit, the output feature of the fourth common sampling unit and the output feature of the fourth downsampling unit are weighted and summed to obtain the sixth feature, the sixth feature and the second feature are used as the two input features of the fifth upsampling unit to obtain the output feature of the fifth upsampling unit, the output feature of the second common sampling unit and the output feature of the fifth upsampling unit are weighted and summed to obtain the third feature, the sixth feature and the output feature of the first downsampling unit are used as the two input features of the fifth common sampling unit to obtain the output feature of the fifth common sampling unit, the third feature and the sixth feature are used as the two input features of the sixth downsampling unit to obtain the sixth downsampling unit. The output features of the sampling unit, the sixth feature and the output feature of the second down-sampling unit are used as the two input features of the fifth down-sampling unit to obtain the output features of the fifth down-sampling unit, the output features of the second down-sampling unit are passed through the first convolution layer to obtain the eighth feature, the eighth feature and the output feature of the second down-sampling unit are used as the two input features of the sixth common sampling unit to obtain the output features of the sixth common sampling unit, the eighth feature and the output feature of the second down-sampling unit are used as the two input features of the third down-sampling unit to obtain the output features of the third down-sampling unit, the output features of the third down-sampling unit and the output features of the second down-sampling unit are used as the two input features of the first up-sampling unit to obtain the output features of the first up-sampling unit,The ninth feature and the sixth feature are used as two input features of the second upsampling unit to obtain the output feature of the second upsampling unit; the seventh feature and the third feature are used as two output features of the third upsampling unit to obtain the output feature of the third upsampling unit; the third feature and the second feature are used as two input features of the third common sampling unit to obtain the output feature of the third common sampling unit; The weight in the weighted sum is the weight of each determined sampling unit .

4. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 3, characterized in that: The input of the internal and external fingerprint extraction network is passed through the downsampling layer to obtain the features as the input of the neural architecture search network; The input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature, and the twenty-fifth feature are processed in sequence by the second convolutional layer and the multi-head attention mechanism group, thereby obtaining the tenth feature, eleventh feature, twelfth feature, thirteenth feature, fourteenth feature, and fifteenth feature corresponding to the input of the neural architecture search network, the output feature of the first downsampling unit, the output feature of the second downsampling unit, the twenty-third feature, the twenty-fourth feature, and the fifteenth feature, respectively; The tenth feature is added to the thirteenth feature to obtain the sixteenth feature, the eleventh feature is added to the fourteenth feature to obtain the seventeenth feature, and the twelfth feature is added to the fifteenth feature to obtain the eighteenth feature. The sixteenth feature, the seventeenth feature, and the eighteenth feature are all passed through the fully connected layer and the activation layer in sequence to obtain the nineteenth feature, the twentieth feature, and the twenty-first feature corresponding to the sixteenth feature, the seventeenth feature, and the eighteenth feature in sequence, respectively. The twenty-first feature is added to the twentieth feature after passing through the upsampling layer to obtain a first addition result, and then the first addition result is multiplied by a preset value to obtain the twenty-second feature. The twenty-second feature is added to the nineteenth feature after passing through the upsampling layer to obtain a second addition result, and then the second addition result is multiplied by the preset value to obtain the output of the internal and external fingerprint extraction network, and the output of the internal and external fingerprint extraction network is the stratum corneum contour and the active epidermal connection layer contour, wherein the stratum corneum contour includes the stratum corneum upper surface contour and the stratum corneum lower surface contour, and each pixel point of each contour has a depth value.

5. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 4, characterized in that: The multi-head attention mechanism group includes four multi-head attention mechanism modules connected in sequence, and each of the multi-head attention mechanism modules includes a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a linear layer connected in sequence, wherein the input of the multi-head attention mechanism module is added to the output of the multi-head attention mechanism layer to obtain an added feature, and the added feature is used as the input of the second normalization layer, and the output of the linear layer is added to the added feature to obtain the output of the multi-head attention mechanism module.

6. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 4, characterized in that: During the preprocessing process, the OCT images were intercepted using a preset rectangular frame to capture the stratum corneum region and the viable epidermal junction layer region; During the training process, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum, and the contour of the living epidermal junction layer of the OCT image are annotated, as well as the area of ​​the stratum corneum and the area of ​​the living epidermal junction layer; After each round of training, the fourth feature in the neural architecture search network is passed through the third convolutional layer and the upsampling layer to obtain a predicted image of the area including the stratum corneum and the area of ​​the living epidermal connection layer; Loss function sampled during training The calculation formula is as follows: ； in, ； ； in, represents the calculated loss of the stratum corneum area and the area of ​​the living epidermal junction layer, express The weight of represents the loss calculated from the upper stratum corneum surface contour, the lower stratum corneum surface contour, and the viable epidermal junction layer contour, express The weight of Indicates all the pixels on an OCT image in the fingerprint OCT image dataset. represents the area of ​​the stratum corneum, the area of ​​the living epidermal junction layer, or the background area on the predicted image, and When is the area of ​​the stratum corneum, When represents the area of ​​the living epidermal junction layer, When , it represents the background area. express On the pixels, and Do not indicate the type In the region The predicted pixel value and the true pixel value of each pixel point are Indicates the number of pixels on the predicted stratum corneum upper surface contour, predicted stratum corneum lower surface contour, or predicted living epidermal junction layer contour. indicates a predicted upper stratum corneum surface contour, a predicted lower stratum corneum surface contour, or a predicted live epidermal junctional layer contour, and When , it indicates the predicted surface contour of the stratum corneum. When , it indicates the predicted surface contour under the stratum corneum. When is the predicted contour of the living epidermal junction layer, and Respectively expressed in the type On the outline of The predicted contour depth value and the actual contour depth value of each pixel.

7. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 4, characterized in that: In the process of calculating the inner and outer fingerprints corresponding to the OCT image to be detected based on the contours of the multiple stratum corneum and the living epidermis connection layer, the inner fingerprint includes a deep inner fingerprint and a grayscale inner fingerprint, and the outer fingerprint includes a deep outer fingerprint and a grayscale outer fingerprint; The OCT image of the fingerprint to be detected is Each OCT image has a size of , Indicates width, After each OCT image is trained through the internal and external fingerprint extraction network, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal connection layer are obtained. Each pixel point of each contour has a depth value, and the width of each contour is , that is, there are Depth values ​​are obtained and each contour is mapped to the corresponding OCT image of the fingerprint to be detected; For each OCT image of the fingerprint to be detected, the upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum, and the contour of the living epidermis-epidermal junction layer are correspondingly subtracted from each other the depth values ​​of the pixel points at the corresponding positions of the lower surface contour of the stratum corneum and the upper surface contour to obtain a first result, and then the depth values ​​of the pixel points at the corresponding positions of the living epidermis-epidermal junction layer contour and the upper surface contour of the stratum corneum are correspondingly subtracted to obtain a second result; for The upper surface contour of the stratum corneum, the lower surface contour of the stratum corneum and the contour of the living epidermal junction layer corresponding to the OCT image of the fingerprint to be detected will be The first results are stitched together and saved as an image to obtain the deep external fingerprint. The second results are stitched together and saved as an image to obtain the deep inner fingerprint; For each mapped stratum corneum upper surface contour on the OCT image of the fingerprint to be detected, the grayscale values ​​of all pixels on the stratum corneum upper surface contour are obtained, and the grayscale values ​​of all pixels are added to the average first result of the preset multiple to obtain a third result, where the average first result is The average of the first results; For the mapped The OCT image of the fingerprint to be detected is obtained A third result, and The three results are stitched together and saved as an image to obtain the grayscale external fingerprint; For the contour of the live epidermis-epidermal connection layer on each mapped OCT image of the fingerprint to be detected, obtain the peak position pixel point and the trough position pixel point closest to each pixel point on the contour of the live epidermis-epidermal connection layer, take the average of the depth values ​​corresponding to the peak position pixel point and the trough position pixel point, and obtain the grayscale value of the pixel point where the average value is located, referred to as the first grayscale value. The first grayscale values ​​corresponding to all pixels on the contour of the live epidermis-epidermal connection layer on each mapped OCT image of the fingerprint to be detected are used as the fourth result; For the mapped The OCT image of the fingerprint to be detected is obtained The fourth result will be The fourth result is spliced ​​and saved as an image to obtain the grayscale inner fingerprint.

8. The OCT internal and external fingerprint reconstruction method based on neural network structure search according to claim 2, characterized in that: The upsampling convolution includes ordinary convolution and upsampling operations connected in sequence, the upsampling dynamic convolution includes dynamic convolution and upsampling operations connected in sequence, the upsampling depth-separable convolution includes depth-separable convolution and upsampling operations connected in sequence, the upsampling hole convolution includes hole convolution and upsampling operations connected in sequence, the downsampling convolution includes ordinary convolution and downsampling operations connected in sequence, the downsampling dynamic convolution includes dynamic convolution and downsampling operations connected in sequence, the downsampling hole convolution includes hole convolution and downsampling operations connected in sequence, and the downsampling depth-separable convolution includes depth-separable convolution and downsampling operations connected in sequence.

Images