Bioinformation recognition method and device based on deep network multimodal information fusion

Abstract

The biometric information recognition method and device based on deep network multimodal information fusion provided in the embodiments of the present application relate to the field of image recognition technology. A biometric information recognition method based on deep network multimodal information fusion includes: first, obtaining multiple modal images to be recognized, wherein the multiple modal images to be processed belong to the same biometric information; second, fusing the multiple modal images to be recognized through a preset recognition model to obtain fusion features; then, performing classification processing according to the fusion features to obtain the category of biometric information, wherein the preset recognition model is trained based on biometric information images of multiple modalities. Through the above method, classification and recognition can be realized according to the features after the fusion of multiple modal images, which improves the problem of low reliability of image recognition caused by the fact that the neural network model in the prior art generally recognizes images of multiple modalities separately.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and more specifically, to a method and device for biometric information recognition based on deep network multimodal information fusion. Background Art

[0002] The image recognition method that obtains the category to which the image belongs by manually identifying and processing the image has the problem of low image recognition efficiency. In the field of bioinformatics recognition technology, in order to improve the efficiency of bioinformatics image recognition, images can be recognized through neural network models, but there are still challenges in the multi-modal image recognition process. Images in multiple modalities have large intra-class morphological differences, making classification more difficult. The neural network models in the prior art generally recognize images of multiple modalities separately, resulting in the problem of low image recognition reliability. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a biological information recognition method and device based on deep network multimodal information fusion to improve the problems existing in the prior art.

[0004] To achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0005] In a first aspect, the present invention provides a biological information recognition method based on deep network multimodal information fusion, comprising:

[0006] Acquire multiple modal images to be recognized, wherein the multiple modal images to be processed belong to the same biological information;

[0007] Performing fusion processing on the to-be-recognized images of the multiple modalities through a preset recognition model to obtain fusion features;

[0008] Classification processing is performed according to the fusion features to obtain the category of the biological information, wherein the preset recognition model is obtained by training biological information images of multiple modalities.

[0009] In an optional embodiment, the step of fusing the images to be recognized of the multiple modalities by using a preset recognition model to obtain a fusion feature includes:

[0010] Performing feature processing on the to-be-recognized images of each modality to obtain learning features of each to-be-recognized image;

[0011] The learning features of each of the images to be processed are spliced ​​to obtain fused features.

[0012] In an optional embodiment, the preset recognition model includes a projection matrix corresponding to the image to be recognized of each modality, and the step of performing feature processing on the image to be recognized of each modality to obtain a learning feature of each image to be recognized includes:

[0013] For each modality of the image to be identified, perform depth recognition processing on the image to be identified to obtain the depth feature of the image to be processed;

[0014] The learning features of the image to be processed are calculated based on the depth features of the image to be processed and the corresponding projection matrix.

[0015] In an optional embodiment, the preset recognition model includes a conca layer, and the step of performing splicing processing on the learning features of each of the images to be processed to obtain a fusion feature includes:

[0016] The learning features of each of the images to be processed are spliced ​​through the conca layer to obtain fused features.

[0017] In an optional embodiment, the biological information recognition method based on deep network multimodal information fusion also includes a step of training a preset recognition model, which includes:

[0018] The total loss function is calculated based on the optimization loss function and classification loss function of the initial model;

[0019] The initial model is trained according to the total loss function to obtain a preset recognition model.

[0020] In an optional implementation, the biological information recognition method based on deep network multimodal information fusion further includes:

[0021] Obtain user feedback data;

[0022] The preset recognition model is updated according to the feedback data.

[0023] In a second aspect, the present invention provides a biological information recognition device based on deep network multimodal information fusion, comprising:

[0024] An image acquisition module, used for acquiring multiple modal images to be recognized, wherein the multiple modal images to be processed belong to the same biological information;

[0025] An image processing module, used for fusing the images to be recognized of the multiple modalities through a preset recognition model to obtain fusion features;

[0026] A classification module is used to perform classification processing according to the fusion features to obtain the category of the biological information, wherein the preset recognition model is obtained by training biological information images of multiple modalities.

[0027] In an optional implementation manner, the image processing module is specifically used to:

[0028] Performing feature processing on the to-be-recognized images of each modality to obtain learning features of each to-be-recognized image;

[0029] The learning features of each of the images to be processed are spliced ​​to obtain fused features.

[0030] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the processor is used to execute an executable computer program stored in the memory to implement the biological information recognition method based on deep network multimodal information fusion as described in any one of the aforementioned embodiments.

[0031] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed, implements the steps of the biological information recognition method based on deep network multimodal information fusion as described in any one of the aforementioned embodiments.

[0032] The embodiments of the present application provide a method and device for biometric information recognition based on deep network multimodal information fusion. Through a preset recognition model, images to be recognized of multiple modalities are fused to obtain fused features, and classification processing is performed based on the fused features. Classification and recognition are achieved based on the features after fusion of multiple modal images, thereby improving the problem of low reliability of image recognition caused by the fact that neural network models in the prior art generally recognize images of multiple modalities separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0035] Figure 2 A flowchart of a biometric information recognition method based on deep network multimodal information fusion provided in an embodiment of the present application.

[0036] Figure 3Another flowchart of the biometric information recognition method based on deep network multimodal information fusion provided in an embodiment of the present application.

[0037] Figure 4 A structural block diagram of the preset recognition model provided in an embodiment of the present application.

[0038] Figure 5 Another flowchart of the biometric information recognition method based on deep network multimodal information fusion provided in an embodiment of the present application.

[0039] Figure 6 Another flowchart of the biometric information recognition method based on deep network multimodal information fusion provided in an embodiment of the present application.

[0040] Figure 7 Another flowchart of the biometric information recognition method based on deep network multimodal information fusion provided in an embodiment of the present application.

[0041] Figure 8 The result diagram of the experimental verification provided in the embodiment of the present application.

[0042] Fig. 9 Another result diagram of the experimental verification provided in the embodiment of the present application.

[0043] Fig.10 This is a structural block diagram of a biometric information recognition device based on deep network multimodal information fusion provided in an embodiment of the present application.

[0044] Icon: 100 - electronic device; 110 - network port; 120 - first processor; 130 - communication bus; 140 - first storage medium; 150 - interface; 1000 - biometric information recognition device; 1010 - image acquisition module; 1020 - image processing module; 1030 - classification module. DETAILED DESCRIPTION

[0045] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this application for the above problems should be the contributions made by the inventor to this application during the application process.

[0046] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0047] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0048] In order to enable those skilled in the art to use the content of the present application, the following implementation methods are provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of the present application. The application of the system or method of the present application may include a web page, a browser plug-in, a client terminal, a customization system, an internal analysis system, or an artificial intelligence robot, etc., or any combination thereof.

[0049] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0051] Figure 1A schematic diagram of exemplary hardware and software components of an electronic device 100 that can implement the ideas of the present application according to some embodiments of the present application is shown. The electronic device 100 may include a network port 110 connected to a network, one or more first processors 120 for executing program instructions, a communication bus 130, and a first storage medium 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the electronic device 100 may also include program instructions stored in ROM, RAM, or other types of non-temporary storage media or any combination thereof, according to which the method of the present application can be implemented. The electronic device 100 may also include an input / output (I / O) interface 150 with other input / output devices (e.g., a keyboard, a display screen).

[0052] In some embodiments, the first processor 120 may process information and / or data related to the embodiments of the present application to perform one or more functions described in the present application. In some embodiments, the first processor 120 may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). By way of example only, the first processor 120 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC) or a microprocessor, or any combination thereof.

[0053] The first processor 120 in the electronic device 100 can be a general-purpose computer or a computer for a specific purpose, both of which can be used to implement the method of the present application. Although only one computer is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0054] For ease of description, only one processor is described in the electronic device 100. However, it should be noted that the electronic device 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the electronic device 100 performs step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor perform steps A and B together.

[0055] The network can be used for the exchange of information and / or data. In some embodiments, one or more components in the electronic device 100 can send information and / or data to other components. For example, the electronic device 100 can obtain a signal via a network. As an example only, the network can include a wireless network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof.

[0056] In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and / or a network switching node, through which one or more components of the electronic device 100 may connect to the network to exchange data and / or information.

[0057] Figure 2 A flowchart of a biological information recognition method based on deep network multimodal information fusion provided in an embodiment of the present application is shown, which can be applied to Figure 1 The electronic device 100 shown comprises Figure 1 It should be understood that in other embodiments, the order of some steps in the method for biometric information recognition based on deep network multimodal information fusion of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. Figure 2 The process of the biological information recognition method based on deep network multimodal information fusion is described in detail.

[0058] Step S210: acquiring images to be recognized in multiple modalities.

[0059] Among them, the images to be processed of multiple modalities belong to the same biological information.

[0060] Step S220: fusing the images to be recognized of multiple modalities through a preset recognition model to obtain fusion features.

[0061] Step S230 , performing classification processing according to the fused features to obtain the category of biological information.

[0062] Among them, the preset recognition model is trained based on bio-information images of multiple modalities.

[0063] The above method fuses the images to be identified of multiple modalities through a preset recognition model to obtain fused features, and performs classification processing based on the fused features, thereby realizing classification and recognition based on the features after the fusion of multiple modal images, and improving the problem of low reliability of image recognition caused by the neural network model in the prior art generally identifying images of multiple modalities separately.

[0064] For step S210, it should be noted that the specific method of acquiring the image to be identified is not limited and can be set according to actual application requirements. For example, in an alternative example, when the electronic device 100 has a module with an image acquisition function, the electronic device 100 can directly acquire multiple modal images to be identified. For another example, in another alternative example, when the electronic device 100 does not have a module with an image acquisition function, multiple modal images to be identified can be acquired by an image acquisition device connected to the electronic device 100.

[0065] The biological information may be information related to cancer lesions, specifically information related to hepatocellular carcinoma. The image to be identified may be a 3D image or a 2D image. The shape of the 3D image is 16*16*16, and the shape of the 2D image is 28*28. After the image to be identified is obtained, the image to be identified may be preprocessed to adjust the shape dimension of the image to adapt to the unified input of the network.

[0066] It should be noted that, in order to obtain a preset recognition model, before step S210, the biological information recognition method based on deep network multimodal information fusion provided in the embodiment of the present application may also include a step of training the preset recognition model. Figure 2 On the basis of Figure 3 A flow chart of another biometric information recognition method provided in the embodiment of the present application is shown in FIG. Figure 3 , the biometric information recognition method may further include:

[0067] Step S240, calculating a total loss function based on the optimization loss function and the classification loss function of the initial model.

[0068] Step S250, training the initial model according to the total loss function to obtain a preset recognition model.

[0069] Regarding step S240, it should be noted that, in combination with Figure 4 The preset recognition model (deep supervision network, DSN) provided in the embodiment of the present application includes three convolutional layers (conv1, conv2 and conv3), two fully connected layers, projection matrices (Wpp, Wap and Wpvp) corresponding to the images to be recognized of each modality, a conca layer and a softmax layer. Each modality has a CNN network for deep feature learning, that is, Figure 2 A single lateral model in front of the conca layer.

[0070] Among them, the purpose of the embodiment of the present application is to provide a biological information identification method based on deep network multimodal information fusion, including the following steps: introducing a discriminant intra-class loss function to reduce the distance of features in the same category and expand the distance of features in different categories of tumors; designing an adaptive weighting strategy based on the loss values ​​of different modalities in the optimization iteration to increase the contribution of modalities with relatively low loss values ​​and reduce the impact of modalities with larger loss values ​​on the final loss function; proposing a total loss function for the proposed deep learning framework, which combines the two proposed parts to better characterize the lesions.

[0071] In detail, the design process steps for introducing the discriminative intra-class loss function are as follows:

[0072] The image blocks of the three modal images to be identified (pre-contrast, arterial and portal venous phases) are placed in the CNN network respectively, and the deep features (Fpp, Fap and Fpvp) of each phase are generated from the second fully connected layer. The intra-modality discrimination module is introduced to reduce the distance of features of the same category and increase the distance between features of different categories. Then, three projection matrices (Wpp, Wap and Wpvp) are learned to project the deep features to the new feature representation, and the correlation of samples of the same category is maximized by reducing the distance of features of samples of the same category.

[0073] In addition, an adaptive deep supervision network module suitable for multi-modal fusion can be designed to send the learned features to the softmax layer for classification and connect to the softmax layer at the same time. In the steps of deep supervision network fusion and adaptive weight adjustment, direct supervision of multi-modal feature extraction provides a good adjustment strategy and specification for improving classification performance.

[0074] In detail, the process steps of deep supervision network fusion using adaptive weighted adjustment are as follows:

[0075] The classification loss function for classifying the original DSN can be shown as follows:

[0076] Lclassify=Lcon+λ1·Lpp+λ2·Lap+λ3·Lpvp;

[0077] Among them, λ1, λ2, and λ3 are weights, Lcon represents the loss value after the fusion of the three modes, and Lpp, Lap, and Lpvp represent the loss value of a single mode. The weights of different modes can be adjusted adaptively:

[0078]

[0079] Among them, β represents the positive focus parameter, which is a customizable hyperparameter. When it is 2, the parameter is proportional to the classification performance of networks m and n, and the classification loss of m and n is focused on the transfer of positive feature knowledge. Assume that "Lm" and "Ln" are the classification losses of networks m and n, corresponding to the mth and nth modes respectively. Let Δ = Lm-Ln, and a positive Δ indicates that the effect of network n is better than that of network m.

[0080] Furthermore, the total loss function Ltotal of the end-to-end multi-modal fusion framework proposed in the embodiment of the present application combines the two proposed parts to better characterize the lesions. The specific process steps are as follows:

[0081] Ltotal=LD+Lclassify, LD is an optimization function that can maximize the intra-modal correlation of samples of the same category and the intra-modal correlation of samples of different categories. Lclassify is a deep supervision loss function with adaptive weights based on cross entropy loss, which is used for classification.

[0082] Among them, the specific steps of generating LD can be as follows:

[0083] The intra-modality discrimination module can design a calculation matrix W to project the deep features of each phase F into space. If they are from the same class, the distance between the samples xi and xj is reduced, otherwise the distance is increased. The constraints are defined as follows: If two tumors are from the same category (yij=1), their relative deep feature distance should be less than a predefined threshold μ-τ, otherwise yij=-1, and the deep feature distance between the two tumors should be greater than μ+τ. xi, xj represent two image samples of the same modality and the same data batch, yij represents the labels of the two samples, if the two labels are the same yij=1, otherwise yij=-1. μ=10, τ=1, μ, τ are two empirical parameters.

[0084] y ij(μ-dW(x i , x j ))>τ (1);

[0085] The distance dW(xi,xj) between a pair of deep features xi and xj is calculated as:

[0086]

[0087] Among them, Wxi represents the projection matrix of xi sample, and Wxj represents the projection matrix of xj sample.

[0088] The intra-modal discriminant function is defined as follows:

[0089]

[0090] Among them, the formula of h() is h(x) = max(0, x), and the corresponding three projection matrices Wpp, Wap and Wpvp can be used to construct the discriminant items of the three stages, namely Dpp, Dap and Dpvp. In order to learn features from these three stages, we construct the objective function of the intra-modal discriminant module as follows:

[0091]

[0092] Among them, α represents the coordination parameter between each mode. The introduction of the hyperparameter k makes α1, α2 and α3 nonlinear, avoiding suboptimal solutions. In the current work, k=2, and several k values ​​are tested in this method. When k=2, the AUC value is higher. If k>3, the network training is suboptimal and may be difficult to converge.

[0093] For step S250, it should be noted that after obtaining the total loss function, the "Tensorflow" algorithm can be used to implement multimodal feature fusion, and the SGD algorithm can be used to minimize the objective function. Tensorflow is just a framework for model implementation, and the SGD algorithm is a stochastic gradient descent algorithm, which is used to minimize the objective function and is an important step in training the model.

[0094] Regarding step S220, it should be noted that the specific method of performing the fusion processing is not limited and can be set according to actual application requirements. For example, in an alternative example, step S220 may include a step of obtaining a fusion feature based on the learning feature. Figure 2 On the basis of Figure 5 A flow chart of another biometric information recognition method provided in the embodiment of the present application is shown in FIG. Figure 5 , step S220 may include:

[0095] Step S221, performing feature processing on the to-be-recognized images of each modality to obtain learning features of each to-be-recognized image.

[0096] Step S222, concatenating the learning features of each image to be processed to obtain a fused feature.

[0097] Regarding step S221, it should be noted that the specific method of performing feature processing is not limited and can be set according to actual application requirements. For example, in an alternative example, the preset recognition model includes a projection matrix corresponding to the image to be recognized in each modality, and step S221 may include a step of calculating the learning feature according to the projection matrix. Figure 5 On the basis of Figure 6 A flow chart of another biometric information recognition method provided in the embodiment of the present application is shown in FIG. Figure 6 , step S221 may include:

[0098] Step S2211: for each modality of the image to be identified, perform depth recognition processing on the image to be identified to obtain the depth feature of the image to be processed.

[0099] Step S2212: Calculate the learning features of the image to be processed based on the depth features of the image to be processed and the corresponding projection matrix.

[0100] Among them, the projection matrix W is learnable, and the deep feature F obtained by the CNN network is multiplied by the learnable W to obtain the learning feature.

[0101] Regarding step S222, it should be noted that the specific method of performing the splicing process is not limited and can be set according to actual application requirements. For example, in an alternative example, the preset recognition model includes a conca layer, and step S222 may include the following sub-steps:

[0102] The learning features of each image to be processed are concatenated through the conca layer to obtain the fused features.

[0103] That is to say, after the conca layer concatenates the feature matrices of the three learning features, it is sent to the softmax layer for classification.

[0104] After step S230, the biometric information recognition method provided in the embodiment of the present application may further include the step of updating the preset recognition model. Figure 2 On the basis of Figure 7 A flow chart of another biometric information recognition method provided in the embodiment of the present application is shown in FIG. Figure 7 , the biometric information recognition method may further include:

[0105] Step S260, obtaining user feedback data.

[0106] Step S270: updating the preset recognition model according to the feedback data.

[0107] The images are labeled (high and low), and the histological grading of HCC is low-grade for stages I and II, and high-grade for stages III and IV. During testing, the labels predicted by the model are compared with the original labels of the images to obtain the accuracy, sensitivity, specificity, AUC value, etc. of the high and low grade classification of tumors (see the table below). The fusion method of discriminant learning and adaptive weights under the deep learning framework solves the problems of the two multimodal fusions mentioned above, and improves the model performance indicators.

[0108] That is to say, the superiority of the method provided by the embodiment of the present application can be verified by comparison with the current multi-mode fusion method and a large number of experiments. The process steps of the experimental verification are as follows.

[0109] 1) Conduct two fusion experiments: multi-phase fusion based on 3D CNN and multi-view fusion based on 2D CNN;

[0110] 2) Compare typical fusion methods;

[0111] 3) Combine the two proposed modules with typical fusion methods to verify the effectiveness of the two modules.

[0112] In detail, 117 patients with histologically confirmed HCC could be included, and the dataset was randomly divided into a training set (77 HCCs) and a test set (40 HCCs). Training and testing were repeated five times, and the values ​​of accuracy, sensitivity, colorimetry, and area under the curve (AUC) were calculated.

[0113] Fig. 9 The result graph of the multi-phase fusion performance based on 3D CNN shows the performance comparison (%) of different methods of three-dimensional multi-phase fusion for 117 HCCs. Figure 8 The result diagram of the performance of 2D CNN multi-view fusion compares the effects of different methods on 2D multi-view fusion of 117 HCC arterial phase. After obtaining the experimental verification result diagram, each module can be studied in detail: The result diagram shows that the method provided in the embodiment of the present application has good performance indicators, further proving the effectiveness of the proposed discriminative modal internal terms and adaptive weighting adjustment strategy in HCC grading, and verifying the effectiveness of the proposed adaptive weighting strategy in multi-phase fusion based on 3D CNN.

[0114] It should be noted that when the performance indicators of the method provided in the embodiment of the present application fail to meet the requirements, user feedback data can be obtained, and the preset recognition model can be updated according to the feedback data until the performance indicators meet the requirements.

[0115] Combination Fig.10 , the embodiment of the present application also provides a biometric information recognition device 1000 based on deep network multimodal information fusion, and the functions implemented by the biometric information recognition device 1000 correspond to the steps performed by the above method. The biometric information recognition device 1000 can be understood as the processor of the above electronic device 100, or as a component independent of the above electronic device 100 or the processor that implements the functions of the present application under the control of the electronic device 100. Among them, the biometric information recognition device 1000 may include an image acquisition module 1010, an image processing module 1020 and a classification module 1030.

[0116] The image acquisition module 1010 is used to acquire multiple modal images to be recognized, wherein the multiple modal images to be processed belong to the same biological information. In the embodiment of the present application, the image acquisition module 1010 can be used to execute Figure 2 As shown in step S210, the relevant contents about the image acquisition module 1010 can refer to the above description of step S210.

[0117] The image processing module 1020 is used to perform fusion processing on the images to be recognized of multiple modalities through a preset recognition model to obtain fusion features. In the embodiment of the present application, the image processing module 1020 can be used to perform Figure 2 As shown in step S220, the relevant contents about the image processing module 1020 can refer to the above description of step S220.

[0118] The classification module 1030 is used to perform classification processing according to the fusion features to obtain the category of biological information, wherein the preset recognition model is obtained by training biological information images of multiple modalities. In the embodiment of the present application, the classification module 1030 can be used to perform Figure 2 As shown in step S230, the relevant contents of the classification module 1030 can refer to the above description of step S230.

[0119] Furthermore, the image processing module 1020 may also be specifically used for:

[0120] Perform feature processing on the images to be identified in each modality to obtain learning features of each image to be identified;

[0121] The learning features of each image to be processed are spliced ​​to obtain fused features.

[0122] In addition, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned biometric information recognition method are executed.

[0123] The computer program product of the biometric information identification method provided in the embodiment of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the biometric information identification method in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0124] To summarize, the biometric information recognition method and device based on deep network multimodal information fusion provided in the embodiments of the present application fuse multiple modal images to be recognized through a preset recognition model to obtain fusion features, and perform classification processing based on the fusion features, thereby realizing classification and recognition based on the features after fusion of multiple modal images, thereby improving the problem of low reliability of image recognition caused by the fact that the neural network model in the prior art generally recognizes images of multiple modalities separately.

[0125] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0127] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0128] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0129] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A biological information recognition method based on deep network multimodal information fusion, characterized in that: include: Acquire multiple modal images to be recognized, wherein the multiple modal images to be processed belong to the same biological information; Performing fusion processing on the to-be-recognized images of the multiple modalities through a preset recognition model to obtain fusion features; Performing classification processing according to the fusion features to obtain the category of the biological information, wherein the preset recognition model is obtained by training biological information images of multiple modalities; The biological information recognition method based on deep network multimodal information fusion also includes the step of training a preset recognition model, which includes: The total loss function is calculated based on the optimization loss function and classification loss function of the initial model; Training the initial model according to the total loss function to obtain a preset recognition model; The total loss function includes: Ltotal = LD +Lclassify, wherein Ltotal represents the total loss function, LD represents the optimization loss function, and Lclassify represents the classification loss function; the optimization loss function is used to maximize the correlation within the same sample mode; the classification loss function is a deep supervision loss function that adaptively adjusts the weights of different modes based on the cross entropy loss; The multimodality is three modes; Lclassify =Lcon+λ1·Lpp + λ2·Lap + λ3·Lpvp; Among them, λ1, λ2, and λ3 are weights, Lcon represents the loss value after the fusion of the three modes, and Lpp, Lap, and Lpvp represent the loss values ​​of a single mode; The preset recognition model includes three convolutional layers, two fully connected layers, a projection matrix corresponding to the image to be recognized of each modality, a conca layer and a softmax layer, and each modality has a CNN network for deep feature learning; The step of fusing the images to be recognized of the multiple modalities by using a preset recognition model to obtain a fusion feature comprises: The image blocks of the images to be recognized of the three modalities are respectively placed into the CNN network, and the deep features of each phase are generated from the second fully connected layer; The learning features of the image to be processed are calculated according to the depth features of the image to be processed and the corresponding projection matrix, so as to maximize the correlation of samples of the same category by reducing the distance between the features of samples of the same category; The learning features of each of the images to be processed are spliced ​​through the conca layer to obtain fused features.

2. The biological information recognition method based on deep network multimodal information fusion as claimed in claim 1, characterized in that: The biological information recognition method based on deep network multimodal information fusion also includes: Obtain user feedback data; The preset recognition model is updated according to the feedback data.

3. A biological information recognition device based on deep network multimodal information fusion, characterized in that: include: An image acquisition module, used for acquiring multiple modal images to be recognized, wherein the multiple modal images to be processed belong to the same biological information; An image processing module, used for fusing the images to be recognized of the multiple modalities through a preset recognition model to obtain fusion features; A classification module, used for performing classification processing according to the fusion features to obtain the category of the biological information, wherein the preset recognition model is obtained by training biological information images of multiple modalities; The biological information recognition device also includes a biological information recognition method based on deep network multimodal information fusion and a module for training a preset recognition model, which is used to: The total loss function is calculated based on the optimization loss function and classification loss function of the initial model; Training the initial model according to the total loss function to obtain a preset recognition model; The total loss function includes: Ltotal = LD +Lclassify, Ltotal represents the total loss function, LD represents the optimization loss function, and Lclassify represents the classification loss function; the optimization loss function is used to maximize the correlation within the same sample mode; the classification loss function is a deep supervision loss function that adaptively adjusts the weights of different modes based on cross entropy loss; The multimodality is three modes; Lclassify =Lcon+λ1·Lpp + λ2·Lap + λ3·Lpvp; Among them, λ1, λ2, and λ3 are weights, Lcon represents the loss value after the fusion of the three modes, and Lpp, Lap, and Lpvp represent the loss values ​​of a single mode; The preset recognition model includes three convolutional layers, two fully connected layers, a projection matrix corresponding to the image to be recognized of each modality, a conca layer and a softmax layer, and each modality has a CNN network for deep feature learning; The image processing module is further used for: The image blocks of the images to be recognized of the three modalities are respectively placed into the CNN network, and the deep features of each phase are generated from the second fully connected layer; The learning features of the image to be processed are calculated according to the depth features of the image to be processed and the corresponding projection matrix, so as to maximize the correlation of samples of the same category by reducing the distance between the features of samples of the same category; The learning features of each of the images to be processed are spliced ​​through the conca layer to obtain fused features.

4. An electronic device, characterized in that: It includes a memory and a processor, wherein the processor is used to execute an executable computer program stored in the memory to implement the biological information recognition method based on deep network multimodal information fusion as described in claim 1 or 2.

5. A storage medium, characterized in that: A computer program is stored thereon, which, when executed, implements the steps of the biological information recognition method based on deep network multimodal information fusion as described in claim 1 or 2.

Images