A medical diagnosis method and device based on cross-modal semantic hashing

Abstract

The present application belongs to the technical field of computer-aided disease diagnosis and treatment, and discloses a medical diagnosis method and device based on cross-modal semantic hash retrieval, comprising: using an artificial neural network to hash encode category annotation information to form a semantic similarity preserving classifier; using the semantic similarity preserving classifier to guide the training process of multiple modal artificial neural network encoders; using the trained encoder to hash encode different modal medical data; and completing medical diagnosis through cross-modal semantic hash retrieval. The present application has the following advantages in medical diagnosis: low storage and computing resource consumption; good semantic representation ability, i.e. the hash encoding of samples can simultaneously represent the inter-class semantic relationship of samples and maintain intra-class mutual aggregation; and the ability to simultaneously process single-modal medical data-based and multi-modal medical data-based medical diagnosis tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided disease diagnosis and treatment technology, and particularly relates to a medical diagnosis method and device based on cross-modal semantic hashing. Background Technology

[0002] Computer-aided disease diagnosis aims to automatically identify the disease type of medical data through computer processing. Existing computer-aided diagnosis technologies can be divided into classifier-based and image retrieval-based systems. While these two approaches have greatly promoted the development of computer-aided disease diagnosis systems, existing systems neglect the diverse modalities of current medical data. Examples include the papers "Graph temporal ensembling based semi-supervised convolutional neural network with noisy labels for histopathology image analysis" (Medical Image Analysis (2020)) and "Multi-label transfer learning for the early diagnosis of breast cancer" (Neurocomputing (2020)). Therefore, this invention uses a cross-modal semantic hashing method to build a computer-aided disease diagnosis system based on cross-modal semantic hashing.

[0003] The main challenges in building computer-aided disease diagnosis systems using cross-modal semantic hashing methods lie in the difficulty of encoding disease semantics and the lack of medical data for some modalities. These two factors lead to the following problems with current cross-modal semantic hashing methods in current computer-aided disease diagnosis applications: existing methods generally encode similarity relationships between instances, failing to simultaneously handle semantic relationships between disease categories and clustering relationships within disease categories; existing methods generally require mining inter-modal relationships in the data, making them unable to handle situations where medical data for some modalities is missing. Therefore, existing cross-modal semantic hashing methods cannot be directly used to build computer-aided disease diagnosis systems. For example, the papers (Triplet-based deep hashing network for cross-modal retrieval, IEEE Transactions on Image Processing, 2018) and (Multi-task consistency-preserving adversarial hashing for cross-modal retrieval, IEEE Transactions on Image Processing, 2020) illustrate this.

[0004] To address this issue, this invention utilizes artificial neural networks to hash-encode category labeling information to form a semantic similarity-preserving classifier. This cross-modal semantic hashing method solves the problem of difficulty in encoding disease semantics and the lack of medical data in some modalities during the construction of computer-aided disease diagnosis systems. Furthermore, the hash encoding obtained by this invention has advantages such as lower storage and computational resource consumption during retrieval (on a Windows 10 64-bit operating system, 10 million 128-bit hash codes only require 160MB of storage space, and calculating 10 million Hamming distances for each 128-bit hash code takes only about 20 seconds), thus meeting the needs of practical applications.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0006] (1) Existing computer-aided diagnostic technologies can be divided into classifier-based computer-aided disease diagnosis systems and image retrieval-based computer-aided disease diagnosis systems, which ignore the diverse modalities of current medical data.

[0007] (2) At present, no computer-aided disease diagnosis system based on cross-modal semantic hashing has been built using cross-modal semantic hashing method.

[0008] (3) Using cross-modal semantic hashing to build a computer-aided disease diagnosis system has the problems of difficulty in encoding disease semantics and lack of medical data in some modalities. Summary of the Invention

[0009] To address the problems existing in the prior art, the present invention provides a medical diagnostic method and apparatus based on cross-modal semantic hashing.

[0010] This invention is implemented as follows: A medical diagnostic method based on cross-modal semantic hashing includes:

[0011] Step 1: Use an artificial neural network to hash-encode the category labeling information to form a semantic similarity-preserving classifier;

[0012] Step 2: Use the semantic similarity preservation classifier to guide the training of the VGG-16 network, which serves as the encoder for X-ray images, and the fully connected neural network encoder corresponding to the physical examination report.

[0013] Step 3: Use the trained modal encoder to perform hash encoding on the X-ray images and medical examination reports;

[0014] Step four: Complete the medical diagnosis through cross-modal semantic hash retrieval.

[0015] Furthermore, the cross-modal semantic hashing medical diagnosis method also includes preprocessing the input X-ray images and physical examination reports to construct category labeling information.

[0016] Furthermore, the preprocessing includes adjusting the size and number of channels of the X-ray image to 224×224×3 and subtracting the mean.

[0017] Furthermore, the preprocessing also includes extracting the bag-of-words model vector of the physical examination report using the bag-of-words model, and representing the diagnostic results as 0 and 1 vectors to form category labels.

[0018] Furthermore, the encoder training includes learning a semantic similarity-preserving classifier and encoder training based on the semantic similarity-preserving classifier.

[0019] Furthermore, the process of learning the semantic similarity-preserving classifier includes:

[0020] Using LabNet to analyze N3 category labels y l Hash encoding is performed, and the objective function is used to optimize labNet, as shown in the following formula:

[0021] in, Indicates semantic relations. To constrain the output of the LabNet network to be close to binarization, S is the manually labeled data similarity matrix, α is a hyperparameter, and f is a hyperparameter.y (θ y ;y l ) is y l The output after processing by labNet It is f y (θ y ;y l The hash code obtained after processing by the sign(·) function, and the α hash code matrix W are used as a semantic similarity preservation classifier.

[0022] The labNet network is a three-layer fully connected artificial neural network based on category labels, with the number of nodes being the length of the category label (4096) and the encoding bit length (c).

[0023] Furthermore, the encoder training is based on the semantic similarity-preserving classifier using the objective function L. n The parameters θ1 and θ2 in the imgNet and txtNet networks are optimized using the following formulas:

[0024]

[0025] in, For spherical classification loss function, Binarization loss function, It is a normalized vector. w l It is the hash code corresponding to the category label, and β, m and k are hyperparameters.

[0026] Furthermore, the encoder is an imgNet artificial neural network for processing X-ray images and a txtNet artificial neural network for processing medical examination reports;

[0027] The imgNet network is based on X-ray images and is formed by changing the last fully connected layer of the original VGG-16 deep neural network to a fully connected layer with an encoding bit length of c.

[0028] The txtNet network is a three-layer fully connected artificial neural network based on physical examination reports, with the number of nodes being d2 (report feature dimension) and 4096 (encoding bit length) respectively.

[0029] Furthermore, the hash encoding process in step three is as follows:

[0030] First, the imgNet network is used to obtain approximate binary encodings of N1 X-ray images. Using the txtNet network to obtain approximate binary encoding of N2 medical examination reports

[0031] Then, use right and Process to obtain {0, 1} c Discrete hash code and

[0032] Finally, the corresponding hash code is extracted according to the type of medical data to be diagnosed.

[0033] Furthermore, the cross-modal semantic hash retrieval process is as follows:

[0034] Calculate the Hamming distance between the hash code corresponding to the medical data to be diagnosed and the hash codes of X-ray images and physical examination reports of different modalities;

[0035] Based on the Hamming distance, find the data in the medical database that is closest to the medical data to be diagnosed, and use the existing category label of the data itself as the diagnostic result for the medical data to be diagnosed.

[0036] Another object of the present invention is to provide a medical diagnostic device based on cross-modal semantic hashing, comprising: at least one processor; and at least two memories communicatively connected to the at least one processor; wherein one of the memories stores instructions executable by the at least one processor, the instructions being programmed to perform the aforementioned medical diagnostic method based on cross-modal semantic hashing to process medical data stored in the other memory.

[0037] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the medical diagnostic method based on cross-modal semantic hashing.

[0038] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0039] This invention features low storage and computational resource consumption. Compared to representing data as floating-point vectors, this invention reduces storage and computational resource consumption during data retrieval by representing medical data as discrete binary hash codes. Specifically, on a Windows 10 64-bit operating system, 10 million 128-bit hash codes require only 160MB of storage space, and calculating the Hamming distance 10 million times for each 128-bit hash code takes only about 20 seconds.

[0040] This invention possesses excellent semantic representation capabilities. Firstly, it constructs a semantic similarity-preserving classifier that reflects the relationships between disease classes, and uses the hash codes of different modalities of medical data as features easily classified by this classifier. On one hand, the hash codes of the data inherit the semantic similarity relationships between class codes in the semantic similarity-preserving classifier. On the other hand, due to the constraint on the clustering of samples of the same class during the classification process, the hash codes of the data can also be clustered within classes.

[0041] This invention can simultaneously process medical diagnostic tasks based on both unimodal and multimodal medical data. Because the method of this invention treats the training process of each modality encoder as a classification task without the participation of other modal information, it avoids the impact of missing modalities in some medical data. When processing unimodal medical data, this data can be treated as data with missing modalities, and thus processed directly using the method of this invention. When processing multimodal medical data, the medical data of different modalities can be processed separately according to the method of this invention.

[0042] As supporting evidence of the inventiveness of the claims of this invention, the commercial value of the technical solution after its transformation is also reflected: (1) The medical diagnosis method and device based on cross-modal semantic hash retrieval disclosed in this invention can directly perform medical diagnosis based on X-ray photos and physical examination reports with corresponding diagnostic results in the archives, saving the labor cost of hiring professional physicians. (2) The cross-modal semantic hash algorithm designed in this invention can handle the problem of partial modality missing in medical data, without requiring X-ray photos and physical examination reports to appear in pairs, thereby reducing the requirements for data quality and saving data acquisition costs.

[0043] The hash code obtained by this invention has advantages such as lower storage and computing resource consumption during the retrieval process. For example, on a Windows 10 64-bit operating system, 10 million 128-bit hash codes require only 160MB of storage space, and calculating the corresponding Hamming distance 10 million times takes only about 20 seconds. Meanwhile, under the same hardware conditions on a Windows 10 64-bit operating system, 10 million 128-dimensional floating-point vectors require 20GB of storage space, and calculating the corresponding Euclidean distance 10 million times takes about 1.5 hours. Through comparison, it is found that the hash code obtained by the cross-modal semantic hashing algorithm designed in this invention can significantly reduce the requirements for storage and computing hardware, saving hardware purchase costs. The expected benefits and commercial value of the technical solution after the transformation of this invention are as follows:

[0044] The hash code obtained by this invention has advantages such as low storage and computing resource consumption during the retrieval process (on a Windows 10 64-bit operating system, 10 million 128-bit hash codes only require 160MB of storage space, and calculating the Hamming distance 10 million times for a 128-bit hash code only takes about 20 seconds), thus meeting the needs of practical applications.

[0045] The technical solution of this invention solves a long-standing but unresolved technical problem: the main difficulty in building a computer-aided disease diagnosis system using cross-modal semantic hashing lies in the difficulty of encoding disease semantics and the lack of medical data for some modalities. This invention addresses the problem of difficulty in encoding disease semantics and the lack of medical data for some modalities by utilizing artificial neural networks to hash-encode category labeling information to form a semantic similarity-preserving classifier. This invention uses cross-modal semantic hashing to build a computer-aided disease diagnosis system, solving the problems of difficulty in encoding disease semantics and the lack of medical data for some modalities in the construction of existing computer-aided disease diagnosis systems. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the medical diagnosis method based on cross-modal semantic hashing provided in an embodiment of the present invention.

[0047] Figure 2 This is a model structure diagram of the medical diagnosis method based on cross-modal semantic hashing provided in an embodiment of the present invention;

[0048] Figure 3 This is a visual illustration of using a medical database with a different modality from the data to be diagnosed, provided in an embodiment of the present invention.

[0049] Figure 4 This is a visual illustration of a medical database using the same modality as the data to be diagnosed, provided in an embodiment of the present invention.

[0050] Figure 5 This is a schematic diagram of the structure of a medical diagnostic device based on cross-modal semantic hashing provided in an embodiment of the present invention;

[0051] Figure 6 This is the PR curve diagram corresponding to the comparison method provided in the embodiments of the present invention;

[0052] Figure 7 This is a PR curve diagram corresponding to the relevant comparison method provided in the embodiments of the present invention;

[0053] Figure 8This is a PR curve diagram corresponding to the present invention and related comparison methods provided in the embodiments of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] To enable those skilled in the art to fully understand how the present invention is specifically implemented, this section provides an explanatory description of the embodiments that expand upon the technical solutions of the claims.

[0056] like Figure 1 As shown, this embodiment of the invention constructs a medical diagnostic method based on cross-modal semantic hashing through four steps. The specific model structure diagram is as follows. Figure 2 As shown, it includes:

[0057] Step 1: Use an artificial neural network to hash-encode the category labeling information to form a semantic similarity-preserving classifier;

[0058] Step 2: Use the semantic similarity-preserving classifier to guide the training of an artificial neural network encoder with multiple modalities;

[0059] Step 3: Use the trained modal encoder to perform hash encoding on the medical data of different modalities;

[0060] Step four: Complete the medical diagnosis through cross-modal semantic hash retrieval.

[0061] Furthermore, the cross-modal semantic hashing medical diagnosis method also includes preprocessing the input X-ray images and physical examination reports to construct category labeling information.

[0062] In this embodiment of the invention, the data preprocessing involves N1 X-ray images The size and number of channels were adjusted to 224×224×3, and their average value was subtracted.

[0063] In this embodiment of the invention, the data preprocessing is performed on N2 medical examination reports X 2 Use the bag-of-words model to extract the bag-of-words vector for each medical examination report. in The dimension is d2.

[0064] In this embodiment of the invention, the data preprocessing involves representing the diagnostic results of X-ray images and physical examination reports with corresponding diagnostic results using 0 and 1 vectorization to form category labels. in N3 represents the number of category labels.

[0065] In this embodiment of the invention, the encoder training includes two parts: learning a semantic similarity-preserving classifier and encoder training guided by the semantic similarity-preserving classifier.

[0066] The encoders refer to the imgNet artificial neural network for processing X-ray images and the txtNet artificial neural network for processing medical examination reports.

[0067] The semantic similarity-preserving classifier uses an artificial neural network to encode the semantic relationships of unique category label vectors, thereby using the corresponding hash encoding matrix. Established.

[0068] In one embodiment of the present invention, the encoder training guided by the semantic similarity-preserving classifier refers to treating the training process of each encoder as a classification task without the participation of other modal information, thereby using classification loss to enhance the clustering degree within disease classes and avoid the impact of partial medical data modality loss.

[0069] In this embodiment of the invention, the data encoding refers to using the trained imgNet network to encode the X-ray images in the X-ray image database and the X-ray images to be diagnosed for the disease, and using the trained txtNet network to encode the physical examination reports in the physical examination report database and the physical examination reports to be diagnosed for the disease.

[0070] In this embodiment of the invention, the cross-modal semantic hash retrieval involves calculating the Hamming distance between the hash code corresponding to the medical data to be diagnosed and the hash code corresponding to data in a medical database of a different modality, finding the data in the medical database that is closest to the medical data to be diagnosed based on the Hamming distance, and finally using the existing category label of the data itself as the diagnostic result of the medical data to be diagnosed.

[0071] In this embodiment of the invention, the encoder training includes two parts: learning a semantic similarity-preserving classifier and encoder training guided by the semantic similarity-preserving classifier.

[0072] To obtain a semantic similarity-preserving classifier, we first use LabNet to classify the category label y. l Hash encoding is performed. LabNet is a three-layer fully connected artificial neural network with d3, 4096 nodes and a bit length c. To ensure that the hash encoding of the category labels reflects the semantic relationships between disease categories, this embodiment of the invention optimizes LabNet using the following objective function:

[0073]

[0074]

[0075] Where f y (θ y ;y l ) is y l The output after processing by labNet, w l It is f y (θ y y l The hash code obtained after processing by the sign(·) function, where α is a hyperparameter that measures different items.

[0076] The first term of the objective function ensures that the hash encoding of the category labels reflects the semantic relationships between disease categories. The second term of the objective function constrains the output of the labNet network to be close to binary, satisfying the discrete nature of hash encoding.

[0077] After the labNet network is trained, it is used with the sign(·) function to process the unique class label vectors to obtain the corresponding hash codes. Then, the hash encoding matrix W is used as a semantic similarity-preserving classifier.

[0078] Once the semantic similarity-preserving classifier is built, the training process of each modality encoder is treated as a classification task without the participation of other modal information. This allows the classification loss to enhance the clustering within disease classes and avoid the impact of missing modalities in some medical data.

[0079] For X-ray images, the last fully connected layer of the original VGG-16 deep neural network is replaced with a fully connected layer of encoding length c to form imgNet. For medical examination reports, a three-layer fully connected artificial neural network txtNet with d2, 4096 nodes and encoding length c is constructed.

[0080] After the network is built, guided by the semantic similarity-preserving classifier, the following objective functions are used to optimize the network parameters θ1 and θ2 in imgNet and txtNet respectively:

[0081]

[0082] in It is a normalized vector. k∈[0, m-1], w l β is the hash code corresponding to the class label, and β, m, and k are hyperparameters. The first term of the objective function is the spherical classification loss function, which constrains the hash codes of samples of the same class to cluster. The second term of the objective function is the binarization loss function, which aims to make the encoder output close to binary to satisfy the discrete nature of hash codes.

[0083] In this embodiment of the invention, the data encoding is achieved by using a trained imgNet network to process X-ray images in the X-ray image database to obtain their corresponding approximate binarized codes. The trained txtNet network is then used to process the medical examination reports in the medical examination report database to obtain their corresponding approximate binary codes. To obtain the result from {0, 1} c Discrete hash code and use right and Process it.

[0084] Extract the corresponding hash code based on the type of medical data to be diagnosed. For X-ray images to be used for disease diagnosis... Obtain the corresponding discrete hash code by following the process of processing X-ray images in the X-ray image database. For medical examination reports pending diagnosis of a specific disease Obtain the corresponding discrete hash code according to the process of processing medical examination reports in the medical examination report database.

[0085] In this embodiment of the invention, the cross-modal semantic hash retrieval process is as follows:

[0086] The discrete hash codes corresponding to the data in the X-ray image database and the physical examination report database were obtained. and and the hash code corresponding to the medical data to be diagnosed or Next, the Hamming distance between the hash code corresponding to the medical data to be diagnosed and the hash codes corresponding to data in a medical database with a different modality is calculated. Based on the Hamming distance, the data in the medical database that is closest to the medical data to be diagnosed is found, and the existing category label of this data is used as the diagnostic result for the medical data to be diagnosed.

[0087] To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides specific product or related technology application examples of the technical solution claimed.

[0088] Example 1

[0089] By using the medical diagnostic method based on cross-modal semantic hashing of this invention, disease type diagnosis can be performed on X-ray images or physical examination reports using medical data of a different modality than the data to be diagnosed. When the data to be diagnosed is an X-ray image and the data in the database is a physical examination report, such as... Figure 3As shown, this invention finds the medical examination report data closest to the X-ray image to be diagnosed in the medical database, and uses the existing category labels of the data itself as the diagnostic result for the medical data to be diagnosed; when the data to be diagnosed is a medical examination report and the data in the database is an X-ray image, such as Figure 3 As shown, this invention finds the X-ray image data in the medical database that is closest to the medical examination report to be diagnosed, and uses the existing category label of the data itself as the diagnostic result of the medical data to be diagnosed.

[0090] Example 2

[0091] By using the medical diagnostic method based on cross-modal semantic hashing of this invention, disease type diagnosis can be performed on X-ray images using medical data of the same modality as the data to be diagnosed. When the data to be diagnosed is an X-ray image and the data in the database is also an X-ray image, such as... Figure 4 As shown, this invention finds the physical examination report data in the medical database that is closest to the X-ray image to be diagnosed, and uses the existing category labels of the data itself as the diagnostic result of the medical data to be diagnosed.

[0092] Example 3

[0093] Furthermore, such as Figure 5 The diagram shown is an architectural schematic of a medical diagnostic device based on cross-modal semantic hashing according to an embodiment of the present invention. The medical diagnostic device based on cross-modal semantic hashing in this embodiment includes one or more processors and at least two memories. Figure 5 The example used is a processor and two memory units.

[0094] Processor 1, Memory 1, and Memory 2 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0095] Memory 1 and Memory 2, as non-volatile computer-readable storage media, can be used to store volatile software programs and non-volatile computer-executable programs, such as the medical diagnosis method based on cross-modal semantic hashing and medical data and their corresponding hash codes in Embodiment 1. Processor 1 processes data in the medical database in Memory 2 by running the non-volatile software program and instructions stored in Memory 1, thereby executing the medical diagnosis method based on cross-modal semantic hashing and storing the hash codes corresponding to the data in the medical database in Memory 2.

[0096] Memory 1 and Memory 2 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, Memory 2 may optionally include memory remotely configured relative to Processor 1 and Memory 1, which can be connected to Processor 1 and Memory 1 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0097] The program instructions / modules are stored in the memory 1. When executed by one or more processors 1, they execute the method of the medical diagnostic device based on cross-modal semantic hashing in Embodiment 1 above to encode data in the medical database in the memory 2 and diagnose the data to be diagnosed. For example, they execute the above-described... Figure 1 or Figure 2 The steps shown.

[0098] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes the experimental process with data, charts and other information.

[0099] To evaluate the accuracy of the medical diagnoses provided by this invention, the Mean Average Precision (MAP) and Precision-Recall (PR) curves were used on the publicly available medical diagnostic benchmark dataset MIMIC-CXR for evaluation. MAP, as a commonly used metric for evaluating algorithm accuracy, indicates higher accuracy in medical diagnosis, with higher values ​​indicating greater accuracy. For the PR curve, the higher the position of the corresponding curve, the higher the accuracy of the algorithm's medical diagnosis. The MIMIC-CXR dataset contains 377,110 chest X-ray images and their corresponding physical examination reports, covering 14 common lung diseases. The hash encoding length was set to 16, 32, 64, and 128 bits, with DDH representing this invention, X representing the modality of the X-ray image data, and R representing the modality of the physical examination report data.

[0100] To evaluate the medical diagnostic effectiveness of this invention when the data to be diagnosed and the data in the database belong to different modalities, the DDH of this invention is compared with nine representative cross-modal semantic hashing methods. The nine comparison methods include: the EGDH method proposed in the paper "Equally-guided discriminative hashing for cross-modal retrieval," Proceedings of the 28th International Joint Conference on Artificial Intelligence (2019); the SSAH method proposed in the paper "Self-supervised adversarial hashing networks for cross-modal retrieval," Proceedings of the IEEE conference on computer vision and pattern recognition (2018); the DCMH method proposed in the paper "Deep cross-modal hashing," Proceedings of the IEEE conference on computer vision and pattern recognition (2017); the SePH method proposed in the paper "Cross-view retrieval via probability-based semantics-preserving hashing," IEEE transactions on cybernetics (2016); and the [unclear text - possibly related to image processing]. The CMFH method proposed in Processing (2016) (Semantic topic multimodal hashing for crossmedia retrieval)The STMH method proposed at the Twenty-Fourth International Joint Conference on Artificial Intelligence (2015), the SCM method proposed at the Twenty-Eighth AAAI Conference on Artificial Intelligence (2014), the CMSSH method proposed at the Twenty-Eighth IEEE Computer Society conference on computer vision and pattern recognition (2010), and the CCA method proposed at the Biometrika (1936) paper are all relevant to this discussion.

[0101] Table 1 shows the diagnostic effectiveness when the data to be diagnosed and the data in the database belong to different modalities (MAP).

[0102]

[0103]

[0104] According to the MAP test results in Table 1, when the data to be diagnosed is an X-ray image and the data in the database is a physical examination report, the method of the present invention improves the MAP by 2.42% compared to the EGDH method; when the data to be diagnosed is a physical examination report and the data in the database is an X-ray image, the method of the present invention improves the MAP by 1.81% compared to the EGDH method.

[0105] When the data to be diagnosed is an X-ray image and the data in the database is a physical examination report, Figure 6 The PR curves of this invention and related comparative methods are plotted. Figure 6 Based on the PR curve results, the present invention has a better diagnostic effect than other methods when the data to be diagnosed is X-ray images and the data in the database is physical examination reports.

[0106] When the data to be diagnosed is a physical examination report and the data in the database is an X-ray image, Figure 7 The PR curves of this invention and related comparative methods are plotted. Figure 7 Based on the PR curve results, the present invention outperforms other methods in the diagnostic effect when the data to be diagnosed is a physical examination report and the data in the database is an X-ray image.

[0107] To evaluate the medical diagnostic effectiveness of this invention when the data to be diagnosed and the data in the database belong to the same modality, the DDH of this invention is compared with six representative single-modal semantic hashing methods. The six comparison methods include: the HashNet method proposed in the paper (Hashnet: Deep learning to hash by continuation, Proceedings of the IEEE international conference on computer vision (2017)), the DSDH method proposed in the paper (Deep supervised discrete hashing, Advances in neural information processing systems (2017)), the DHN method proposed in the paper (Deep hashing network for efficient similarity retrieval, Thirtieth AAAI Conference on Artificial Intelligence (2016)), the DPSH method proposed in the paper (Feature learning based deep supervised hashing with pairwise labels, Twenty-Fifth International Joint Conference on Artificial Intelligence (2016)), the ITQ method proposed in the paper (Iterative quantization: A procedural approach to learning binary codes for large-scale image retrieval, IEEE transactions on pattern analysis and machine intelligence (2012)), and the Spectral hashing, Advances in neural information processing. The SH method proposed by systems (2015).

[0108] Table 2 shows the diagnostic effectiveness (MAP) when the data to be diagnosed and the data in the database belong to the same modality.

[0109]

[0110] According to the MAP test results in Table 2, when the data to be diagnosed is an X-ray image and the data in the database is also an X-ray image, the method of the present invention improves the MAP by 2.15% compared with the HashNet method.

[0111] When the data to be diagnosed is an X-ray image and the data in the database is also an X-ray image, Figure 8 The PR curves of this invention and related comparative methods are plotted. Figure 8 Based on the PR curve results, the present invention outperforms other methods when the data to be diagnosed is an X-ray image and the data in the database is also an X-ray image.

[0112] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or using software executed by various types of processors, or using a combination of the above-described hardware circuitry and software, such as firmware.

[0113] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A medical diagnostic method based on cross-modal semantic hashing, characterized in that, The medical diagnostic method using cross-modal semantic hashing includes: Step 1: Use an artificial neural network to hash-encode the category labeling information to form a semantic similarity-preserving classifier; Step 2: Use the semantic similarity-preserving classifier to guide the training of an artificial neural network encoder with multiple modalities; Step 3: Use the trained modal encoder to perform hash encoding on the medical data of different modalities; Step four: Complete the medical diagnosis through cross-modal semantic hash retrieval; The encoder training includes learning a semantic similarity-preserving classifier and encoder training based on the semantic similarity-preserving classifier; The process of learning the semantic similarity-preserving classifier includes: Using labNet for category labels Hash encoding is performed, and the objective function is used to optimize labNet, as shown in the following formula: ； in, Indicates semantic relations. This is used to constrain the output of the labNet network to be close to binarization. yes The output after processing by labNet yes through The hash code obtained after function processing For hyperparameters, hash encoding matrix As a semantic similarity-preserving classifier, S is a manually labeled data similarity matrix; The encoder training is based on the semantic similarity-preserving classifier using an objective function. Parameters in imgNet and txtNet networks and The optimized formula is as follows: ； in, For spherical classification loss function, For binary loss function, , It is a normalized vector. , , It is the hash code corresponding to the category label, and β, m and k are hyperparameters; The encoder consists of an imgNet artificial neural network for processing X-ray images and a txtNet artificial neural network for processing medical examination reports. The imgNet network modifies the last fully connected layer of the original VGG-16 deep neural network to a longer encoding bit length. The fully connected layer is formed; The txtNet network is built with the following number of nodes: 4096 and the encoding bit length A three-layer fully connected artificial neural network.

2. The medical diagnostic method based on cross-modal semantic hashing as described in claim 1, characterized in that, The cross-modal semantic hashing medical diagnosis method also includes preprocessing the input X-ray images and physical examination reports to construct category labeling information.

3. The medical diagnostic method based on cross-modal semantic hashing as described in claim 2, characterized in that, The preprocessing includes adjusting the size and channels of the X-ray image. And subtract the mean.

4. The medical diagnostic method based on cross-modal semantic hashing as described in claim 2, characterized in that, The preprocessing also includes extracting the bag-of-words model vector of the physical examination report using the bag-of-words model, and representing the diagnostic results as 0 and 1 vectors to form category labels.

5. The medical diagnostic method based on cross-modal semantic hashing as described in claim 1, characterized in that, The hash encoding process in step three is as follows: First, the imgNet network is used to obtain approximate binarization encoding of X-ray images. Approximate binarization encoding of physical examination reports is obtained using the txtNet network. ; Then, use right and Process and obtain from Discrete hash code and ; Finally, the corresponding hash code is extracted according to the type of medical data to be diagnosed.

6. The medical diagnostic method based on cross-modal semantic hashing as described in claim 1, characterized in that, The cross-modal semantic hash retrieval process is as follows: Calculate the Hamming distance between the hash code corresponding to the medical data to be diagnosed and the hash codes of X-ray images and physical examination reports of different modalities; Based on the Hamming distance, find the data in the medical database that is closest to the medical data to be diagnosed, and use the existing category label of the data itself as the diagnostic result for the medical data to be diagnosed.

7. A medical diagnostic device based on cross-modal semantic hashing, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being programmed to perform the medical diagnostic method based on cross-modal semantic hashing as described in any one of claims 1-6.

Images