Auxiliary diagnosis information generation method and device, electronic equipment and readable medium

Abstract

The embodiment of the invention discloses an auxiliary diagnosis information generation method and device, electronic equipment and a readable medium. A specific embodiment of the method comprises the following steps: acquiring case information, a scanning image set and a physiological signal set for a target case; performing information preprocessing on the case information; performing image preprocessing on each scanning image in the scanning image set; performing signal preprocessing on each physiological signal in the physiological signal set; respectively carrying out feature extraction on the preprocessed case information, the preprocessed scanning image set and the preprocessed physiological signal set; performing multi-modal feature fusion on the case features, the scanning image feature set and the physiological signal feature set; and generating auxiliary diagnosis information for the target case according to the fusion features and an auxiliary diagnosis information generator. According to the embodiment, the auxiliary diagnosis information combined with the multi-modal physiological information can be obtained to assist doctors in diagnosis, so that the diagnosis efficiency of the doctors is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and readable medium for generating auxiliary diagnosis information. Background Art

[0002] Cardiovascular disease is a disease of the heart or blood vessels, also known as circulatory system disease, which is a type of nervous system disease. Cardiovascular disease is one of the most common causes of death worldwide. Currently, the treatment and diagnosis of cardiovascular disease usually requires clinicians to screen and treat the cause based on the risk factors.

[0003] However, when the above method is adopted, the following technical problems often occur:

[0004] First, the screening method of clinical doctors relies heavily on the doctors’ diagnostic experience. At the same time, due to the imbalance between doctors and patients, it is difficult to improve the diagnostic efficiency.

[0005] In the process of adopting technical solutions to solve the above technical problem 1, the following technical problem 2 is often accompanied:

[0006] For medical images used as reference for disease diagnosis, it is difficult to effectively extract image content because common medical images often have low brightness.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0009] Some embodiments of the present disclosure propose auxiliary diagnostic information generation methods, devices, electronic devices, and readable media to solve one or more of the technical problems mentioned in the above background technology section.

[0010] In a first aspect, some embodiments of the present disclosure provide a method for generating auxiliary diagnostic information, the method comprising: acquiring case information, a scan image set and a physiological signal set for a target case, wherein the scan image set comprises: a first scan image and a second scan image, and the physiological signal set comprises: a first physiological signal, a second physiological signal and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetoencephalogram signal, and the third physiological signal is a cerebral cortical electrical signal; performing information preprocessing on the case information to generate preprocessed case information; for each of the scan image sets, The invention relates to a method for carrying out image preprocessing on each scanned image to generate a preprocessed scanned image and obtain a preprocessed scanned image set; carrying out signal preprocessing on each physiological signal in the above physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set; carrying out feature extraction on the above preprocessed case information, the above preprocessed scanned image set and the above preprocessed physiological signal set respectively to generate case features, a scanned image feature set and a physiological signal feature set; carrying out multimodal feature fusion on the above case features, the above scanned image feature set and the above physiological signal feature set to obtain fused features; and generating auxiliary diagnosis information for the above target case according to the above fused features and an auxiliary diagnosis information generator.

[0011] In a second aspect, some embodiments of the present disclosure provide an auxiliary diagnostic information generating device, the device comprising: an acquisition unit, configured to acquire case information, a scan image set and a physiological signal set for a target case, wherein the scan image set comprises: a first scan image and a second scan image, and the physiological signal set comprises: a first physiological signal, a second physiological signal and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetic signal, and the third physiological signal is a cerebral cortical electrical signal; an information preprocessing unit, configured to perform information preprocessing on the case information to generate preprocessed case information; an image preprocessing unit, configured to perform information preprocessing on each scan image in the scan image set. Image preprocessing to generate a preprocessed scan image and obtain a preprocessed scan image set; a signal preprocessing unit is configured to perform signal preprocessing on each physiological signal in the above-mentioned physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set; a feature extraction unit is configured to perform feature extraction on the above-mentioned preprocessed case information, the above-mentioned preprocessed scan image set and the above-mentioned preprocessed physiological signal set respectively to generate case features, a scan image feature set and a physiological signal feature set; a multimodal feature fusion unit is configured to perform multimodal feature fusion on the above-mentioned case features, the above-mentioned scan image feature set and the above-mentioned physiological signal feature set to obtain a fused feature; a generation unit is configured to generate auxiliary diagnosis information for the above-mentioned target case based on the above-mentioned fused features and an auxiliary diagnosis information generator.

[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0014] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the auxiliary diagnosis information generation method of some embodiments of the present disclosure, the auxiliary diagnosis information generated can indirectly improve the diagnosis efficiency of doctors. Specifically, the reason for the low diagnostic efficiency is that the way of screening by clinical doctors relies more on the diagnosis experience of doctors, and at the same time, due to the imbalance between doctors and patients, it is difficult to improve the diagnostic efficiency. Based on this, the auxiliary diagnosis information generation method of some embodiments of the present disclosure first obtains case information, a scan image set and a physiological signal set for the target case, wherein the scan image set includes: a first scan image and a second scan image, and the physiological signal set includes: a first physiological signal, a second physiological signal and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetic signal, and the third physiological signal is a cerebral cortex electrical signal. In this way, multimodal physiological information for the above target case is obtained. Secondly, the above case information is preprocessed to generate preprocessed case information. Next, image preprocessing is performed on each scanned image in the above-mentioned scanned image set to generate a preprocessed scanned image and obtain a preprocessed scanned image set. Next, signal preprocessing is performed on each physiological signal in the above-mentioned physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set. The availability of data is improved through preprocessing. In addition, feature extraction is performed on the above-mentioned preprocessed case information, the above-mentioned preprocessed scanned image set and the above-mentioned preprocessed physiological signal set respectively to generate case features, a scanned image feature set and a physiological signal feature set. Next, multimodal feature fusion is performed on the above-mentioned case features, the above-mentioned scanned image feature set and the above-mentioned physiological signal feature set to obtain fusion features. In this way, feature fusion corresponding to different modal data is achieved. Finally, auxiliary diagnosis information for the above-mentioned target case is generated according to the above-mentioned fusion features and the auxiliary diagnosis information generator. In this way, auxiliary diagnosis information combined with multimodal physiological information can be obtained to assist doctors in diagnosis, thereby indirectly improving the diagnosis efficiency of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1 is a flow chart of some embodiments of the auxiliary diagnosis information generating method according to the present disclosure;

[0017] Figure 2is a schematic diagram of the structure of some embodiments of the auxiliary diagnosis information generating device according to the present disclosure;

[0018] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] With regard to the collection, storage, and use of case information, scan image sets, and physiological signal sets involved in this disclosure, before performing corresponding operations, relevant organizations or individuals shall fulfill obligations including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0025] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0026] Continue to refer Figure 1 , shows a process 100 of some embodiments of the auxiliary diagnosis information generation method according to the present disclosure. The auxiliary diagnosis information generation method comprises the following steps:

[0027] Step 101, obtaining case information, a scan image set, and a physiological signal set for a target case.

[0028] In some embodiments, the execution subject (for example, a computing device) of the auxiliary diagnostic information generation method can obtain case information, a scan image set, and a physiological signal set for the target case through a wired connection or a wireless connection. Among them, the scan image set includes: a first scan image and a second scan image. The above-mentioned physiological signal set includes: a first physiological signal, a second physiological signal, and a third physiological signal. Among them, the above-mentioned first scan image is a magnetic resonance image. The above-mentioned second scan image is a tomography image. The above-mentioned first physiological signal is an electroencephalogram signal. The above-mentioned second physiological signal is a brain magnetic signal. The above-mentioned third physiological signal is a cortical electrical signal. In practice, the target case may be a case suspected of having cardiovascular disease.

[0029] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future.

[0030] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0031] Step 102, preprocessing the case information to generate preprocessed case information.

[0032] In some embodiments, the execution subject may perform information preprocessing on the case information to generate preprocessed case information. In practice, the execution subject may remove outliers from the case information to obtain the preprocessed case information.

[0033] Optionally, the pre-processed case information includes: basic case information, case disease history information and case follow-up record information. The basic case information may include: case identity, case age, case living habits description, etc. The case disease history information may represent the description of the disease that occurred in the target case in history. The case follow-up record information may represent at least one follow-up record for the target case.

[0034] In some optional implementations of some embodiments, the execution subject performs information preprocessing on the case information to generate preprocessed case information, which may include the following steps:

[0035] The first step is to determine the data format of the above case information.

[0036] The data format represents the storage format of the above case information. In practice, the case information can be a scanned copy, or a file in image format, or a text file with a structure. When the case information is a text file with a structure, it can be directly used as the pre-processed case information. When the data format is a scanned copy or an image file, character recognition is required before generating the above pre-processed case information.

[0037] In the second step, in response to the data format being a scan file format or an image file format, the following first preprocessing step is performed:

[0038] In the first sub-step, row-level ROI extraction is performed on the case information to generate row ROIs and obtain a row ROI set.

[0039] In practice, the execution subject can extract row-level regions of interest from the case information through the feature extraction part of the TextCNN model to generate row regions of interest and obtain a set of row regions of interest. In practice, the region of interest can be a horizontal region corresponding to a text line and containing text.

[0040] The second sub-step is to perform character recognition on each row region of interest in the row region of interest set to generate a character group.

[0041] In practice, the execution subject may perform character recognition on the region of interest in the line through the text classification part included in the TextCNN model to generate a character group.

[0042] The third sub-step is to extract semantic features from the character groups in the obtained character group set to generate semantic features.

[0043] In practice, first, the execution subject can embed characters into words according to their positions in the character group set through the Word2Vec model to obtain a character feature set, and then concatenate the character features in the character feature set to obtain the semantic features.

[0044] The fourth sub-step is to perform semantic segmentation on the characters in the above-mentioned character group set according to the above-mentioned semantic features to obtain the above-mentioned basic information of the case, the above-mentioned medical history information of the case and the above-mentioned case follow-up record information.

[0045] In practice, first, for semantic features, for example, the dimension of semantic features is M×N. The above subject can perform horizontal scanning of semantic features by setting a scanning frame. Then, semantic recognition is performed on the features in the scanning frame. When the semantic category is basic case information, the characters corresponding to the features in the scanning frame are used as the basic case information. When the semantic category is case disease history information, the characters corresponding to the features in the scanning frame are used as case disease history information. When the semantic category is case follow-up record information, the characters corresponding to the features in the scanning frame are used as case follow-up record information. Specifically, taking the semantic category as basic case information as an example, when the corresponding semantic category is basic case information, but the corresponding confidence is less than the threshold, the scanning frame can be enlarged, that is, more features are contained in the scanning frame, and then semantic recognition is performed to obtain the threshold. When the threshold is greater than the preset threshold, the characters corresponding to the features in the scanning frame are used as the basic case information. At the same time, the scanning frame is initialized and rescanned from the truncation position. Among them, the truncation position can be the position of the last feature value contained in the feature in the scanning frame.

[0046] Step 103: perform image preprocessing on each scanned image in the scanned image set to generate a preprocessed scanned image, thereby obtaining a preprocessed scanned image set.

[0047] In some embodiments, the execution subject may perform image preprocessing on each scanned image in the scanned image set to generate a preprocessed scanned image and obtain a preprocessed scanned image set. In practice, the execution subject may adjust the brightness of the scanned image to obtain a preprocessed scanned image corresponding to the scanned image.

[0048] In some optional implementations of some embodiments, the execution subject performs image preprocessing on each scanned image in the scanned image set to generate a preprocessed scanned image, and obtaining the preprocessed scanned image set may include the following steps:

[0049] In the first step, the above scanned image is denoised by salt and pepper noise to generate a first denoised scanned image.

[0050] In the second step, Gaussian noise denoising is performed on the first denoised scanned image to obtain a second denoised scanned image.

[0051] In the third step, the encoder included in the image enhancement model is used to encode the second denoised scanned image to obtain candidate image features.

[0052] In practice, the encoder contains 4 encoding blocks and 5 brightness adjustment blocks. Among them, the 4 decoding blocks and the 5 brightness adjustment blocks are linearly connected. Specifically, the encoding block includes: SC (Sparse Convolution) layer, upsampling layer and RCB (Residual Convolution Block) layer. The brightness adjustment block can be simplified as: M out =M in ×φ(B)×(1+φ(E)). Among them, M out Represents the output of the brightness adjustment block. M in represents the input of the brightness adjustment block. φ() represents the linear operation. B represents the feature map obtained by subtracting each standardized pixel value in the second denoised scanned image from 1. E represents the image edge feature corresponding to the second denoised scanned image.

[0053] The fourth step is to generate brightness adjustment information for the second denoised scanned image using the brightness prediction model included in the image enhancement model and the candidate image features.

[0054] Among them, the brightness prediction model contains 4 linearly connected brightness embedding blocks. Among them, the brightness embedding block includes the first convolution layer, the second convolution layer, the first sigmoid layer, and the second sigmoid layer. The first convolution layer and the first sigmoid layer are linearly connected. The second convolution layer and the second sigmoid layer are linearly connected. Specifically, taking the first brightness embedding block as an example, after the candidate image feature is input into the first convolution layer and the first sigmoid layer, the first feature is obtained. The first feature and the candidate image feature are cross-multiplied to obtain the second feature. After the candidate image feature is input into the second convolution layer and the second sigmoid layer, the third feature is obtained. The third feature and the candidate image feature are cross-multiplied to obtain the fourth feature. The third feature and the fourth feature are superimposed to obtain the fifth feature, which is the output of 1 brightness embedding block.

[0055] The fifth step is to generate chromaticity adjustment information for the second denoised scanned image through the chromaticity prediction model included in the image enhancement model and the candidate image features.

[0056] The model structure of the chromaticity prediction model is consistent with that of the luminance prediction model, but the corresponding processing tasks are different. Specifically, the chromaticity prediction model is used to extract chromaticity features. The brightness prediction model is used to extract brightness features.

[0057] The sixth step is to perform image enhancement on the second denoised scanned image according to the brightness feature information and the chromaticity adjustment information to generate a preprocessed scanned image in the preprocessed scanned image set corresponding to the scanned image.

[0058] The above-mentioned content of "in some optional implementations in some embodiments", as an inventive point of the present disclosure, solves the second technical problem mentioned in the background technology, namely, "for medical images used for disease diagnosis reference, since common medical images are often low-brightness, it is difficult to effectively extract image content". Specifically, the present disclosure, first, performs salt and pepper noise denoising and Gaussian noise denoising on the scanned image to eliminate image noise. Then, feature extraction of the second denoised scanned image is implemented through an encoder. Then, from the two perspectives of chroma and brightness, the second denoised scanned image is enhanced through a brightness prediction model and a chroma prediction model, thereby improving the image brightness and reducing image noise, thereby improving the effectiveness of subsequent image content extraction.

[0059] Step 104 , performing signal preprocessing on each physiological signal in the physiological signal set to generate a preprocessed physiological signal, thereby obtaining a preprocessed physiological signal set.

[0060] In some embodiments, the execution subject may perform signal preprocessing on each physiological signal in the physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set. In practice, the execution subject may perform signal standardization on the physiological signal to obtain the corresponding preprocessed physiological signal.

[0061] Step 105 , extracting features from the preprocessed case information, the preprocessed scan image set, and the preprocessed physiological signal set, respectively, to generate case features, a scan image feature set, and a physiological signal feature set.

[0062] In some embodiments, the above-mentioned execution entity can extract features from the preprocessed case information, the preprocessed scan image set and the preprocessed physiological signal set in various ways to generate case features, scan image feature sets and physiological signal feature sets.

[0063] In some optional implementations of some embodiments, the execution subject extracts features from the preprocessed case information, the preprocessed scan image set, and the preprocessed physiological signal set to generate case features, a scan image feature set, and a physiological signal feature set, respectively, which may include the following steps:

[0064] In the first step, the above case information including basic case information, case disease history information and case follow-up record information is standardized to generate standardized basic case information, standardized case disease history information and standardized case follow-up record information.

[0065] In practice, the above-mentioned execution entity can convert the basic case information, the case disease history information and the case follow-up record information into the form of KV pairs as standardized case basic information, standardized case disease history information and standardized case follow-up record information.

[0066] The second step is to perform text feature extraction steps on the above standardized case basic information:

[0067] In the first sub-step, word embedding is performed on the above-mentioned standardized case basic information to generate sentence vectors and obtain the first sentence vector sequence.

[0068] In practice, the execution entity may perform word embedding processing on the basic information of the standardized cases through a Seq2Seq model to generate sentence vectors and obtain a first sentence vector sequence.

[0069] In the second sub-step, for each first sentence vector in the sentence vector sequence, convolution processing is performed on the first sentence vector through K convolution kernels to obtain a first sentence vector feature group.

[0070] Among them, the convolution kernel sizes in the K convolution kernels are different, K ≥ 3.

[0071] In the third sub-step, down-sampling is performed on the first sentence vector features in the obtained first sentence vector feature group set to generate first down-sampled sentence vector features, and to obtain a first down-sampled sentence vector feature set.

[0072] Among them, there is a one-to-one correspondence between the sentence vector feature after the first downsampling and the first sentence vector.

[0073] The third step is to perform the above text feature extraction steps on the above case medical history information:

[0074] In the first sub-step, word embedding is performed on the above-mentioned standardized case basic information to generate sentence vectors, thereby obtaining a second sentence vector sequence;

[0075] In the second sub-step, for each second sentence vector in the sentence vector sequence, convolution processing is performed on the second sentence vector through K convolution kernels to obtain a second sentence vector feature group.

[0076] Among them, the convolution kernel sizes in the K convolution kernels are different, K ≥ 3;

[0077] In the third sub-step, down-sampling is performed on the second sentence vector features in the obtained second sentence vector feature group set to generate second down-sampled sentence vector features, and obtain a second down-sampled sentence vector feature set.

[0078] Among them, there is a one-to-one correspondence between the second down-sampled sentence vector feature and the second sentence vector.

[0079] Step 4: Perform the above text feature extraction steps on the above case follow-up record information:

[0080] In the first sub-step, word embedding is performed on the above-mentioned standardized case basic information to generate sentence vectors, thereby obtaining a third sentence vector sequence;

[0081] In the second sub-step, for each third sentence vector in the above sentence vector sequence, convolution processing is performed on the above third sentence vector through K convolution kernels to obtain a third sentence vector feature group.

[0082] Among them, the convolution kernel sizes in the K convolution kernels are different, K ≥ 3.

[0083] In the third sub-step, down-sampling is performed on the third sentence vector features in the obtained third sentence vector feature group set to generate third down-sampled sentence vector features, and to obtain a third down-sampled sentence vector feature set.

[0084] Among them, there is a one-to-one correspondence between the third down-sampled sentence vector feature and the third sentence vector.

[0085] The fourth sub-step is to perform feature concatenation on the first down-sampled sentence vector feature set, the second down-sampled sentence vector feature set and the third down-sampled sentence vector feature set to obtain the case features.

[0086] In some optional implementations of some embodiments, the execution subject respectively extracts features from the preprocessed case information, the preprocessed scan image set, and the preprocessed physiological signal set to generate case features, a scan image feature set, and a physiological signal feature set, further comprising the following steps:

[0087] For each preprocessed scanned image in the above preprocessed scanned image set, the following image feature extraction steps are performed:

[0088] In the first step, a convolution process is performed on the preprocessed scanned image through a first convolution layer to generate a first image feature.

[0089] Among them, the convolution kernel size corresponding to the above-mentioned first convolution layer is 3×3, and the step size corresponding to the above-mentioned first convolution layer is 1.

[0090] In the second step, the first image feature is pooled through the first pooling layer to generate a pooled first image feature.

[0091] Among them, the pooling window size corresponding to the above-mentioned first pooling layer is 2×2, and the step size corresponding to the above-mentioned first pooling layer is 2.

[0092] The third step is to perform convolution processing on the first image features after pooling through the second convolution layer to generate the second image features.

[0093] Among them, the convolution kernel size corresponding to the above-mentioned second convolution layer is 3×3, and the step size corresponding to the above-mentioned first convolution layer is 1.

[0094] In the fourth step, the second image features are pooled through a second pooling layer to generate pooled second image features.

[0095] Among them, the pooling window size corresponding to the above-mentioned second pooling layer is 2×2, and the step size corresponding to the above-mentioned second pooling layer is 2.

[0096] The fifth step is to perform feature expansion on the second image feature through a flattening layer to obtain the expanded image feature as the image feature in the scanned image feature set corresponding to the preprocessed scanned image.

[0097] Among them, the above-mentioned expanded image features are one-dimensional feature vectors.

[0098] In some optional implementations of some embodiments, the execution subject respectively extracts features from the preprocessed case information, the preprocessed scan image set, and the preprocessed physiological signal set to generate case features, a scan image feature set, and a physiological signal feature set, further comprising the following steps:

[0099] For each preprocessed physiological signal in the preprocessed physiological signal set, a signal feature extraction is performed on the preprocessed physiological signal through a LTSM (Long Short Term Memory Networks) model to obtain a corresponding physiological signal feature.

[0100] Step 106, performing multimodal feature fusion on the case features, the scan image feature set and the physiological signal feature set to obtain fused features.

[0101] In some embodiments, the execution entity may perform multimodal feature fusion on the case features, the scan image feature set, and the physiological signal feature set to obtain fused features.

[0102] In some optional implementations of some embodiments, the execution subject performs multimodal feature fusion on the case feature, the scan image feature set, and the physiological signal feature set to obtain the fused feature, which may include the following steps:

[0103] In the first step, deep feature extraction is performed on the above case features through the first modality feature extractor to generate deep extracted case features.

[0104] Among them, the first modal feature extractor adopts the ResNeXt model.

[0105] In the second step, a second modality feature extractor is used to perform depth feature extraction on the scanned image features in the above scanned image feature set to generate depth-extracted scanned image features and obtain a depth-extracted scanned image feature set.

[0106] Among them, the second modality feature extractor adopts the TextCNN model.

[0107] In the third step, a third modality feature extractor is used to perform deep feature extraction on the physiological signal features in the above physiological signal feature set to generate deep extracted physiological signal features and obtain a deep extracted physiological signal feature set.

[0108] Among them, the third modality feature extractor adopts the Bi-LSTM model.

[0109] The fourth step is to fuse the case features after the depth extraction, the scanned image feature set after the depth extraction and the physiological signal feature set after the depth extraction through a feature fusion model to obtain the fused features.

[0110] Among them, the feature fusion model is based on the Cross-Attention mechanism, which performs cross-modal feature combination of the case features after deep extraction, the above-mentioned scanned image feature set after deep extraction, and the above-mentioned physiological signal feature set after deep extraction to obtain the above-mentioned fusion features.

[0111] Step 107, generating auxiliary diagnosis information for the target case according to the fusion features and the auxiliary diagnosis information generator.

[0112] In some embodiments, auxiliary diagnosis information for a target case is generated based on the fused features and the auxiliary diagnosis information generator. In practice, the auxiliary diagnosis information generator includes a text prediction head for generating auxiliary diagnosis information.

[0113] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the auxiliary diagnosis information generation method of some embodiments of the present disclosure, the auxiliary diagnosis information generated can indirectly improve the diagnosis efficiency of doctors. Specifically, the reason for the low diagnostic efficiency is that the way of screening by clinical doctors relies more on the diagnosis experience of doctors, and at the same time, due to the imbalance between doctors and patients, it is difficult to improve the diagnostic efficiency. Based on this, the auxiliary diagnosis information generation method of some embodiments of the present disclosure first obtains case information, a scan image set and a physiological signal set for the target case, wherein the scan image set includes: a first scan image and a second scan image, and the physiological signal set includes: a first physiological signal, a second physiological signal and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetic signal, and the third physiological signal is a cerebral cortex electrical signal. In this way, multimodal physiological information for the above target case is obtained. Secondly, the above case information is preprocessed to generate preprocessed case information. Next, image preprocessing is performed on each scanned image in the above-mentioned scanned image set to generate a preprocessed scanned image and obtain a preprocessed scanned image set. Next, signal preprocessing is performed on each physiological signal in the above-mentioned physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set. The availability of data is improved through preprocessing. In addition, feature extraction is performed on the above-mentioned preprocessed case information, the above-mentioned preprocessed scanned image set and the above-mentioned preprocessed physiological signal set respectively to generate case features, a scanned image feature set and a physiological signal feature set. Next, multimodal feature fusion is performed on the above-mentioned case features, the above-mentioned scanned image feature set and the above-mentioned physiological signal feature set to obtain fusion features. In this way, feature fusion corresponding to different modal data is achieved. Finally, auxiliary diagnosis information for the above-mentioned target case is generated according to the above-mentioned fusion features and the auxiliary diagnosis information generator. In this way, auxiliary diagnosis information combined with multimodal physiological information can be obtained to assist doctors in diagnosis, thereby indirectly improving the diagnosis efficiency of doctors.

[0114] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an auxiliary diagnosis information generating device, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the auxiliary diagnosis information generating device can be specifically applied to various electronic devices.

[0115] like Figure 2As shown, some embodiments of the auxiliary diagnosis information generating device 200 include: an acquisition unit 201, an information preprocessing unit 202, an image preprocessing unit 203, a signal preprocessing unit 204, a feature extraction unit 205, a multimodal feature fusion unit 206 and a generation unit 207. The acquisition unit 201 is configured to acquire case information, a scan image set and a physiological signal set for a target case, wherein the scan image set includes: a first scan image and a second scan image, and the physiological signal set includes: a first physiological signal, a second physiological signal and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetoencephalogram signal, and the third physiological signal is a cerebral cortical electrical signal; the information preprocessing unit 202 is configured to perform information preprocessing on the case information to generate preprocessed case information; the image preprocessing unit 203 is configured to perform image preprocessing on each scan image in the scan image set to generate a preprocessed scan image, A preprocessed scan image set is obtained; the signal preprocessing unit 204 is configured to perform signal preprocessing on each physiological signal in the above physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set; the feature extraction unit 205 is configured to perform feature extraction on the above preprocessed case information, the above preprocessed scan image set and the above preprocessed physiological signal set respectively to generate case features, a scan image feature set and a physiological signal feature set; the multimodal feature fusion unit 206 is configured to perform multimodal feature fusion on the above case features, the above scan image feature set and the above physiological signal feature set to obtain fused features; the generation unit 207 is configured to generate auxiliary diagnosis information for the above target case according to the above fused features and the auxiliary diagnosis information generator.

[0116] It is understandable that the units described in the auxiliary diagnostic information generating device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the auxiliary diagnosis information generating device 200 and the units included therein, and will not be described in detail here.

[0117] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. Various programs and data required for the operation of the electronic device 300 are also stored in the random access memory 303. The processing device 301, the read-only memory 302, and the random access memory 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0119] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0120] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the read-only memory 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0121] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0122] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0123] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains case information, a scanning image set and a physiological signal set for a target case, wherein the above-mentioned scanning image set includes: a first scanning image and a second scanning image, and the above-mentioned physiological signal set includes: a first physiological signal, a second physiological signal and a third physiological signal, wherein the above-mentioned first scanning image is a magnetic resonance image, the above-mentioned second scanning image is a tomography image, the above-mentioned first physiological signal is an electroencephalogram signal, the above-mentioned second physiological signal is a brain magnetic signal, and the above-mentioned third physiological signal is a cerebral cortical electrical signal; performs information preprocessing on the above-mentioned case information to generate preprocessed case information; and performs information preprocessing on the above-mentioned scanning image. Perform image preprocessing on each scanned image in the image set to generate a preprocessed scanned image and obtain a preprocessed scanned image set; perform signal preprocessing on each physiological signal in the above physiological signal set to generate a preprocessed physiological signal and obtain a preprocessed physiological signal set; perform feature extraction on the above preprocessed case information, the above preprocessed scanned image set and the above preprocessed physiological signal set respectively to generate case features, a scanned image feature set and a physiological signal feature set; perform multimodal feature fusion on the above case features, the above scanned image feature set and the above physiological signal feature set to obtain fused features; generate auxiliary diagnosis information for the above target case based on the above fused features and the auxiliary diagnosis information generator.

[0124] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, an information preprocessing unit, an image preprocessing unit, a signal preprocessing unit, a feature extraction unit, a multimodal feature fusion unit and a generation unit. Among them, the names of these units do not constitute a limitation on the unit itself in certain cases. For example, the multimodal feature fusion unit may also be described as "a unit that performs multimodal feature fusion on the above-mentioned case features, the above-mentioned scan image feature set and the above-mentioned physiological signal feature set to obtain fused features".

[0127] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0128] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for generating auxiliary diagnosis information, comprising: Acquire case information, a scan image set, and a physiological signal set for a target case, wherein the scan image set includes: a first scan image and a second scan image, and the physiological signal set includes: a first physiological signal, a second physiological signal, and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetoencephalogram signal, and the third physiological signal is a cerebral cortex electrical signal; Performing information preprocessing on the case information to generate preprocessed case information; Performing image preprocessing on each scanned image in the scanned image set to generate a preprocessed scanned image, thereby obtaining a preprocessed scanned image set; Performing signal preprocessing on each physiological signal in the physiological signal set to generate a preprocessed physiological signal, thereby obtaining a preprocessed physiological signal set; Performing feature extraction on the preprocessed case information, the preprocessed scan image set, and the preprocessed physiological signal set, respectively, to generate case features, a scan image feature set, and a physiological signal feature set; Performing multimodal feature fusion on the case feature, the scan image feature set and the physiological signal feature set to obtain fused features; Auxiliary diagnosis information for the target case is generated according to the fusion feature and the auxiliary diagnosis information generator.

2. The method according to claim 1, wherein: The pre-processed case information includes: basic case information, case medical history information and case follow-up record information; and The preprocessing of the case information to generate preprocessed case information includes: Determining the data format of the case information; In response to the data format being a scan file format or an image file format, performing the following first preprocessing step: Performing row-level region of interest extraction on the case information to generate row regions of interest and obtain a set of row regions of interest; For each row region of interest in the row region of interest set, performing character recognition on the row region of interest to generate a character group; Extracting semantic features from the character groups in the obtained character group set to generate semantic features; According to the semantic features, semantic segmentation is performed on the characters in the character group set to obtain the basic information of the case, the medical history information of the case and the follow-up record information of the case.

3. The method according to claim 2, wherein: The extracting features of the preprocessed case information, the preprocessed scanned image set and the preprocessed physiological signal set respectively to generate case features, a scanned image feature set and a physiological signal feature set includes: Standardizing the case basic information, case disease history information, and case follow-up record information included in the case information to generate standardized case basic information, standardized case disease history information, and standardized case follow-up record information; For the standardized case basic information, perform the text feature extraction step: Performing word embedding processing on the standardized case basic information to generate sentence vectors, thereby obtaining a first sentence vector sequence; For each first sentence vector in the sentence vector sequence, convolution processing is performed on the first sentence vector through K convolution kernels to obtain a first sentence vector feature group, wherein the convolution kernels in the K convolution kernels have different sizes, and K≥3; Downsampling the first sentence vector feature in the obtained first sentence vector feature group set to generate a first downsampled sentence vector feature, and obtaining a first downsampled sentence vector feature set, wherein the first downsampled sentence vector feature and the first sentence vector have a one-to-one correspondence; For the case medical history information, perform the text feature extraction step: Performing word embedding processing on the basic information of the standardized case to generate a sentence vector to obtain a second sentence vector sequence; For each second sentence vector in the sentence vector sequence, convolution processing is performed on the second sentence vector through K convolution kernels to obtain a second sentence vector feature group, wherein the convolution kernels in the K convolution kernels have different sizes, and K≥3; Downsampling the second sentence vector features in the obtained second sentence vector feature group set to generate second downsampled sentence vector features, and obtaining a second downsampled sentence vector feature set, wherein the second downsampled sentence vector features and the second sentence vector have a one-to-one correspondence; For the case follow-up record information, perform the text feature extraction step: Performing word embedding processing on the standardized case basic information to generate sentence vectors, thereby obtaining a third sentence vector sequence; For each third sentence vector in the sentence vector sequence, convolution processing is performed on the third sentence vector through K convolution kernels to obtain a third sentence vector feature group, wherein the convolution kernels in the K convolution kernels have different sizes, and K≥3; Downsampling the third sentence vector features in the obtained third sentence vector feature group set to generate third downsampled sentence vector features, and obtaining a third downsampled sentence vector feature set, wherein the third downsampled sentence vector features and the third sentence vector have a one-to-one correspondence; The first down-sampled sentence vector feature set, the second down-sampled sentence vector feature set and the third down-sampled sentence vector feature set are concatenated to obtain the case features.

4. The method according to claim 3, wherein: The extracting features of the preprocessed case information, the preprocessed scanned image set and the preprocessed physiological signal set to generate case features, a scanned image feature set and a physiological signal feature set also includes: For each preprocessed scanned image in the preprocessed scanned image set, the following image feature extraction steps are performed: Performing convolution processing on the preprocessed scanned image through a first convolution layer to generate a first image feature, wherein a convolution kernel size corresponding to the first convolution layer is 3×3, and a step size corresponding to the first convolution layer is 1; Performing pooling processing on the first image feature through a first pooling layer to generate a pooled first image feature, wherein a pooling window size corresponding to the first pooling layer is 2×2, and a step size corresponding to the first pooling layer is 2; Performing convolution processing on the pooled first image feature through a second convolution layer to generate a second image feature, wherein the convolution kernel size corresponding to the second convolution layer is 3×3, and the step size corresponding to the first convolution layer is 1; Performing pooling processing on the second image feature through a second pooling layer to generate a pooled second image feature, wherein the pooling window size corresponding to the second pooling layer is 2×2, and the step size corresponding to the second pooling layer is 2; The second image feature is feature expanded through a flattening layer to obtain an expanded image feature as an image feature in the scanned image feature set corresponding to the preprocessed scanned image, wherein the expanded image feature is a one-dimensional feature vector.

5. The method according to claim 4, wherein: The multimodal feature fusion of the case feature, the scanned image feature set and the physiological signal feature set to obtain the fused feature includes: Performing deep feature extraction on the case features by a first modality feature extractor to generate deep extracted case features; Performing depth feature extraction on the scanned image features in the scanned image feature set by a second modality feature extractor to generate depth-extracted scanned image features, thereby obtaining a depth-extracted scanned image feature set; Performing deep feature extraction on the physiological signal features in the physiological signal feature set by a third modality feature extractor to generate deep-extracted physiological signal features, thereby obtaining a deep-extracted physiological signal feature set; The feature fusion model is used to fuse the case features after depth extraction, the scan image feature set after depth extraction, and the physiological signal feature set after depth extraction to obtain the fused features.

6. An auxiliary diagnostic information generating device, comprising: an acquisition unit, configured to acquire case information, a scan image set, and a physiological signal set for a target case, wherein the scan image set includes: a first scan image and a second scan image, and the physiological signal set includes: a first physiological signal, a second physiological signal, and a third physiological signal, wherein the first scan image is a magnetic resonance image, the second scan image is a tomography image, the first physiological signal is an electroencephalogram signal, the second physiological signal is a brain magnetoencephalogram signal, and the third physiological signal is a cerebral cortex electrical signal; An information preprocessing unit, configured to perform information preprocessing on the case information to generate preprocessed case information; An image preprocessing unit is configured to perform image preprocessing on each scanned image in the scanned image set to generate a preprocessed scanned image, thereby obtaining a preprocessed scanned image set; A signal preprocessing unit is configured to perform signal preprocessing on each physiological signal in the physiological signal set to generate a preprocessed physiological signal, thereby obtaining a preprocessed physiological signal set; A feature extraction unit, configured to perform feature extraction on the preprocessed case information, the preprocessed scan image set and the preprocessed physiological signal set, respectively, to generate case features, a scan image feature set and a physiological signal feature set; A multimodal feature fusion unit is configured to perform multimodal feature fusion on the case feature, the scan image feature set and the physiological signal feature set to obtain a fused feature; A generating unit is configured to generate auxiliary diagnosis information for the target case according to the fusion feature and the auxiliary diagnosis information generator.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images