Uric acid data acquisition and abnormality detection method and device based on multi-modal fusion

Abstract

Embodiments of the present disclosure disclose a uric acid data acquisition and abnormality detection method and device based on multi-modal fusion. A specific embodiment of the method includes: controlling a medical data query end to query a historical uric acid detection data set corresponding to uric acid detail retrieval information; performing image enhancement processing on an initial uric acid detection image according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image; performing uric acid region identification on the enhanced uric acid detection image to obtain a uric acid identification region image group; optimizing each uric acid identification region image in the uric acid identification region image group to generate an optimized uric acid identification region image, thereby obtaining an optimized uric acid identification region image group; and inputting the historical uric acid detection data set and the optimized uric acid identification region image group into a pre-trained multi-modal uric acid abnormality detection model to generate a uric acid abnormality detection result. The embodiment improves detection accuracy and realizes precise and dynamic detection of uric acid abnormalities.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of uric acid data collection and anomaly detection, and in particular to a uric acid data collection and anomaly detection method and device based on multi-modal fusion. BACKGROUND

[0002] In the field of medical detection, early and accurate diagnosis of uric acid abnormalities (such as hyperuricemia and gout) is crucial for preventing joint damage and kidney failure. There are three major bottlenecks in the existing technology:

[0003] First, data fragmentation: uric acid images (CT / ultrasound), biochemical indicators (blood uric acid value), and historical medical records are scattered in different systems, and there is a lack of dynamic correlation analysis capability;

[0004] Second, image misdiagnosis: traditional imaging equipment has low recognition rate (about 37%) for small uric acid salt crystals (<3mm) and is easily affected by false positives;

[0005] Third, response delay: the detection results rely on manual review, and cannot provide real-time early warning of abnormal trends, delaying the intervention opportunity.

[0006] Although there are deep learning-based uric acid recognition models, the problem of multi-source data fusion and asynchronous monitoring has not been solved, resulting in high false positive rate and insufficient system robustness.

[0007] The above information disclosed in this BACKGROUND section is only for enhancing the understanding of the background of the present inventive concepts, and therefore, it can include information that does not form the prior art that is already known in this field to those of ordinary skill in the art. SUMMARY

[0008] The summary section of the present disclosure is provided to introduce the concepts in a simplified form, which will be described in detail in the following detailed description section. The summary section of the present disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to be used to limit the scope of the claimed technical solutions.

[0009] Some embodiments of the present disclosure propose a uric acid data collection and anomaly detection method and device based on multi-modal fusion to solve the technical problems mentioned in the above BACKGROUND section.

[0010] In a first aspect, some embodiments of the present disclosure provide a method for uric acid data acquisition and anomaly detection based on multi-modal fusion, the method comprising: controlling a uric acid detection device to collect an initial uric acid detection image of a target user; generating corresponding uric acid detail search information according to historical uric acid search information corresponding to the target user, and generating uric acid search association information according to search bytecode information; controlling a medical data query monitoring end to monitor the search performance of the historical uric acid search information according to asynchronous search operation information corresponding to the uric acid search association information; controlling a medical data query end to query a historical uric acid detection data set corresponding to the uric acid detail search information; performing image enhancement processing on the initial uric acid detection image according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image; performing uric acid region recognition on the enhanced uric acid detection image to obtain a uric acid recognition region image group; optimizing each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image and obtain an optimized uric acid recognition region image group; and inputting the historical uric acid detection data set and the optimized uric acid recognition region image group into a pre-trained multi-modal uric acid anomaly detection model to generate a uric acid anomaly detection result.

[0011] In a second aspect, some embodiments of the present disclosure provide an apparatus for uric acid data acquisition and anomaly detection based on multi-modal fusion, the apparatus comprising: a collection unit configured to control a uric acid detection device to collect an initial uric acid detection image of a target user; a generation unit configured to generate corresponding uric acid detail search information according to historical uric acid search information corresponding to the target user, and generate uric acid search association information according to search bytecode information; a control unit configured to control a medical data query monitoring end to monitor the search performance of the historical uric acid search information according to asynchronous search operation information corresponding to the uric acid search association information; a query unit configured to control a medical data query end to query a historical uric acid detection data set corresponding to the uric acid detail search information; an enhancement unit configured to perform image enhancement processing on the initial uric acid detection image according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image; a recognition unit configured to perform uric acid region recognition on the enhanced uric acid detection image to obtain a uric acid recognition region image group; an optimization unit configured to optimize each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image and obtain an optimized uric acid recognition region image group; and an input unit configured to input the historical uric acid detection data set and the optimized uric acid recognition region image group into a pre-trained multi-modal uric acid anomaly detection model to generate a uric acid anomaly detection result.

[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementation manners of the first aspect.

[0014] The above various embodiments of the present application have the following beneficial effects: through the uric acid data acquisition and abnormality detection method based on multi-modal fusion of some embodiments of the present disclosure, multi-modal fusion and intelligent asynchronous monitoring are realized, and precise dynamic detection of uric acid abnormalities is realized. Specifically, first, control the uric acid detection device to collect the initial uric acid detection image of the target user; generate the corresponding uric acid detail retrieval information according to the historical uric acid retrieval information corresponding to the target user, and generate the uric acid retrieval association information according to the retrieval bytecode information; control the medical data query monitoring end to monitor the retrieval performance of the historical uric acid retrieval information according to the asynchronous retrieval operation information, wherein the asynchronous retrieval operation information corresponds to the uric acid retrieval association information; control the medical data query end to query the historical uric acid detection data set corresponding to the uric acid detail retrieval information. Thus, based on the retrieval bytecode information, the query performance is dynamically monitored, and the data retrieval delay is shortened; the asynchronous operation mechanism processes the image enhancement and data query in parallel, greatly speeding up the overall detection process. Then, according to the pre-trained uric acid detection image enhancement model, the initial uric acid detection image is subjected to image enhancement processing to generate an enhanced uric acid detection image; the enhanced uric acid detection image is subjected to uric acid region recognition to obtain a uric acid recognition region image group; each uric acid recognition region image in the uric acid recognition region image group is optimized to generate an optimized uric acid recognition region image, and an optimized uric acid recognition region image group is obtained. Thus, the historical uric acid data and real-time image features are fused to construct a cross-modal association model, and the microcrystal recognition accuracy is improved; the image enhancement model optimizes the low-contrast region and reduces the artifact interference. Finally, the historical uric acid detection data set and the optimized uric acid recognition region image group are input into the pre-trained multi-modal uric acid abnormality detection model to generate a uric acid abnormality detection result. Thus, the multi-modal abnormality detection model integrates the uric acid region image group and the historical data set, greatly improving the detection accuracy and realizing precise dynamic detection of uric acid abnormalities. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the annexed drawings in which: like reference numerals refer to like elements throughout. The annexed drawings are schematic and are not intended to accurately depict the proportions or relative positions of elements.

[0016] Figure 1 is a flowchart of some embodiments of a uric acid data acquisition and abnormality detection method based on multi-modal fusion according to the present disclosure;

[0017] Figure 2 is a model structure diagram of a uric acid detection image enhancement model in a uric acid data acquisition and abnormality detection method based on multi-modal fusion of the present disclosure;

[0018] Figure 3 is a structure diagram of some embodiments of a uric acid data acquisition and abnormality detection apparatus based on multi-modal fusion according to the present disclosure;

[0019] Figure 4 is a structure diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should also be noted that, for the sake of brevity, only the parts of the drawings that are relevant to the present application are shown. The embodiments and features of the present disclosure can be combined with each other as long as there is no conflict.

[0022] It should be noted that the terms "first", "second", and the like in the present disclosure are only used to distinguish different devices, modules, or units, and do not limit the functions performed by these devices, modules, or units or the mutual dependency therebetween.

[0023] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes and do not limit the scope of the messages or information.

[0025] The collection, storage, and use of user personal information (e.g., uric acid detection images) involved in the present disclosure are performed by the relevant organization or individual in accordance with the obligations including conducting a personal information security impact assessment, informing the personal information subject of the obligations, and obtaining the authorization consent of the personal information subject in advance.

[0026] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0027] Figure 1 The flow 100 of some embodiments of the uric acid data collection and anomaly detection method based on multi-modal fusion according to the present disclosure is shown. The uric acid data collection and anomaly detection method based on multi-modal fusion includes the following steps:

[0028] Step 101, control the uric acid detection device to collect the initial uric acid detection image of the target user.

[0029] In some embodiments, the execution subject (e.g., a computing device) of the uric acid data collection and anomaly detection method based on multi-modal fusion can control the uric acid detection device to collect the initial uric acid detection image of the target user. The uric acid detection device can refer to a device for collecting uric acid data that is communicatively connected to the above-mentioned execution subject. For example, the uric acid detection device can be a medical ultrasonic diagnostic instrument, mainly having two parts, namely the device host and the ultrasonic probe. The host part of the ultrasonic diagnostic instrument mainly processes and displays the signals received from the probe. The ultrasonic probe can transmit and receive ultrasonic waves, convert electrical and acoustic signals, convert electrical signals sent by the host into high-frequency oscillating ultrasonic signals, and convert ultrasonic signals reflected from tissues and organs into electrical signals to be displayed on the display of the host. For another example, the uric acid detection device can be a urine analyzer for detecting uric acid in urine, which adopts photoelectric colorimetric analysis technology and is an automatic instrument for measuring certain chemical components in urine. The initial uric acid detection image can refer to the ultrasonic image of the target user to be detected. The target user can refer to a patient to be detected for uric acid. For example, the above-mentioned execution subject can control the medical ultrasonic diagnostic instrument to perform ultrasonic detection on the target user to obtain the initial uric acid detection image.

[0030] Step 102, according to the historical uric acid search information corresponding to the above-mentioned target user, generate the corresponding uric acid detail search information, and according to the search bytecode information, generate the uric acid search association information.

[0031] In some embodiments, the execution subject can generate the uric acid detail retrieval information corresponding to the historical uric acid retrieval information of the target user according to the historical uric acid retrieval information of the target user, and generate the uric acid retrieval association information according to the retrieval bytecode information. The historical uric acid retrieval information can represent operation information of retrieving historical uric acid data of the target user. For example, the historical uric acid retrieval information can represent an operation of inputting a retrieval statement on a retrieval page by a medical staff. For example, the retrieval statement can be "query historical uric acid data of the target user".

[0032] In some optional implementations of some embodiments, the execution subject can generate the uric acid detail retrieval information corresponding to the historical uric acid retrieval information of the target user according to the historical uric acid retrieval information of the target user by the following steps:

[0033] Firstly, the retrieval disease logic information corresponding to the historical uric acid retrieval information is determined according to the historical uric acid retrieval information. The retrieval disease logic information can represent the retrieval uric acid disease logic corresponding to the historical uric acid retrieval information. For example, the execution subject can determine the retrieval type corresponding to the historical uric acid retrieval information. Then, the retrieval disease logic information corresponding to the historical uric acid retrieval information is determined from a preset uric acid disease logic type table according to the retrieval type. The uric acid disease logic type table can represent the corresponding relationship between the retrieval type and the disease logic information.

[0034] Secondly, the combined retrieval statement information corresponding to the historical uric acid retrieval information is generated according to the retrieval disease logic information. The combined retrieval statement information can represent the combined retrieval statement. For example, the execution subject can use the dynamic splicing DSL method to generate the combined retrieval statement information corresponding to the historical uric acid retrieval information according to the retrieval disease logic information.

[0035] Thirdly, the call retrieval statement information corresponding to the historical uric acid retrieval information is generated according to the combined retrieval statement information. For example, the execution subject can determine the call retrieval statement information corresponding to the historical uric acid retrieval information in the form of calling an API interface.

[0036] Fourthly, the retrieval disease logic information, the combined retrieval statement information, and the call retrieval statement information are combined into the uric acid detail retrieval information corresponding to the historical uric acid retrieval information.

[0037] The uric acid search associated information can represent search data associated with the historical uric acid search information. The search bytecode information can represent bytecode corresponding to source code of the call search expression information. The uric acid search associated information can include search conditions, a user ID, and a search time. For example, the execution subject can capture the uric acid search associated information corresponding to the query bytecode information through a SpringAOP method.

[0038] At step 103, the medical data query monitoring terminal is controlled to monitor the search performance of the historical uric acid search information according to the asynchronous search operation information.

[0039] In some embodiments, the execution subject can control the medical data query monitoring terminal to monitor the search performance of the historical uric acid search information according to the asynchronous search operation information. The asynchronous search operation information corresponds to the uric acid search associated information. The asynchronous search operation information can represent set asynchronous search operation information. For example, the asynchronous search operation information includes server information, and the included server information represents an IP address that is a preset IP address. The asynchronous search operation information can represent asynchronous operation triggered by bytecode enhancement technology. The asynchronous search operation information can include a DSL query statement, a user ID, a time, and a server IP. For example, the execution subject can determine the preset asynchronous search operation setting information as the asynchronous search operation information corresponding to the uric acid search associated information. For example, the execution subject can control the medical data query monitoring terminal to monitor the search performance of the historical uric acid search information according to the asynchronous search operation information through a search performance tool. For example, the search performance tool can be an APM (application performance management) tool. The medical data query monitoring terminal can be a terminal that monitors the search query capability of a database storing medical data.

[0040] In some optional implementations of some embodiments, the execution subject can monitor the search performance of the historical uric acid search information by the following steps:

[0041] First, the medical data query monitoring terminal is controlled to listen to the asynchronous search operation information to determine whether the asynchronous search operation information meets a preset monitoring condition. For example, the execution subject can control the medical data query monitoring terminal to listen to the asynchronous search operation information through a listener. The monitoring condition can be that the server ID included in the asynchronous search operation information is the same as a preset server ID.

[0042] The second step involves extracting fields from the asynchronous retrieval operation information to obtain the corresponding retrieval field information, in response to the determination that the asynchronous retrieval operation information meets the aforementioned monitoring conditions. Each of these retrieval field information represents a retrieval field extracted from the asynchronous retrieval operation information. For example, the executing entity can control the medical data query monitoring terminal to perform field extraction processing on the asynchronous retrieval operation information using a field parsing and extraction tool to obtain the corresponding retrieval field information. For example, the field extraction and parsing tool could be a JSONPath tool.

[0043] The third step is to merge the information from each of the above search fields to obtain the uric acid search tracking information. For example, the executing entity can use a general monitoring tracking SDK to assemble and process the information from each of the above search fields to obtain the uric acid search tracking information.

[0044] The fourth step is to add the aforementioned uric acid retrieval tracking information to a pre-defined asynchronous retrieval information queue. This asynchronous retrieval information queue can be a Message Queue.

[0045] Fifth, in response to the determination that the aforementioned uric acid retrieval tracking information meets the preset tracking storage conditions, corresponding retrieval performance monitoring information is generated based on the aforementioned uric acid retrieval tracking information to monitor the retrieval performance of the aforementioned historical uric acid retrieval information. The preset tracking storage conditions can be either a preset order of the aforementioned uric acid retrieval tracking information in the preset retrieval information queue or a preset priority of the aforementioned uric acid retrieval tracking information. The preset order can be the first one. The preset priority can be the first level. The aforementioned retrieval performance monitoring information can characterize the monitored retrieval behavior or the monitored retrieval performance.

[0046] This improves the flexibility and maintainability of retrieval performance monitoring, reduces the difficulty of migrating retrieval performance monitoring, and lowers the complexity of retrieval performance monitoring while reducing the computational resources consumed.

[0047] Step 104: Control the medical data query terminal to query the historical uric acid test dataset corresponding to the above-mentioned uric acid details retrieval information.

[0048] In some embodiments, the execution subject can control the medical data query terminal to query a historical uric acid detection data set corresponding to the uric acid detail retrieval information. The historical uric acid detection data set can represent the search result corresponding to the uric acid detail retrieval information. In practice, the execution subject can control the medical data query terminal to input the uric acid detail retrieval information into a query search engine to obtain the historical uric acid detection data set. For example, the query search engine can be Elasticsearch. The historical uric acid detection data can refer to the uric acid detection data of the target user at a historical time point, and can include image data, detection results, etc.

[0049] In step 105, the initial uric acid detection image is subjected to image enhancement processing according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image.

[0050] In some embodiments, the execution subject can subject the initial uric acid detection image to image enhancement processing according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image. For example, the uric acid detection image enhancement model can be a LLNet (Low Light Network) model. The uric acid detection image enhancement model includes a surface image feature extraction network, a multi-level image feature encoding network, and an image enhancement network. The surface image feature extraction network includes a surface cross-encoding layer, the multi-level image feature encoding network includes a first cross-encoder, a second cross-encoder, and a third cross-encoder, the first cross-encoder, the second cross-encoder, and the third cross-encoder are connected in series, the image enhancement network includes a first enhancement decoder, a second enhancement decoder, and a third enhancement decoder, the first enhancement decoder, the second enhancement decoder, and the third enhancement decoder are connected in series, a first attention fusion network layer is arranged between the first cross-encoder and the first enhancement decoder, a second attention fusion network layer is arranged between the second cross-encoder and the second enhancement decoder, a third attention fusion network layer is arranged between the third cross-encoder and the third enhancement decoder, and an encoding network is arranged after the third attention fusion network layer.

[0051] In some optional implementations of some embodiments, the execution subject can subject the initial uric acid detection image to image enhancement processing by the following steps:

[0052] S1, the surface feature of the initial uric acid detection image is extracted by the surface image feature extraction network included in the uric acid detection image enhancement model to generate a surface uric acid detection image feature.

[0053] S2, the surface uric acid detection image features are encoded by the multi-level image feature encoding network included in the uric acid detection image enhancement model to generate uric acid detection encoded image features.

[0054] S3, the uric acid detection encoded image features are image enhanced by the image enhancement network included in the uric acid detection image enhancement model to generate enhanced uric acid detection images.

[0055] As an example, see Figure 2 The structure diagram of the uric acid detection image enhancement model is shown. The input of the surface cross-encoding layer is the initial uric acid detection image. The input of the first cross-encoder is the output of the surface cross-encoding layer. The output of the first cross-encoder is sent to the second cross-encoder, the first attention fusion network layer, and the first enhancement decoder. The output of the second cross-encoder is sent to the third cross-encoder, the second attention fusion network layer, and the second enhancement decoder. The output of the third cross-encoder is sent to the third attention fusion network layer and the third enhancement decoder. The output of the first enhancement decoder is sent to the first attention fusion network layer and the second enhancement decoder. The output of the second enhancement decoder is sent to the second attention fusion network layer and the third enhancement decoder. The output of the third enhancement decoder is sent to the third attention fusion network layer.

[0056] The surface cross-encoding layer and the encoding network each include three encoding layers. The encoding layer includes one normalization layer, one convolution layer (convolution kernel is 1x1), two multi-head attention mechanism modules, one normalization layer, and one feedforward network. The first cross-encoder, the second cross-encoder, and the third cross-encoder include two sub-encoding modules and a down-sampling network. The two multi-head attention mechanism modules include a multi-scale convolution layer and a multi-head attention mechanism layer. The first enhancement decoder, the second enhancement decoder, and the third enhancement decoder include an up-sampling network and three sub-encoding modules.

[0057] The first attention fusion network layer, the second attention fusion network layer, and the third attention fusion network layer each include three average pooling layers, three multi-layer perceptrons, one Sigmoid activation function, and one ReLU activation function. The three average pooling layers and the three multi-layer perceptrons are symmetrically arranged.

[0058] Thus, the surface layer image feature extraction network is used to extract the surface layer features of the image. Then, the encoder and decoder with a symmetrical structure are used to encode and decode the features. The surface layer features are extracted by the surface layer cross-encoding layer, and the noise in the output features is suppressed by the encoding network. In addition, in order to reduce the processing amount of the features, the cross-encoder decomposes the feature attention into horizontal and vertical directions, realizes the correlation of the results by cross design, and improves the image enhancement capability in this way.

[0059] In step 106, the enhanced uric acid detection image is subjected to uric acid region recognition to obtain a uric acid recognition region image group.

[0060] In some embodiments, the execution subject can perform uric acid region recognition on the enhanced uric acid detection image to obtain a uric acid recognition region image group. For example, each uric acid region in the enhanced uric acid detection image can be recognized by dual-energy CT, X-ray plain film (KUB).

[0061] For another example, the enhanced uric acid detection image can be subjected to uric acid region recognition by a dual-energy CT (DECT) recognition model, a three-dimensional reconstruction and quantitative analysis model, a traditional CT density analysis model, an ultrasonic image feature model, etc., to obtain a uric acid recognition region image group.

[0062] Principle of the dual-energy CT (DECT) recognition model: using 80kVp and 140kVp dual-energy scanning, the attenuation value difference of uric acid stones under low / high energy spectrum is collected. Uric acid has a unique attenuation slope in the energy spectrum curve due to its low atomic number characteristics (effective atomic number 6.8-7.8).

[0063] Recognition method of the dual-energy CT (DECT) recognition model: 1. Substances separation map: the post-processing software generates a uric acid exclusive color marking map (such as green color marking uric acid, and other components showing different colors). 2. Dual-energy ratio analysis: calculate the attenuation ratio under high and low energy (the uric acid ratio is significantly lower than the calcium-containing stones).

[0064] Technical process of the three-dimensional reconstruction and quantitative analysis model: based on dual-source CT data, use Mimics software to generate a tophus three-dimensional model, label the position, volume and adjacent relationship with blood vessels / nerves.

[0065] Principle of the traditional CT density analysis model: the density of uric acid stones is significantly lower than that of calcium-containing stones, and the typical CT value is 300-500HU (calcium stones >1000HU); recognition method: threshold segmentation method (such as setting HU<500 region as suspicious uric acid stones).

[0066] The characteristic signs of the ultrasonic image feature model are: 1. Double contour sign: uric acid deposits on the surface of the articular cartilage, forming a high echo line; 2. Aggregates: point-like / massive high echo with acoustic shadow in the joint cavity.

[0067] In some optional implementations of some embodiments, the execution subject can perform uric acid region identification on the enhanced uric acid detection image by the following steps:

[0068] Firstly, each uric acid stone region in the enhanced uric acid detection image is labeled to obtain a labeled uric acid detection image. The radiologists can manually outline, automatically, semi-automatically, manually, or the like to label the uric acid stone region in the enhanced uric acid detection image to obtain the labeled uric acid detection image.

[0069] Secondly, the labeled uric acid detection image is subjected to two-dimensional image segmentation to obtain a two-dimensional uric acid detection segmentation image set. For example, the labeled uric acid detection image can be input into a trained image segmentation model to obtain the two-dimensional uric acid detection segmentation image set. The image segmentation model is used for segmenting two-dimensional images. The image segmentation model can be a Swin Transformer Unet (Swin-Unet) model for medical image segmentation. The image segmentation model is constructed by a Swin transformer (Shifted Window Transformer) model and a Unet structure. The Swin-Unet model deeply integrates the global context modeling capability of the Swin Transformer and the detail retention advantage of the U-Net. The Unet model is a convolutional neural network for image segmentation, which adopts a symmetrical encoder-decoder structure.

[0070] Thirdly, for each two-dimensional uric acid detection segmentation image in the two-dimensional uric acid detection segmentation image set, the following processing steps are performed:

[0071] S1. High semantic feature extraction is performed on the two-dimensional uric acid detection segmentation image to generate high semantic features. The execution subject can perform high semantic feature extraction on the two-dimensional uric acid detection segmentation image by an encoder to generate high semantic features.

[0072] S2. Up-sampling feature extraction is performed on the two-dimensional uric acid detection segmentation image to generate up-sampling features. The execution subject can perform up-sampling feature extraction on the two-dimensional uric acid detection segmentation image by a transposed convolution layer to generate up-sampling features.

[0073] S3, the high semantic feature and the up-sampling feature are spliced to generate a spliced feature. As an example, the execution body can splice the high semantic feature and the up-sampling feature in the same channel dimension using Python to generate a spliced feature.

[0074] S4, the spliced feature is subjected to spatial weight image generation to generate a spatial weight image. As an example, the execution body can generate a spatial weight image by convolving the spliced feature through a convolution layer.

[0075] S5, the spatial weight image is subjected to weighting processing to generate a weighted feature image. As an example, the execution body can multiply the spatial weight image and the spliced feature element by element to generate a weighted feature image.

[0076] S6, the weighted feature image is decoded to generate a decoded uric acid feature map. As an example, the execution body can decode the weighted feature image through a decoder to generate a decoded uric acid feature map.

[0077] A fourth step is to determine each decoded uric acid feature map obtained as a uric acid region image group.

[0078] In this way, the readability of the uric acid region image can be improved through image conversion and two-dimensional segmentation, thereby reducing the probability of losing key information and improving the robustness of the uric acid region image.

[0079] Step 107, each uric acid recognition region image in the uric acid recognition region image group is optimized to generate an optimized uric acid recognition region image, and an optimized uric acid recognition region image group is obtained.

[0080] In some embodiments, the execution body can optimize each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image, and an optimized uric acid recognition region image group is obtained. For example, the execution body can optimize the uric acid recognition region image by the following steps, which can include: contrast adjustment, brightness / color correction; can also include: edge sharpening (Laplacian operator, non-sharpening mask); can also include: medical image special optimization (CT / MRI enhancement, ultrasound image optimization) and various optimization methods.

[0081] In some optional implementations of some embodiments, the execution body can optimize each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image by the following steps:

[0082] In a first step, the sub-region boundary of the above uric acid recognition area image is optimized by morphological operation to obtain a first optimized uric acid recognition area image. The morphological operation can include but is not limited to: dilation, erosion, opening operation, closing operation and the like. For example, small noise and isolated regions can be removed by opening operation, and the sub-region boundary can be smoothed, and small holes in the sub-region can be filled by closing operation to ensure the continuity of the sub-region.

[0083] In a second step, the first optimized uric acid recognition area image is topologically repaired to obtain a repaired uric acid recognition area image as an optimized uric acid recognition area image. For example, a closing operation priority and an erosion-dilation combination can be used to smooth the boundary of the first optimized uric acid recognition area image.

[0084] In step 108, the historical uric acid detection data set and the optimized uric acid recognition area image group are input into the pre-trained multi-modal uric acid abnormality detection model to generate a uric acid abnormality detection result.

[0085] In some embodiments, the execution subject can input the historical uric acid detection data set and the optimized uric acid recognition area image group into the pre-trained multi-modal uric acid abnormality detection model to generate a uric acid abnormality detection result. For example, the multi-modal uric acid abnormality detection model can be a multi-modal neural network model that comprehensively detects whether the uric acid is abnormal according to the user's historical uric acid detection data and the current uric acid recognition area image. For example, the multi-modal uric acid abnormality detection model can be a multi-modal machine learning prediction model.

[0086] The technical principle of the multi-modal machine learning prediction model is: integrating genetic data (such as SNP sites) and clinical indicators (BMI, blood lipids, kidney function, etc.), generating a comprehensive risk score (such as ISHUA) through a stacked machine learning architecture (base model + meta model) to realize early screening and complication prediction of hyperuricemia (HUA); base model: LightGBM, XGBoost and other 7 classifiers process genetic and clinical features in parallel. Meta model: XGBoost integrates the base model output to generate the final prediction probability; feature selection: LASSO regression selects 1378 key genetic markers from 38277 SNPs, combined with 10 clinical indicators.

[0087] Optionally, the multi-modal uric acid abnormality detection model can be trained by the following steps:

[0088] The first step is to obtain a historical uric acid sign detection data set and a corresponding detection label set. The historical uric acid sign detection data includes uric acid sign detection data and a historical uric acid recognition region image group, and one historical uric acid sign detection data corresponds to one detection label. The detection label can include a data label and an image label. The data label corresponds to the uric acid sign detection data. The image label corresponds to the historical uric acid recognition region image group. The data label can represent whether the user corresponding to the uric acid sign detection data will have uric acid abnormalities at a future time node. The image label represents whether the corresponding historical uric acid recognition region image group is uric acid abnormal.

[0089] The second step is to preprocess each uric acid sign detection data in the historical uric acid sign detection data set to generate preprocessed uric acid sign detection data, thereby obtaining a preprocessed uric acid sign detection data set. The data preprocessing can include data normalization, data completion, and data enhancement.

[0090] The third step is to determine the network structure of an initial multi-modal uric acid abnormality detection model. The initial multi-modal uric acid abnormality detection model includes an initial uric acid sign detection data detection model and an initial uric acid recognition region image detection model. The initial uric acid sign detection data detection model can be an untrained uric acid sign detection data detection model. The uric acid sign detection data detection model is used to identify and predict uric acid sign detection data, and can include an input layer, a multi-level convolution layer (Conv2D+ReLU), a pooling layer (MaxPooling), and a classification output layer. The initial uric acid recognition region image detection model can be an untrained uric acid recognition region image detection model. For example, the uric acid recognition region image detection model can be a fusion model of a dual-energy CT urate crystal recognition model and an ultrasound image feature recognition model. The ultrasound image feature recognition model can include an input layer, a convolution layer (primary convolution (Conv2D+ReLU), a pooling layer (MaxPooling), and a fully connected layer.

[0091] The fourth step is to train an initial detection data semantic extraction model according to the preprocessed uric acid sign detection data set to generate a detection data semantic extraction model. The detection data semantic extraction model includes a feature extraction layer for extracting cross-feature information between each sign data in the uric acid sign detection data. The initial detection data semantic extraction model can be a semantic extraction model that has not yet been trained. The semantic extraction model can be a neural network model corresponding to the detection content of the uric acid sign detection task. For example, the semantic extraction model can be a multi-layer series of encoding layers (e.g., multiple series of down-sampling layers). The semantic extraction model includes a feature extraction layer for extracting cross-feature information between each sign data.

[0092] In the fifth step, the initial uric acid sign detection data detection model is trained according to the preprocessed uric acid sign detection data set and the detection label set to generate a uric acid sign detection data detection model. The uric acid sign detection data detection model includes the feature extraction layer of the detection data semantic extraction model. The model training method can refer to the training method of a deep neural network model, which will not be described here. For example, hinge loss or cross-entropy loss can be used to identify the loss of the model during the model training process. The optimization method can use the stochastic gradient descent method for optimization.

[0093] In the sixth step, the initial uric acid recognition region image detection model is trained according to each historical uric acid recognition region image group included in the historical uric acid sign detection data set to obtain a trained uric acid recognition region image detection model. The model training method can refer to the training method of a deep neural network model, which will not be described here. For example, hinge loss or cross-entropy loss can be used to identify the loss of the model during the model training process. The optimization method can use the stochastic gradient descent method for optimization.

[0094] In the seventh step, the uric acid sign detection data detection model and the uric acid recognition region image detection model are fused into a trained multi-modal uric acid anomaly detection model.

[0095] Therefore, by using the pre-trained feature extraction layer as the feature extraction module of the initial uric acid sign detection data detection model, the cross-content features of the detection content under each sign data can be learned, which facilitates efficient model training of the multi-modal uric acid anomaly detection model, improves the understanding of sign data content, and shortens the training time. In summary, by pre-training the initial detection data semantic extraction model, a feature extraction layer that extracts cross-feature information between parameters in sign data can be obtained. By using the feature extraction layer as the feature extraction module of the initial uric acid sign detection data detection model, the initial uric acid sign detection data detection model can extract cross-relation information between parameters in each sign data, ensuring the model training accuracy of the multi-modal uric acid anomaly detection model and shortening the training time.

[0096] Further reference Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multi-modal fusion based uric acid data acquisition and anomaly detection apparatus, which corresponds to the method embodiments shown in Figure 1 The multi-modal fusion based uric acid data acquisition and anomaly detection apparatus can be applied to various electronic devices.

[0097] As Figure 3As shown, the multi-modal fusion based uric acid data acquisition and anomaly detection apparatus 300 of some embodiments includes an acquisition unit 301, a generation unit 302, a control unit 303, a query unit 304, an enhancement unit 305, an identification unit 306, an optimization unit 307, and an input unit 308. Among them, the acquisition unit 301 is configured to control the uric acid detection device to acquire an initial uric acid detection image of a target user; the generation unit 302 is configured to generate corresponding uric acid detail retrieval information according to historical uric acid retrieval information corresponding to the target user, and generate uric acid retrieval association information according to retrieval bytecode information; the control unit 303 is configured to control the retrieval performance of the historical uric acid retrieval information of the medical data query monitoring end according to asynchronous retrieval operation information, wherein the asynchronous retrieval operation information corresponds to the uric acid retrieval association information; the query unit 304 is configured to control the medical data query end to query a historical uric acid detection data set corresponding to the uric acid detail retrieval information; the enhancement unit 305 is configured to perform image enhancement processing on the initial uric acid detection image according to a pre-trained uric acid detection image enhancement model, to generate an enhanced uric acid detection image; the identification unit 306 is configured to identify a uric acid region in the enhanced uric acid detection image to obtain a uric acid identification region image group; the optimization unit 307 is configured to optimize each uric acid identification region image in the uric acid identification region image group to generate an optimized uric acid identification region image, to obtain an optimized uric acid identification region image group; and the input unit 308 is configured to input the historical uric acid detection data set and the optimized uric acid identification region image group into a pre-trained multi-modal uric acid anomaly detection model, to generate a uric acid anomaly detection result.

[0098] It can be understood that the units described in the multi-modal fusion based uric acid data acquisition and anomaly detection apparatus 300 correspond to the respective steps in the method described above. Figure 1 The operations, features, and advantages described above for the method also apply to the multi-modal fusion based uric acid data acquisition and anomaly detection apparatus 300 and the units contained therein, and will not be repeated here.

[0099] Reference is made below to Figure 4 which shows a structural schematic diagram of an electronic device (such as a computing device) suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of embodiments of the present disclosure. As Figure 4As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory can include a non-volatile storage medium and an internal memory. The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions which, when executed, can cause the processor to perform any one of the above-mentioned uric acid data acquisition and anomaly detection methods based on multi-modal fusion. The processor is used to provide computing and control capabilities to support the operation of the entire computer device. The internal memory provides an environment for the computer program in the non-volatile storage medium to run, and the computer program, when executed by the processor, can cause the processor to perform any one of the above-mentioned uric acid data acquisition and anomaly detection methods based on multi-modal fusion. The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the present disclosure, and does not constitute a limitation on the computer device to which the present disclosure is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0101] In one embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps: controlling a uric acid detection device to collect an initial uric acid detection image of a target user; generating corresponding uric acid detail search information according to historical uric acid search information corresponding to the target user, and generating uric acid search association information according to search bytecode information; controlling the medical data query monitoring end to monitor the search performance of the historical uric acid search information according to asynchronous search operation information corresponding to the uric acid search association information; controlling the medical data query end to query a historical uric acid detection data set corresponding to the uric acid detail search information; performing image enhancement processing on the initial uric acid detection image according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image; performing uric acid region recognition on the enhanced uric acid detection image to obtain a uric acid recognition region image group; optimizing each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image, thereby obtaining an optimized uric acid recognition region image group; and inputting the historical uric acid detection data set and the optimized uric acid recognition region image group into a pre-trained multi-modal uric acid anomaly detection model to generate a uric acid anomaly detection result.

[0102] The embodiments of the present disclosure further provide a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed, a method is implemented, which can refer to each embodiment of the method of the present disclosure.

[0103] The computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0104] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or system that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article, or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article, or system that includes the element.

[0105] The above description is merely exemplary of some of the many possible embodiments of the present disclosure and of the principles thereof. It is to be understood that those skilled in the art will be able to devise various embodiments of the present disclosure without departing from the scope of the present disclosure as disclosed in the above description and claims. For example, the technical features of the above-described embodiments can be combined with other technical features disclosed in the present disclosure (but not limited to) to form other technical solutions.

Claims

1. A method for uric acid data acquisition and anomaly detection based on multimodal fusion, characterized in that, include: Control the uric acid detection device to acquire the initial uric acid test images of the target user; Based on the historical uric acid retrieval information of the target user, generate corresponding uric acid detail retrieval information, and based on the retrieval bytecode information, generate uric acid retrieval association information; Based on the asynchronous retrieval operation information, the medical data query monitoring terminal is controlled to monitor the retrieval performance of the historical uric acid retrieval information, wherein the asynchronous retrieval operation information corresponds to the uric acid retrieval association information; The medical data query terminal can query historical uric acid test datasets corresponding to the uric acid details retrieval information. Based on a pre-trained uric acid detection image enhancement model, the initial uric acid detection image is enhanced to generate an enhanced uric acid detection image. The uric acid detection image enhancement model includes a surface image feature extraction network, a multi-level image feature encoding network, and an image enhancement network. The surface image feature extraction network includes a surface cross-encoding layer. The multi-level image feature encoding network includes a first cross-encoder, a second cross-encoder, and a third cross-encoder; the first, second, and third cross-encoders are serially connected. The image enhancement network includes a first enhancement decoder, a second enhancement decoder, and a third enhancement decoder; the first, second, and third enhancement decoders are serially connected. A first attention fusion network layer is provided between the first cross-encoder and the first enhancement decoder; a second attention fusion network layer is provided between the second cross-encoder and the second enhancement decoder; a third attention fusion network layer is provided between the third cross-encoder and the third enhancement decoder; and an encoding network is provided after the third attention fusion network layer. The enhanced uric acid detection images are used to identify uric acid regions, resulting in a group of uric acid identification region images. Each uric acid recognition region image in the uric acid recognition region image group is optimized to generate an optimized uric acid recognition region image, resulting in an optimized uric acid recognition region image group. The optimization of the uric acid recognition region image includes: removing small noise and isolated regions, and smoothing sub-region boundaries. The historical uric acid detection dataset and the optimized uric acid identification region image group are input into a pre-trained multimodal uric acid abnormality detection model to generate uric acid abnormality detection results. The image enhancement processing of the initial uric acid test image includes: The surface image feature extraction network included in the uric acid detection image enhancement model is used to extract surface features from the initial uric acid detection image to generate surface uric acid detection image features. The surface uric acid detection image features are encoded using the multi-level image feature coding network included in the uric acid detection image enhancement model to generate uric acid detection coded image features. The uric acid detection image enhancement model includes an image enhancement network to enhance the uric acid detection encoded image features, thereby generating an enhanced uric acid detection image.

2. The method according to claim 1, characterized in that, The step of optimizing each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image includes: The sub-region boundaries of the uric acid recognition region image are optimized through morphological operations to obtain a first optimized uric acid recognition region image; Topological restoration is performed on the first optimized uric acid recognition region image to obtain a restored uric acid recognition region image, which is used as the optimized uric acid recognition region image.

3. The method according to claim 1, characterized in that, The step of generating corresponding uric acid detail search information based on the historical uric acid search information of the target user includes: Based on the historical uric acid retrieval information, determine the retrieval disease logic information corresponding to the historical uric acid retrieval information; Based on the retrieval of the disease logic information, generate combined retrieval statements corresponding to the historical uric acid retrieval information; Based on the combined search query information, generate the corresponding historical uric acid search information; The retrieval of symptom logic information, the combined retrieval statement information, and the retrieval of call information are combined into uric acid detail retrieval information corresponding to the historical uric acid retrieval information.

4. A uric acid data acquisition and anomaly detection device based on multimodal fusion, characterized in that, include: The acquisition unit is configured to control the uric acid detection device to acquire the initial uric acid test image of the target user; The generation unit is configured to generate corresponding uric acid detail search information based on the historical uric acid search information of the target user, and to generate uric acid search association information based on the search bytecode information. The control unit is configured to control the medical data query monitoring terminal to monitor the retrieval performance of the historical uric acid retrieval information according to the asynchronous retrieval operation information, wherein the asynchronous retrieval operation information corresponds to the uric acid retrieval association information; The query unit is configured to control the medical data query terminal to query historical uric acid test datasets corresponding to the uric acid details retrieval information; An enhancement unit is configured to perform image enhancement processing on the initial uric acid detection image according to a pre-trained uric acid detection image enhancement model to generate an enhanced uric acid detection image. The uric acid detection image enhancement model includes a surface image feature extraction network, a multi-level image feature encoding network, and an image enhancement network. The surface image feature extraction network includes a surface cross-encoding layer. The multi-level image feature encoding network includes a first cross-encoder, a second cross-encoder, and a third cross-encoder; the first cross-encoder, second cross-encoder, and third cross-encoder are connected serially. The image enhancement network includes a first enhancement decoder, a second enhancement decoder, and a third enhancement decoder; the first enhancement decoder, second enhancement decoder, and third enhancement decoder are connected serially. A first attention fusion network layer is provided between the first cross-encoder and the first enhancement decoder; a second attention fusion network layer is provided between the second cross-encoder and the second enhancement decoder; a third attention fusion network layer is provided between the third cross-encoder and the third enhancement decoder; an encoding network is provided after the third attention fusion network layer. The enhancement unit is further configured to: The surface image feature extraction network included in the uric acid detection image enhancement model is used to extract surface features from the initial uric acid detection image to generate surface uric acid detection image features. The surface uric acid detection image features are encoded using the multi-level image feature coding network included in the uric acid detection image enhancement model to generate uric acid detection coded image features. The uric acid detection image enhancement model includes an image enhancement network to enhance the uric acid detection encoded image features, thereby generating an enhanced uric acid detection image. The recognition unit is configured to identify the uric acid region in the enhanced uric acid detection image to obtain a group of uric acid recognition region images. The optimization unit is configured to optimize each uric acid recognition region image in the uric acid recognition region image group to generate an optimized uric acid recognition region image, thereby obtaining an optimized uric acid recognition region image group. The optimization of the uric acid recognition region image includes: removing small noise and isolated regions, and smoothing sub-region boundaries. The input unit is configured to input the historical uric acid detection dataset and the optimized uric acid identification region image group into a pre-trained multimodal uric acid anomaly detection model to generate uric acid anomaly detection results.

5. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

6. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the program, when executed by a processor, implements the method as described in any one of claims 1-3.

Images