Abnormality processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of image processing, and relates to an exception handling method and device, computer equipment and a storage medium, and the method comprises the following steps: fusing endoscopic image information and medical image information to obtain multi-modal fusion information; performing three-dimensional reconstruction to obtain a three-dimensional abnormal region model; performing abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; spectral features are extracted from the multi-modal fusion information for feature matching, and an abnormal property recognition result is obtained; performing information association according to the abnormal region identification result and the abnormal property identification result to obtain abnormal association information; performing region segmentation according to the abnormal associated information to obtain abnormal region contour structure information; and fusing the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generating an abnormal processing scheme. According to the method, the reliable and effective three-dimensional anomaly model can be generated according to the endoscopic image information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, specifically to the field of digital medicine, and in particular to an exception handling method, apparatus, computer equipment and storage medium. Background Art

[0002] In the field of digital medicine, endoscopy technology has become an important means of examining and treating a variety of diseases due to its non-invasive and intuitive characteristics. However, in practical applications, the processing and analysis of endoscopic video images faces many technical difficulties.

[0003] The imaging quality of endoscopic images is easily affected by the intracavitary environment. For example, digestive tract peristalsis and residual fluid can cause image blur and distortion, which not only increases the difficulty of image analysis, but also may mislead the doctor's judgment. In addition, due to the large differences in visual features of the lesion area, such as size, shape, color, etc., it is extremely difficult to establish a unified lesion recognition model. The movement and posture changes of the endoscopic probe are also a major challenge. The movement and rotation of the probe in the cavity will cause changes in the perspective and scale of the image sequence, which seriously interferes with subsequent analysis steps such as image registration and three-dimensional reconstruction. In order to obtain accurate recognition results, these changes must be accurately corrected. During endoscopic examination, the parameters of image acquisition are difficult to accurately control. The variability of physical conditions such as light intensity and spectrum leads to poor consistency in the collected image data. This not only affects the quality of the image, but also makes quantitative analysis difficult. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an exception handling method, apparatus, computer equipment and storage medium to solve the problem of being unable to effectively generate a reliable and effective three-dimensional exception model based on endoscopic image information and generate a corresponding exception handling solution.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an exception handling method, which adopts the following technical solution:

[0006] Acquiring endoscopic image information and medical imaging information, and fusing the endoscopic image information and the medical imaging information to obtain multimodal fusion information;

[0007] Performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model;

[0008] Perform abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result;

[0009] Extracting spectral features from the multimodal fusion information, and performing feature matching based on the spectral features to obtain an abnormal property identification result;

[0010] Performing spatial registration and information association according to the abnormal region recognition result and the abnormal property recognition result to obtain abnormal association information;

[0011] Segmenting the three-dimensional abnormal region according to the abnormal association information to obtain contour structure information of the abnormal region;

[0012] The abnormal region contour structure information and the three-dimensional abnormal region model are fused to obtain a three-dimensional abnormal model, and an abnormality processing solution is generated according to the three-dimensional abnormal model.

[0013] Furthermore, the step of fusing the endoscopic image information and the medical imaging information to obtain multimodal fusion information specifically includes:

[0014] Preprocessing the endoscopic image information and the medical image information to obtain standard endoscopic image information and standard medical image information;

[0015] Extracting features from the standard endoscopic image information and the standard medical image information to obtain endoscopic image features and medical image features;

[0016] The endoscopic image features and the medical image features are input into a pre-trained image fusion model for fusion to obtain the multimodal fusion information.

[0017] Furthermore, the step of performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal region model specifically includes:

[0018] Acquire voxel information of the abnormal area from the multimodal fusion information, and establish a three-dimensional voxel model according to the voxel information of the abnormal area;

[0019] Performing triangulation processing on the three-dimensional voxel model according to a light refraction algorithm to obtain a triangulated facet model;

[0020] Optimizing the coordinates of the triangular patch model according to a vertex coordinate optimization algorithm to obtain an optimized triangular patch model;

[0021] Mapping the multimodal fusion information to an optimized triangular patch model based on a texture mapping algorithm to obtain a textured triangular patch model;

[0022] The texture triangle patch model is subjected to illumination rendering to obtain the three-dimensional abnormal region model.

[0023] Furthermore, the step of identifying the abnormal region according to the three-dimensional abnormal region model to obtain the abnormal region identification result specifically includes:

[0024] Extracting geometric features and texture features from the three-dimensional abnormal region model;

[0025] Matching the geometric features and the texture features with a preset feature database according to a feature matching algorithm to obtain a feature matching result;

[0026] The feature matching results are identified and classified according to a preset classification algorithm to obtain the abnormal area identification result.

[0027] Furthermore, the step of extracting spectral features from the multimodal fusion information and performing feature matching according to the spectral features to obtain abnormal property identification results specifically includes:

[0028] Performing spectral analysis on the multimodal fusion information to obtain spectral information;

[0029] Extracting features from the spectral information based on a spectral analysis algorithm to obtain the spectral features;

[0030] Performing similarity matching between the spectral feature and a preset spectral feature database to obtain a spectral feature matching result;

[0031] The spectral feature matching result is classified into property types based on a support vector machine classification algorithm to obtain the abnormal property identification result.

[0032] Furthermore, the step of performing spatial registration and information association according to the abnormal region recognition result and the abnormal property recognition result to obtain abnormal association information specifically includes:

[0033] Performing spatial registration based on the abnormal region recognition result and the abnormal property recognition result based on a spatial registration algorithm to obtain a registration abnormal region result and a registration abnormal property result;

[0034] Obtaining a first confidence level of the registration abnormal region result and a second confidence level of the registration abnormal property result;

[0035] Determining whether the first confidence level and the second confidence level are both greater than or equal to a preset confidence level threshold;

[0036] If the first confidence level and the second confidence level are both greater than or equal to the confidence level threshold, the abnormal registration region result and the abnormal registration property result are associated to obtain the abnormal association information.

[0037] Furthermore, the step of segmenting the three-dimensional abnormal region according to the abnormal association information to obtain the contour structure information of the abnormal region specifically includes:

[0038] Extracting features of the three-dimensional abnormal region according to the abnormal association information to obtain a multi-scale feature representation of the abnormality;

[0039] Performing feature fusion on the abnormal multi-scale feature representation to obtain a fused feature map;

[0040] Segmenting the three-dimensional abnormal area according to the fused feature map based on a fully convolutional network to obtain an initial area contour;

[0041] Optimizing the initial region contour to obtain an optimized region contour, and extracting the tissue structure of the optimized region contour according to a region growing algorithm to obtain region structure information;

[0042] The optimized region contour and the region structure information are combined to obtain the abnormal region contour structure information.

[0043] In order to solve the above technical problems, the embodiment of the present application also provides an exception handling device, which adopts the following technical solution:

[0044] An information fusion module, used for acquiring endoscopic image information and medical image information, and fusing the endoscopic image information and the medical image information to obtain multimodal fusion information;

[0045] A three-dimensional reconstruction module, used for performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model;

[0046] A region identification module, used to identify abnormal regions according to the three-dimensional abnormal region model to obtain abnormal region identification results;

[0047] A property recognition module is used to extract spectral features from the multimodal fusion information and perform feature matching based on the spectral features to obtain an abnormal property recognition result;

[0048] An abnormality association module, used for performing spatial registration and information association according to the abnormal area identification result and the abnormality property identification result to obtain abnormality association information;

[0049] A region segmentation module, used to segment the three-dimensional abnormal region according to the abnormal association information to obtain the contour structure information of the abnormal region;

[0050] The solution generation module is used to fuse the abnormal area contour structure information and the three-dimensional abnormal area model to obtain a three-dimensional abnormal model, and generate an abnormality processing solution according to the three-dimensional abnormal model.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0052] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above exception handling methods when executing the computer-readable instructions.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0054] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of any of the above exception handling methods are implemented.

[0055] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment obtains endoscopic image information and medical imaging information, and fuses the endoscopic image information and the medical imaging information to obtain multimodal fusion information; performs three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal region model; performs abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; extracts spectral features from the multimodal fusion information, and performs feature matching according to the spectral features to obtain an abnormal property identification result; performs spatial registration and information association according to the abnormal region identification result and the abnormal property identification result to obtain abnormal association information; performs segmentation of the three-dimensional abnormal region according to the abnormal association information to obtain abnormal region contour structure information; fuses the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generates an abnormal processing scheme according to the three-dimensional abnormal model. Thus, a reliable and effective three-dimensional abnormal model is effectively generated according to the endoscopic image information and the medical imaging information, and a corresponding abnormal processing scheme is generated according to the three-dimensional abnormal model to facilitate subsequent abnormal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0058] Figure 2 A flowchart of an embodiment of an exception handling method according to the present application;

[0059] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S10;

[0060] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S20;

[0061] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S30;

[0062] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S40;

[0063] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S50;

[0064] Figure 8 yes Figure 2 A flowchart of a specific implementation of step S60;

[0065] Fig. 9 is a structural schematic diagram of an embodiment of an exception handling device according to the present application;

[0066] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0068] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0071] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0073] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0074] It should be noted that the exception handling method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the exception handling device is generally arranged in the server / terminal device.

[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0076] Continue to refer Figure 2 , shows a flow chart of an embodiment of an exception handling method according to the present application. The exception handling method comprises the following steps:

[0077] Step S10, acquiring endoscopic image information and medical imaging information, and fusing the endoscopic image information and the medical imaging information to obtain multimodal fusion information;

[0078] In this embodiment, the endoscopic image information refers to a video image collected by an endoscope. An endoscope is a medical device that is inserted into the patient's body through a slender tube. One end of the tube is equipped with a camera and a lighting device, which can capture and transmit the situation in the body to an external display in real time for image acquisition. Medical image information includes at least one of a computed tomography image, a magnetic resonance imaging image, and an ultrasound image. The fusion of the endoscopic image information and the medical image information is implemented based on a pre-trained image fusion model, wherein the image fusion model can adopt a convolutional neural network (CNN) model, and a fusion network is constructed in the model to perform weighted fusion of endoscopic image information and medical image information under different modalities, thereby effectively obtaining fused multi-modal fusion information.

[0079] Step S20, performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model;

[0080] In this embodiment, 3D reconstruction refers to the technology of restoring and reconstructing the 3D mathematical model of an object or scene based on information such as 2D images or 3D scanning data using computer technology. 3D reconstruction is performed by multimodal fusion information to effectively obtain a 3D abnormal region model that clearly displays the abnormal region.

[0081] Step S30, performing abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result;

[0082] In this embodiment, the abnormal region recognition result includes the type corresponding to the abnormal region, and the abnormal region refers to the region where abnormal conditions such as lesions occur. By extracting geometric features and texture features from the three-dimensional abnormal region model, and then performing feature matching based on the geometric features and texture features and a preset feature database to obtain matched feature data, and performing identification and classification based on the feature data, an abnormal region recognition result corresponding to the abnormal region of the three-dimensional abnormal region model is obtained.

[0083] Step S40, extracting spectral features from the multimodal fusion information, and performing feature matching based on the spectral features to obtain an abnormal property recognition result;

[0084] In this embodiment, spectral features are extracted based on spectral analysis. Spectral analysis is a method of inferring the composition and properties of a substance by measuring and analyzing its spectral characteristics at different wavelengths. Through spectral analysis processing, spectral information can be extracted from the fused multimodal fusion information, and then spectral features can be extracted from the spectral information according to the spectral analysis algorithm. The spectral features include a series of characteristic values ​​related to the spectral curve, such as peak wavelength, peak intensity, spectral shape parameters, etc. Feature matching based on spectral features is to perform similarity matching based on spectral features and a preset spectral feature database to obtain corresponding spectral feature matching results.

[0085] Step S50, performing spatial registration and information association according to the abnormal region recognition result and the abnormal property recognition result to obtain abnormal association information;

[0086] In this embodiment, spatial registration refers to a technology that determines and corrects the spatial position relationship between different data sets or images through a specific technology or system so that they can accurately correspond and overlap in the same coordinate system. By spatially registering the abnormal region recognition results and the abnormal property recognition results, and then associating information with the results obtained by spatial registration, abnormal association information is obtained.

[0087] Step S60, segmenting the three-dimensional abnormal region according to the abnormal association information to obtain contour structure information of the abnormal region;

[0088] In this embodiment, the segmentation of the three-dimensional abnormal region according to the abnormal association information is implemented based on a fully convolutional network, and a fully convolutional network (FCN) is used as a segmentation model. The model can process the feature map generated by the abnormal association information and output an initial region outline with the same size as the input. The initial region outline is optimized, and then the tissue feature extraction is performed based on the optimized initial region outline to obtain regional structure information. The optimized initial region outline and the regional structure information are combined to obtain the abnormal region outline structure information.

[0089] Step S70: Fusing the abnormal region contour structure information with the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generating an abnormality processing solution according to the three-dimensional abnormal model.

[0090] In this embodiment, the contour of the lesion area is determined according to the structural information of the contour of the abnormal area, and the contour of the lesion area is mapped to the corresponding position of the three-dimensional abnormal area model to obtain a three-dimensional abnormal model that integrates the lesion position information. In the process of generating the three-dimensional abnormal model, the tissue property information of the lesion area can be obtained. According to the tissue property information, different preset color mapping schemes are used to color the voxels of the lesion area in the three-dimensional abnormal model to obtain a three-dimensional visualization model that integrates the lesion tissue property information. The three-dimensional visualization model is then rendered and interactively processed to provide a preset number of types of visualization operation functions including rotation, scaling and sectioning, so as to effectively determine whether the abnormal area in the three-dimensional visualization model requires surgery. If surgery is required, the proposed surgical path and range are marked on the three-dimensional visualization model to generate an abnormality handling plan.

[0091] In this embodiment, the above method can be applied to a medical service system, in which a three-dimensional abnormality model is constructed based on endoscopic image information and medical imaging information, thereby effectively observing abnormal conditions in the patient's body and generating corresponding abnormality processing solutions. Specifically, in this embodiment, the medical service system can be one or more of a medical insurance system and a disease insurance system, the endoscopic image information and medical imaging information are image data collected by medical equipment, the endoscopic image information and medical imaging information are stored in the medical insurance system and the disease insurance system and are obtained from the database of the above system, and the three-dimensional abnormality model and abnormality processing solution are generated by the above system through processing by the method of this embodiment and stored in the database of the above system.

[0092] This embodiment obtains endoscopic image information and medical imaging information, and fuses the endoscopic image information and the medical imaging information to obtain multimodal fusion information; performs three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal region model; performs abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; extracts spectral features from the multimodal fusion information, and performs feature matching according to the spectral features to obtain an abnormal property identification result; performs spatial registration and information association according to the abnormal region identification result and the abnormal property identification result to obtain abnormal association information; segments the three-dimensional abnormal region according to the abnormal association information to obtain abnormal region contour structure information; fuses the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generates an abnormal processing scheme according to the three-dimensional abnormal model. Thus, a reliable and effective three-dimensional abnormal model is effectively generated according to the endoscopic image information and the medical imaging information, and a corresponding abnormal processing scheme is generated according to the three-dimensional abnormal model to facilitate subsequent abnormal processing.

[0093] refer to Figure 3 In some optional implementations of this embodiment, step S10 includes the following steps:

[0094] Step S101, preprocessing the endoscopic image information and the medical image information to obtain standard endoscopic image information and standard medical image information;

[0095] In this embodiment, the preprocessing of the endoscopic image information includes denoising, contrast enhancement, color correction, etc., and the preprocessing of the medical image information includes removing artifacts, contrast adjustment, etc. By performing the above preprocessing steps on the endoscopic image information and the medical image information, standard endoscopic image information and standard medical image information are effectively obtained.

[0096] Step S102, extracting features from the standard endoscopic image information and the standard medical image information to obtain endoscopic image features and medical image features;

[0097] In this embodiment, endoscopic image features include shape, texture, color, etc., medical image features include anatomical structure, tissue density, lesion area, etc. Feature extraction can be achieved by using filters, edge detection, texture analysis, deep learning and other methods.

[0098] Step S103: Input the endoscopic image features and the medical image features into a pre-trained image fusion model for fusion to obtain the multimodal fusion information.

[0099] In this embodiment, the multimodal fusion information is image data that fuses endoscopic image information and medical imaging information. The pre-trained image fusion model can use a convolutional neural network model. By inputting endoscopic image features and medical imaging features into the pre-trained convolutional neural network model, the model will fuse these features into a new representation, namely, multimodal fusion information, according to the learned weights and rules.

[0100] In this embodiment, the endoscopic image information and the medical image information are preprocessed to obtain standard endoscopic image information and standard medical image information; feature extraction is performed on the standard endoscopic image information and the standard medical image information to obtain endoscopic image features and medical image features; the endoscopic image features and the medical image features are input into a pre-trained image fusion model for fusion, thereby obtaining multimodal fusion information that effectively fuses the endoscopic image information and the medical image information, so as to provide reliable data support for subsequent three-dimensional reconstruction.

[0101] refer to Figure 4 In some optional implementations of this embodiment, step S20 includes the following steps:

[0102] Step S201, obtaining abnormal region voxel information from the multimodal fusion information, and establishing a three-dimensional voxel model according to the abnormal region voxel information;

[0103] In this embodiment, the multimodal fusion information includes abnormal areas, and the features of the abnormal areas are extracted using image processing techniques (such as edge detection, texture analysis, morphological operations, etc.), and the extracted features of the abnormal areas are converted into voxel information. Among them, voxels are the basic units in three-dimensional space, similar to pixels in two-dimensional images. By creating an empty three-dimensional grid as the initial structure of the model, the resolution and size of the grid are determined to ensure that the abnormal area can be accurately represented, and the screened voxel information of the abnormal area is filled into the three-dimensional grid. The grid is colored or marked according to the properties of the voxels (such as density, color, etc.), and the three-dimensional voxel model is smoothed to reduce jagged edges and uneven areas in the grid. Thus, a three-dimensional voxel model is obtained.

[0104] Step S202, triangulating the three-dimensional voxel model according to a light refraction algorithm to obtain a triangular facet model;

[0105] In this embodiment, the light refraction algorithm refers to a voxel to triangle face conversion algorithm, which generates a smoother triangle face representation by simulating the propagation and refraction of light in the voxel model. The triangle face model is composed of many triangles, which can more accurately represent the surface of a three-dimensional object.

[0106] Step S203, optimizing the coordinates of the triangular patch model according to a vertex coordinate optimization algorithm to obtain an optimized triangular patch model;

[0107] In this embodiment, a vertex coordinate optimization algorithm is applied to the triangular patch model to improve its geometric accuracy and visual effect. The coordinate optimization algorithm includes smoothing, Laplace smoothing, mesh simplification and other technologies, aiming to reduce noise and redundancy in the model while maintaining its key features. The vertex coordinates of the triangular patch model are optimized by the coordinate optimization algorithm so that the normal vector directions between adjacent triangular patches are consistent, thereby obtaining an optimized triangular patch model. The optimized triangular patch model has a smoother surface and fewer vertices, thereby improving rendering efficiency and visual effects.

[0108] Step S204, mapping the multimodal fusion information to an optimized triangular patch model based on a texture mapping algorithm to obtain a textured triangular patch model;

[0109] In this embodiment, a texture mapping algorithm is used to map the texture information in the multimodal fusion information to the optimized triangular patch model. Texture mapping is a technique for applying a two-dimensional image (texture) to the surface of a three-dimensional model to increase the details and realism of the model. In the above steps, the color, texture, pattern and other features in the multimodal fusion information are mapped to the surface of the triangular patch, thereby generating a textured triangular patch model with rich details.

[0110] Step S205, performing lighting rendering on the texture triangle patch model to obtain the three-dimensional abnormal region model.

[0111] In this embodiment, lighting rendering includes simulating the illumination effect of light source on the model surface, and calculating visual effects such as shadows and reflections on the model surface. The textured triangular patch model is processed through lighting rendering to make the model present a more realistic appearance and richer details, thereby more accurately reflecting the characteristics of the abnormal area.

[0112] This embodiment obtains the abnormal area voxel information from the multimodal fusion information, and establishes a three-dimensional voxel model according to the abnormal area voxel information; triangulates the three-dimensional voxel model according to a light refraction algorithm to obtain a triangular patch model; optimizes the coordinates of the triangular patch model according to a vertex coordinate optimization algorithm to obtain an optimized triangular patch model; maps the multimodal fusion information to the optimized triangular patch model based on a texture mapping algorithm to obtain a textured triangular patch model; and performs illumination rendering on the textured triangular patch model, thereby effectively obtaining a three-dimensional abnormal area model that effectively reflects the characteristics of the abnormal area, so as to facilitate subsequent abnormal area identification processing.

[0113] refer to Figure 5 In some optional implementations of this embodiment, step S30 includes the following steps:

[0114] Step S301, extracting geometric features and texture features from the three-dimensional abnormal region model;

[0115] In this embodiment, the geometric features include volume: the size of the space occupied by the three-dimensional abnormal region; surface area: the outer surface area of ​​the three-dimensional abnormal region; shape parameters: such as aspect ratio, roundness, sphericity, etc., used to describe the shape characteristics of the three-dimensional abnormal region; spatial position: the position information of the three-dimensional abnormal region in the three-dimensional space, such as coordinates, direction, etc.; directional features: such as normals, curvature, etc., used to describe the directionality of the surface of the three-dimensional abnormal region. Texture features include color features: describing the color information of the surface of the three-dimensional abnormal region, such as color histogram, color moment, etc.; texture pattern: describing the texture pattern of the surface of the three-dimensional abnormal region, such as texture primitives, texture period, etc.; texture direction: describing the directional characteristics of the surface texture of the three-dimensional abnormal region; texture frequency: describing the coarseness or density of the surface texture of the three-dimensional abnormal region; statistical features: such as the energy, inertia, entropy and correlation of the gray level co-occurrence matrix (GLCM), used to describe the statistical characteristics of the texture.

[0116] Step S302, matching the geometric features and the texture features with a preset feature database according to a feature matching algorithm to obtain a feature matching result;

[0117] In this embodiment, the preset feature database refers to a preset abnormal region feature database, and the feature matching algorithm can use the nearest neighbor search, and the nearest neighbor search algorithm is used to perform similarity matching in the abnormal region feature database with geometric features and texture features to find the abnormal region features with the closest similarity. Similarity matching can be achieved by similarity calculation, specifically by calculating the cosine similarity between the geometric features and texture features and the features in the abnormal region feature database, thereby effectively finding the closest abnormal region features as feature matching results.

[0118] Step S303: classify the feature matching results according to a preset classification algorithm to obtain the abnormal area recognition result.

[0119] In this embodiment, the preset classification algorithm may adopt a random forest, and the random forest algorithm is used to identify and classify abnormal regions according to the features included in the feature matching results to obtain an abnormal region identification result indicating the abnormal type corresponding to the abnormal region.

[0120] This embodiment extracts geometric features and texture features from the three-dimensional abnormal region model; matches the geometric features and the texture features with a preset feature database according to a feature matching algorithm to obtain a feature matching result; identifies and classifies the feature matching result according to a preset classification algorithm, thereby effectively obtaining an abnormal region identification result representing the abnormal type of the abnormal region, so as to facilitate subsequent spatial registration processing.

[0121] Continue to refer Figure 6In some optional implementations of this embodiment, step S40 includes the following steps:

[0122] Step S401, performing spectral analysis on the multimodal fusion information to obtain spectral information;

[0123] In this embodiment, the spectrum analysis process includes the steps of spectrum data extraction, spectrum correction, spectrum smoothing, spectrum derivative, spectrum difference analysis, etc., and spectrum data including wavelength (or frequency) and corresponding intensity value are obtained by performing spectrum data extraction on multimodal fusion information. Then, spectrum data is processed by performing spectrum correction, spectrum smoothing, spectrum derivative, spectrum difference analysis, etc. on the spectrum data to obtain effective spectrum data. Distinguishing features such as peak position, peak intensity, valley position, valley intensity, etc. are extracted from the effective spectrum data to obtain spectrum information.

[0124] Step S402, extracting features from the spectral information based on a spectral analysis algorithm to obtain the spectral features;

[0125] In this embodiment, the spectral features include spectral peak features, spectral interval features, spectral wavelength features, spectral wavelength features, etc. The step of extracting spectral features includes peak analysis: identifying the peak in the spectral curve and recording its position, intensity, shape and other information. Spectral band analysis: dividing the spectral curve into different intervals according to wavelength, and analyzing the characteristics of each interval. Feature wavelength selection: using algorithms such as continuous projection algorithm (SPA), uninformation variable elimination method (UVE), adaptive reweighted sampling method (CARS), etc., to screen out the most distinguishing feature wavelengths for classification or identification. Feature spectrum area extraction: Spectral feature extraction methods based on wavelength intervals, such as interval partial least squares (iPLS), joint interval partial least squares (siPLS), etc., extract characteristic spectral areas with significant distinguishing properties.

[0126] Step S403, performing similarity matching between the spectral feature and a preset spectral feature database to obtain a spectral feature matching result;

[0127] In this embodiment, the preset spectral feature database includes spectral features of known substances or abnormal properties that are pre-collected and organized. Each entry in the database should contain the name of the substance, spectral features (such as peak position, intensity, etc.) and the corresponding property type. For spectral features, matching is performed by calculating the similarity between them and each entry in the preset spectral feature database. Similarity calculation methods may include Euclidean distance, cosine similarity, Manhattan distance, etc. The specific selection depends on the nature and distribution of the spectral features. In this embodiment, the similarity calculation method may use cosine similarity. According to the similarity calculation results, the preset spectral feature entry with the highest similarity to the spectral feature to be identified is screened out as the spectral feature matching result.

[0128] Step S404, classifying the spectral feature matching result by property type based on a support vector machine classification algorithm to obtain the abnormal property identification result.

[0129] In this embodiment, the support vector machine classification algorithm is implemented by an SVM classification model. The SVM classification model is trained by using a spectral feature data set of known property types so that the SVM learns how to map the spectral features to the corresponding property types to form a classification decision boundary. The spectral feature matching result is used as the input feature vector of the SVM classification model for classification prediction. The SVM classification model outputs the corresponding property type label based on the relationship between the input feature vector and the classification decision boundary. According to the output result of the SVM classification model, the property type of the spectral feature to be identified is determined. If the output property type matches the preset abnormal property type, it is identified as an abnormal property; otherwise, it is identified as a normal property. After the spectral feature matching result is input into the SVM classification model, the identified abnormal property or normal property is output as the abnormal property identification result.

[0130] This embodiment obtains spectral information by performing spectral analysis on the multimodal fusion information; extracts features from the spectral information based on a spectral analysis algorithm to obtain the spectral features; performs similarity matching between the spectral features and a preset spectral feature database to obtain spectral feature matching results; and classifies the spectral feature matching results into property types based on a support vector machine classification algorithm, thereby obtaining an abnormal property recognition result that effectively represents the properties of abnormal diseased tissue, so as to facilitate subsequent spatial registration processing.

[0131] Continue to refer Figure 7 In some optional implementations of this embodiment, step S50 includes the following steps:

[0132] Step S501, performing spatial registration according to the abnormal region recognition result and the abnormal property recognition result based on a spatial registration algorithm to obtain a registered abnormal region result and a registered abnormal property result;

[0133] In this embodiment, the spatial registration algorithm can adopt feature-based registration, by extracting features for registration from the abnormal region and abnormal property data, the registered features can be obvious geometric features such as boundaries, intersections of linear objects, and regional contours, or spectral features, texture features, etc. Then, based on the extracted features, a mapping relationship between the abnormal region and the abnormal property is established, so as to be realized by calculating the similarity measure between the features (such as Euclidean distance, cosine similarity, etc.). According to the mapping relationship, the abnormal region is spatially transformed so that it is spatially aligned with the abnormal property. The spatial transformation may include operations such as translation, rotation, and scaling. The registration process can be implemented by an iterative optimization algorithm to minimize the registration error. After the spatial registration of the above steps is performed, the registration processing of the abnormal region recognition result and the abnormal property recognition result is completed, and the registration abnormal region result and the registration abnormal property result are obtained.

[0134] Step S502, obtaining a first confidence level of the abnormal region registration result and a second confidence level of the abnormal property registration result;

[0135] In this embodiment, the first confidence level can be evaluated based on the accuracy of abnormal region identification, the accuracy of spatial registration, the contrast between the abnormal region and the surrounding environment, etc. The second confidence level can be calculated based on factors such as the accuracy of abnormal property identification, the reliability of spectral feature matching, and the performance of the SVM classification model. The evaluation of the first confidence level and the second confidence level can be implemented using a machine learning algorithm, which can use a logistic regression algorithm or a support vector machine.

[0136] Step S503, determining whether the first confidence level and the second confidence level are both greater than or equal to a preset confidence level threshold;

[0137] In this embodiment, the confidence threshold includes a first confidence threshold and a second confidence threshold, wherein the first confidence threshold corresponds to the first confidence setting, and the second confidence threshold corresponds to the second confidence setting. The first confidence is numerically compared with the first confidence threshold, and the second confidence is numerically compared with the second confidence threshold, and the conditional judgment is performed by comparing them respectively.

[0138] Step S504: if the first confidence and the second confidence are both greater than or equal to the confidence threshold, the abnormal registration region result and the abnormal registration property result are associated to obtain the abnormal association information;

[0139] In this embodiment, when the first confidence is greater than or equal to the first confidence threshold, and the second confidence is greater than or equal to the second confidence threshold, the corresponding judgment condition of this embodiment is established.

[0140] By extracting key features from the registration abnormal region results, such as the boundary, center point, area, etc. of the region. Extract features from the registration abnormal property results, such as the type of property, peak value, intensity, etc. of the spectral feature. Then, the features of the registration abnormal region results and the features of the registration abnormal property results are matched based on a similarity matching algorithm. The similarity-based matching algorithm can be calculated using cosine similarity. According to the similarity value calculated by cosine similarity, the registration abnormal region results and the features with high similarity in the registration abnormal property results are matched and associated to obtain abnormal association information.

[0141] Step S505, if the first confidence level and the second confidence level are not both greater than or equal to the confidence threshold, then re-perform spatial alignment based on the abnormal area identification result and the abnormal property identification result to obtain the re-aligned first adjusted confidence level and the second adjusted confidence level, and determine whether the first adjusted confidence level and the second adjusted confidence level are both greater than or equal to the confidence threshold, until the first adjusted confidence level and the second adjusted confidence level are both greater than or equal to the confidence threshold.

[0142] In this embodiment, the registration algorithm parameters in the spatial registration are adjusted, such as the transformation parameters, the matching threshold, etc., to improve the registration accuracy and re-register the abnormal region recognition result and the abnormal property recognition result.

[0143] This embodiment performs spatial registration based on the abnormal region recognition result and the abnormal property recognition result based on a spatial registration algorithm to obtain a registered abnormal region result and a registered abnormal property result; obtains a first confidence of the registered abnormal region result and a second confidence of the registered abnormal property result; determines whether the first confidence and the second confidence are both greater than or equal to a preset confidence threshold; if the first confidence and the second confidence are both greater than or equal to the confidence threshold, associates the registered abnormal region result and the registered abnormal property result to obtain the abnormal association information. If the first confidence and the second confidence are not both greater than or equal to the confidence threshold, spatial registration is performed again based on the abnormal region recognition result and the abnormal property recognition result to obtain a first adjusted confidence and a second adjusted confidence after re-registration, and determines whether the first adjusted confidence and the second adjusted confidence are both greater than or equal to the confidence threshold, until the first adjusted confidence and the second adjusted confidence are both greater than or equal to the confidence threshold. Thereby, the abnormal region recognition result and the abnormal property recognition result are effectively associated to obtain the associated abnormality association information, which is convenient for the subsequent segmentation of the abnormal region.

[0144] Continue to refer Figure 8 In some optional implementations of this embodiment, step S60 includes the following steps:

[0145] S601, extracting features of the three-dimensional abnormal region according to the abnormal association information to obtain an abnormal multi-scale feature representation;

[0146] In this embodiment, the abnormal multi-scale feature representation includes abnormal feature representations of multiple different scales of the three-dimensional abnormal area. The three-dimensional abnormal area is scanned by using different scales (such as windows, filters or grids of different sizes), and features are extracted at each scale. These features may include geometric features (such as volume, surface area), texture features (such as gray-level co-occurrence matrix), statistical features (such as mean, variance), etc. The features at different scales are combined to form an abnormal multi-scale feature representation.

[0147] S602, performing feature fusion on the abnormal multi-scale feature representation to obtain a fused feature map;

[0148] In this embodiment, the abnormal feature representations of different scales in the abnormal multi-scale feature representation are effectively fused in the form of feature splicing to form a fused feature map. Before performing feature splicing, it is necessary to determine the dimension of splicing. If the abnormal feature representation is a two-dimensional matrix (for example, the feature representation at each scale is a matrix), the splicing can be performed on rows or columns. If the abnormal feature representation is a three-dimensional or higher-dimensional tensor, it is necessary to select an appropriate dimension for splicing. According to the determined splicing dimension, the feature representations at different scales are stacked or connected together on the specified dimension to organize a fused feature map, which is a comprehensive representation containing feature information of multiple scales.

[0149] S603, segmenting the three-dimensional abnormal area according to the fused feature map based on a fully convolutional network to obtain an initial area contour;

[0150] In this embodiment, the fused feature map is used as the input of the full convolutional network and the three-dimensional abnormal area is segmented by the forward propagation of the full convolutional network to obtain the initial area contour, which is the contour information of the specific abnormal area part in the three-dimensional abnormal area.

[0151] S604, optimizing the initial region contour to obtain an optimized region contour, and extracting the tissue structure of the optimized region contour according to a region growing algorithm to obtain region structure information;

[0152] In this embodiment, the initial region contour is optimized by a morphological processing method, and small noises and breaks in the initial region contour are eliminated to obtain a smooth optimized region contour. The step of extracting tissue structure according to the region growing algorithm includes determining the growth starting point: selecting a point on the optimized region contour as the starting point of region growing. Define the growth criterion: determine the similarity measure or criterion used in the region growing process, which may include the similarity of features such as grayscale value, color, texture, shape, etc. Perform region growing: starting from the starting point, gradually expand the region according to the growth criterion. In each step, check whether the points in the neighborhood meet the growth criterion. If so, add the point to the grown region, and continue to check its neighborhood. Repeat this process until no more points meet the growth criterion. Obtain regional structure information: After the region growing is completed, record the regional structure information of the grown region. The regional structure information may include the shape, size, connectivity, internal features, etc. of the region.

[0153] S605: Combine the optimized region contour and the region structure information to obtain the abnormal region contour structure information.

[0154] In this embodiment, the region structure information can be attached as an attribute or metadata to the contour points of the optimized region contour to combine the optimized region contour and the region structure information to obtain abnormal region contour structure information that effectively describes the contour and structure of the abnormal region.

[0155] This embodiment extracts features from the three-dimensional abnormal area according to the abnormal association information to obtain an abnormal multi-scale feature representation; performs feature fusion on the abnormal multi-scale feature representation to obtain a fused feature map; segments the three-dimensional abnormal area according to the fused feature map based on a fully convolutional network to obtain an initial area contour; optimizes the initial area contour to obtain an optimized area contour, and extracts the tissue structure of the optimized area contour according to a regional growing algorithm to obtain regional structure information; combines the optimized area contour and the regional structure information, thereby effectively obtaining the abnormal area contour structure information that describes the specific contour and structure of the abnormal area, so as to facilitate the subsequent construction of a three-dimensional abnormal model.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0157] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0158] Further references Fig. 9 , as a response to the above Figure 1 The present application provides an embodiment of an exception handling device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0159] like Fig. 9 As shown, the exception handling device 800 described in this embodiment includes: an information fusion module 801, a three-dimensional reconstruction module 802, a region identification module 803, a property identification module 804, an exception association module 805, a region segmentation module 806, and a solution generation module 807. Among them:

[0160] The information fusion module 801 is used to obtain endoscopic image information and medical image information, and fuse the endoscopic image information and the medical image information to obtain multimodal fusion information;

[0161] A three-dimensional reconstruction module 802 is used to perform three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model;

[0162] A region identification module 803 is used to identify abnormal regions according to the three-dimensional abnormal region model to obtain abnormal region identification results;

[0163] The property identification module 804 is used to extract spectral features from the multimodal fusion information, and perform feature matching based on the spectral features to obtain an abnormal property identification result;

[0164] An abnormality association module 805 is used to perform spatial registration and information association according to the abnormal region identification result and the abnormality property identification result to obtain abnormality association information;

[0165] A region segmentation module 806 is used to segment the three-dimensional abnormal region according to the abnormal association information to obtain the contour structure information of the abnormal region;

[0166] The solution generation module 807 is used to fuse the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generate an abnormality processing solution according to the three-dimensional abnormal model.

[0167] By adopting the above-mentioned abnormality processing device, this embodiment can obtain endoscopic image information and medical imaging information, and fuse the endoscopic image information and the medical imaging information to obtain multimodal fusion information; perform three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal region model; perform abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; extract spectral features from the multimodal fusion information, and perform feature matching according to the spectral features to obtain an abnormal property identification result; perform spatial registration and information association according to the abnormal region identification result and the abnormal property identification result to obtain abnormal association information; segment the three-dimensional abnormal region according to the abnormal association information to obtain abnormal region contour structure information; fuse the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generate an abnormality processing scheme according to the three-dimensional abnormal model. Thereby, a reliable and effective three-dimensional abnormal model is effectively generated according to the endoscopic image information and the medical imaging information, and a corresponding abnormality processing scheme is generated according to the three-dimensional abnormal model to facilitate subsequent abnormality processing.

[0168] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0169] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 with components 91-93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0170] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0171] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of the exception handling method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0172] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run computer-readable instructions or process data stored in the memory 91, such as computer-readable instructions for running the exception handling method.

[0173] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0174] By adopting the above-mentioned computer device, this embodiment can obtain endoscopic image information and medical imaging information, and fuse the endoscopic image information and the medical imaging information to obtain multimodal fusion information; perform three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal region model; perform abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; extract spectral features from the multimodal fusion information, and perform feature matching according to the spectral features to obtain an abnormal property identification result; perform spatial registration and information association according to the abnormal region identification result and the abnormal property identification result to obtain abnormal association information; segment the three-dimensional abnormal region according to the abnormal association information to obtain abnormal region contour structure information; fuse the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generate an abnormal processing scheme according to the three-dimensional abnormal model. Thereby, a reliable and effective three-dimensional abnormal model is effectively generated according to the endoscopic image information and the medical imaging information, and a corresponding abnormal processing scheme is generated according to the three-dimensional abnormal model to facilitate subsequent abnormal processing.

[0175] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the exception handling method as described above.

[0176] By adopting the above-mentioned computer-readable storage medium, this embodiment can obtain endoscopic image information and medical imaging information, and fuse the endoscopic image information and the medical imaging information to obtain multimodal fusion information; perform three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal region model; perform abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; extract spectral features from the multimodal fusion information, and perform feature matching according to the spectral features to obtain an abnormal property identification result; perform spatial registration and information association according to the abnormal region identification result and the abnormal property identification result to obtain abnormal association information; segment the three-dimensional abnormal region according to the abnormal association information to obtain abnormal region contour structure information; fuse the abnormal region contour structure information and the three-dimensional abnormal region model to obtain a three-dimensional abnormal model, and generate an abnormal processing scheme according to the three-dimensional abnormal model. Thereby, a reliable and effective three-dimensional abnormal model is effectively generated according to the endoscopic image information and the medical imaging information, and a corresponding abnormal processing scheme is generated according to the three-dimensional abnormal model to facilitate subsequent abnormal processing.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0179] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. An exception handling method, characterized in that: The steps include: Acquiring endoscopic image information and medical imaging information, and fusing the endoscopic image information and the medical imaging information to obtain multimodal fusion information; Performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model; Perform abnormal region identification according to the three-dimensional abnormal region model to obtain an abnormal region identification result; Extracting spectral features from the multimodal fusion information, and performing feature matching based on the spectral features to obtain an abnormal property identification result; Performing spatial registration and information association according to the abnormal region recognition result and the abnormal property recognition result to obtain abnormal association information; Segmenting the three-dimensional abnormal region according to the abnormal association information to obtain contour structure information of the abnormal region; The abnormal region contour structure information and the three-dimensional abnormal region model are fused to obtain a three-dimensional abnormal model, and an abnormality processing solution is generated according to the three-dimensional abnormal model.

2. The exception handling method according to claim 1, characterized in that: The step of fusing the endoscopic image information and the medical imaging information to obtain multimodal fusion information specifically includes: Preprocessing the endoscopic image information and the medical image information to obtain standard endoscopic image information and standard medical image information; Extracting features from the standard endoscopic image information and the standard medical image information to obtain endoscopic image features and medical image features; The endoscopic image features and the medical image features are input into a pre-trained image fusion model for fusion to obtain the multimodal fusion information.

3. The exception handling method according to claim 1, characterized in that: The step of performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model specifically includes: Acquire voxel information of the abnormal area from the multimodal fusion information, and establish a three-dimensional voxel model according to the voxel information of the abnormal area; Performing triangulation processing on the three-dimensional voxel model according to a light refraction algorithm to obtain a triangulated facet model; Optimizing the coordinates of the triangular patch model according to a vertex coordinate optimization algorithm to obtain an optimized triangular patch model; Mapping the multimodal fusion information to an optimized triangular patch model based on a texture mapping algorithm to obtain a textured triangular patch model; The texture triangle patch model is subjected to illumination rendering to obtain the three-dimensional abnormal region model.

4. The exception handling method according to claim 1, characterized in that: The step of identifying the abnormal region according to the three-dimensional abnormal region model to obtain the abnormal region identification result specifically includes: Extracting geometric features and texture features from the three-dimensional abnormal region model; Matching the geometric features and the texture features with a preset feature database according to a feature matching algorithm to obtain a feature matching result; The feature matching results are identified and classified according to a preset classification algorithm to obtain the abnormal area identification result.

5. The exception handling method according to claim 1, characterized in that: The step of extracting spectral features from the multimodal fusion information and performing feature matching according to the spectral features to obtain an abnormal property identification result specifically includes: Performing spectral analysis on the multimodal fusion information to obtain spectral information; Extracting features from the spectral information based on a spectral analysis algorithm to obtain the spectral features; Performing similarity matching between the spectral feature and a preset spectral feature database to obtain a spectral feature matching result; The spectral feature matching result is classified into property types based on a support vector machine classification algorithm to obtain the abnormal property identification result.

6. The exception handling method according to claim 1, characterized in that: The step of performing spatial registration and information association according to the abnormal region recognition result and the abnormal property recognition result to obtain abnormal association information specifically includes: Performing spatial registration based on the abnormal region recognition result and the abnormal property recognition result based on a spatial registration algorithm to obtain a registration abnormal region result and a registration abnormal property result; Obtaining a first confidence level of the registration abnormal region result and a second confidence level of the registration abnormal property result; Determining whether the first confidence level and the second confidence level are both greater than or equal to a preset confidence level threshold; If the first confidence level and the second confidence level are both greater than or equal to the confidence level threshold, the abnormal registration region result and the abnormal registration property result are associated to obtain the abnormal association information.

7. The exception handling method according to claim 1, characterized in that: The step of segmenting the three-dimensional abnormal region according to the abnormal association information to obtain the contour structure information of the abnormal region specifically includes: Extracting features of the three-dimensional abnormal region according to the abnormal association information to obtain a multi-scale feature representation of the abnormality; Performing feature fusion on the abnormal multi-scale feature representation to obtain a fused feature map; Segmenting the three-dimensional abnormal area according to the fused feature map based on a fully convolutional network to obtain an initial area contour; Optimizing the initial region contour to obtain an optimized region contour, and extracting the tissue structure of the optimized region contour according to a region growing algorithm to obtain region structure information; The optimized region contour and the region structure information are combined to obtain the abnormal region contour structure information.

8. An exception handling device, characterized in that: include: An information fusion module, used for acquiring endoscopic image information and medical image information, and fusing the endoscopic image information and the medical image information to obtain multimodal fusion information; A three-dimensional reconstruction module, used for performing three-dimensional reconstruction based on the multimodal fusion information to obtain a three-dimensional abnormal area model; A region identification module, used to identify abnormal regions according to the three-dimensional abnormal region model to obtain abnormal region identification results; A property recognition module is used to extract spectral features from the multimodal fusion information and perform feature matching based on the spectral features to obtain an abnormal property recognition result; An abnormality association module, used for performing spatial registration and information association according to the abnormal area identification result and the abnormality property identification result to obtain abnormality association information; A region segmentation module, used to segment the three-dimensional abnormal region according to the abnormal association information to obtain the contour structure information of the abnormal region; The solution generation module is used to fuse the abnormal area contour structure information and the three-dimensional abnormal area model to obtain a three-dimensional abnormal model, and generate an abnormality processing solution according to the three-dimensional abnormal model.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the exception handling method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the exception handling method according to any one of claims 1 to 7 are implemented.

Images