Hysteroscope image target detection method and system

Abstract

The invention relates to the technical field of hysteroscopes, and provides a hysteroscope image target detection method and system.The method comprises the steps that a hysteroscope is inserted into a uterine cavity of a patient, and high-definition images and environment data of tissue in the uterine cavity are collected in real time; performing automatic analysis by using an image recognition technology, recognizing different types of lesion tissues, and automatically performing lesion assessment to obtain first assessment data; according to the physiological data set of the patient, the future development potential of the lesion is analyzed and evaluated through association of a multi-level neural network model, risk grade division is carried out on a lesion area, and second evaluation data is generated; and automatically generating a risk assessment report according to the second assessment data. According to the invention, a targeted risk assessment report can be output objectively and automatically, and reliable data support and technical support are provided for scientific assessment and intelligent management of intrauterine lesions.

Description

Technical Field

[0001] The present invention relates to the technical field of hysteroscopy, and in particular, to a method and system for detecting target images of hysteroscopy. Background Art

[0002] With the development of medical technology, hysteroscopy technology, as a precise diagnosis and treatment tool, has been widely used in the field of gynecology. Hysteroscopy can directly observe the lesions in the uterine cavity and help doctors make diagnoses through high-definition images. Traditional hysteroscopy mainly relies on manual operation and empirical judgment for lesion localization, etc. Although certain success has been achieved in some aspects of this technology, it still faces problems such as insufficient operation accuracy and inconsistent diagnostic criteria.

[0003] There are still some obvious deficiencies in the current hysteroscopy technology. On the one hand, although image recognition technology can help doctors identify the lesion area, its processing speed and accuracy are limited by the image quality and the real-time nature of data analysis. In addition, most of the existing lesion assessment systems rely on manual analysis and lack the ability to monitor and predict the dynamic evolution trend of lesions in real time. Summary of the Invention

[0004] Embodiments of the present invention provide a method and system for detecting target images of hysteroscopy, which can at least to a certain extent improve the accuracy and efficiency of target image detection of hysteroscopy and avoid the problem of differences caused by human factors in traditional methods.

[0005] Other characteristics and advantages of the present invention will become obvious through the following detailed description, or will be learned in part through the practice of the present invention.

[0006] According to one aspect of the present invention, there is provided a method for detecting target images of hysteroscopy, including: inserting a hysteroscope into the uterine cavity of a patient to collect high-definition images of the tissues in the uterine cavity in real time to obtain first image data; and collecting environmental data in the uterine cavity through a sensor installed on the hysteroscope to obtain first environmental data; automatically analyzing the first image data by using image recognition technology to identify different types of lesion tissues to obtain a first recognition result; combining the first recognition result with the first environmental data to automatically perform lesion assessment to obtain first assessment data; the first assessment data includes the infection degree, inflammation degree of the lesion, and the size and distribution of the lesion area; based on the first assessment data, according to the patient's physiological data set, associatively analyzing and evaluating the future development potential of the lesion through a multi-level neural network model, classifying the risk level of the lesion area, and generating second assessment data; automatically generating a risk assessment report according to the second assessment data.

[0007] In the present invention, based on the foregoing solution, the identification of different types of diseased tissues includes: automatically analyzing the diseased regions of the preprocessed first image data by using a convolutional neural network, identifying and segmenting the diseased regions in the uterine cavity, and the steps are as follows: the input image extracts high-level features through multiple convolutional layers and pooling layers, and finally obtains the probability map of the diseased regions through a fully connected layer; combining edge detection and region growing strategies to assist in segmenting the boundaries and improving the segmentation accuracy; the output segmentation result is a binary image, where the diseased regions are marked as 1 and the other parts are marked as 0; using the diseased regions in the segmentation result to perform object detection tasks in the image to determine the specific types of the diseases; using a classification network to classify the types of each diseased region, and obtaining the positions and sizes of each diseased region by calculating the areas and surrounding boundaries of the segmented regions.

[0008] In the present invention, based on the foregoing solution, the patient physiological data set includes the patient's genetic information and immune system status data.

[0009] In the present invention, based on the foregoing solution, the multi-level neural network model processes image data and structured data; the structured data includes the genetic information and immune system status data.

[0010] In the present invention, based on the foregoing solution, the construction of the multi-level neural network model includes the following operating steps: through a multi-scale feature fusion mechanism, fusing the features of the structured data and the image data; the feature extraction of the image data should consider not only global features, but also local features and multi-scale features: performing multi-scale processing on the image data through a pyramid convolutional neural network to obtain a multi-level spatial feature representation:

[0011]

[0012] Among them, is the image feature obtained after being processed by the pyramid convolutional network, is the k-scale feature of the image data, W k is the k-th layer convolutional kernel, K is the number of scales of the multi-scale convolutional operation in the image data; designing a feature fusion layer to jointly model the image data and the structured data to obtain the fused feature F fu , and introducing a coupling loss function to ensure the semantic consistency between different modality data.

[0013] In the present invention, based on the foregoing solution, the evaluation of the future development potential of the lesion through the correlation analysis of the multi-level neural network model includes: processing the fused feature F fu through a deep neural network to output the probability of the future development potential of the lesion, and the specific calculation formula is:

[0014] ypred = Softmax(W × F fu + b)

[0015] where y pred is the predicted probability of lesion development, W is the weight matrix of the neural network, and b is the bias term.

[0016] In the present invention, based on the foregoing solution, the risk level division of the lesion area includes the following steps: The risks include local risks and global risks of the lesion; the local risk score R local is evaluated through the result of image segmentation. Based on edge detection and region growing, the change rate of the lesion area is used to calculate the local risk:

[0017]

[0018] where is the area change amount of the e-th lesion area between two image acquisitions, is the total area of the e-th lesion area, γ e is the regional weight of the lesion area, is the image gradient of the e-th region, reflecting the degree of pixel change within the lesion area, N is the total number of lesion areas; the global risk score R global is evaluated through the combination of gene data, immune status, and environmental factors:

[0019]

[0020] where ω1 to ω3 are coefficients in the global risk, β j , λ v and μ l are the weight coefficients of gene data, immune system data, and environmental data respectively, G j is the j-th gene data feature, T v is the v-th immune system status feature, E l is the l-th environmental data feature, ρ is the influence coefficient of local risk on global risk, M, P, and Q respectively represent the number of features of gene, immune system, and environmental data; by integrating local risk and global risk, the risk is classified according to the lesion characteristics and clinical needs of different patients.

[0021] According to one aspect of the present invention, a hysteroscopy image target detection system is provided, including:

[0022] A hysteroscopy image acquisition module, which inserts a hysteroscope into the uterine cavity of a patient to collect high-definition images of the intrauterine tissue in real time, obtaining first image data; and collects environmental data in the uterine cavity through a sensor installed on the hysteroscope, obtaining first environmental data;

[0023] An image recognition and analysis module uses image recognition technology to automatically analyze the first image data, identify different types of diseased tissues, and obtain a first recognition result. The first recognition result is combined with the first environmental data to automatically perform a lesion assessment and obtain first assessment data. The first assessment data includes the infection degree, inflammation degree of the lesion, and the size and distribution of the lesion area.

[0024] A lesion assessment and prediction module, based on the first assessment data, according to the patient's physiological data set, associates and analyzes through a multi-level neural network model to evaluate the future development potential of the lesion, divides the risk level of the lesion area, and generates second assessment data.

[0025] An assessment report generation module automatically generates a risk assessment report according to the second assessment data.

[0026] According to one aspect of the present invention, there is provided a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements the hysteroscopy image target detection method as described in the above embodiments.

[0027] According to one aspect of the present invention, there is provided an electronic device, including: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the hysteroscopy image target detection method as described in the above embodiments.

[0028] According to one aspect of the present invention, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the hysteroscopy image target detection method provided in the above various optional implementation manners.

[0029] In the technical solution of the present invention, by real-time collecting high-definition images and environmental data of the intrauterine tissues, and combining multi-level analysis of image recognition technology, sensor data and neural network models, the present invention can realize a comprehensive assessment, evolution trend analysis and risk level division of the intrauterine lesion area.

[0030] The present invention effectively integrates image data, environmental data and patient physiological data, and significantly improves the accuracy of data processing and the reliability of assessment results through the method of multi-modal data fusion and feature modeling. Compared with the traditional method relying on subjective judgment, the present invention can objectively and automatically output a targeted risk assessment report, providing reliable data support and technical guarantee for the scientific assessment and intelligent management of intrauterine lesions.

[0031] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0033] Figure 1 The flowchart of the hysteroscopy image target detection method in an embodiment of the present invention is schematically shown.

[0034] Figure 2 The flowchart of identifying different types of diseased tissues in an embodiment of the present invention is schematically shown.

[0035] Figure 3 The schematic diagram of the hysteroscopy image target detection system in an embodiment of the present invention is schematically shown.

[0036] Figure 4 The structural schematic diagram of the computer system of the electronic device suitable for implementing the embodiments of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this invention will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0038] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present invention. However, those skilled in the art will realize that the technical solutions of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be used. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present invention.

[0039] The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the drawings are merely illustrative and not necessarily include all the content and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0041] The implementation details of the technical solution of the present invention are elaborated in detail as follows:

[0042] Figure 1 The flowchart of the hysteroscopy image target detection method according to an embodiment of the present invention is shown. Referring to Figure 1 as shown, the hysteroscopy image target detection method at least includes steps S1 to S5, which are introduced in detail as follows:

[0043] S1: Insert the hysteroscope into the patient's uterine cavity, and collect high-definition images of the tissues in the uterine cavity in real time to obtain the first image data; during the image collection process, collect the environmental data in the uterine cavity through a sensor to obtain the first environmental data, including parameters such as temperature, pressure, humidity, and pH value.

[0044] Specifically, when inserting the hysteroscope into the patient's uterine cavity, combined with the real-time image and the sensor feedback, use positioning to accurately guide the hysteroscope to ensure that the hysteroscope is accurately positioned in the target lesion area.

[0045] Among them, the positioning is based on the multi-dimensional sensor information built in the hysteroscope, such as angle, position, and real-time image data, automatically calculates the positional relationship between the hysteroscope and the lesion, optimizes the insertion angle and depth, reduces errors, and avoids damaging the surrounding healthy tissues.

[0046] Furthermore, use the high-definition camera built in the hysteroscope to collect high-definition images of the tissues in the uterine cavity in real time as the first image data, and upload the collected first image data in real time through a high-speed data transmission interface.

[0047] Preferably, preprocess the first image data, including noise removal, contrast enhancement, and image edge sharpening.

[0048] This process initially optimizes the image through a deep learning algorithm, improves the readability of the image, and ensures the accuracy of the subsequent recognition process.

[0049] Synchronously collect the environmental data in the uterine cavity through the sensors integrated on the hysteroscope, including physiological parameters such as temperature, pressure, humidity, and pH value, and record these parameters in real time as the first environmental data.

[0050] Optionally, through the self-calibration function of the sensor, ensure the accuracy of each collected data point, and at the same time perform filtering processing on the first environmental data to reduce external interference and ensure data stability.

[0051] S2: Use image recognition technology to automatically analyze the first image data, identify different types of diseased tissues to obtain the first recognition result, and combine the first environmental data to automatically perform lesion evaluation to obtain the first evaluation data, including the degree of infection, inflammation degree of the lesion, and the size and distribution of the lesion area.

[0052] S2.1: Use a convolutional neural network (CNN) to automatically analyze the lesion area of the preprocessed first image data, and identify and segment the lesion area in the uterine cavity.

[0053] Preferably, the segmentation algorithm accurately distinguishes the lesion area from the normal tissue area through edge detection and region growing strategies, and generates a segmentation result.

[0054] The advantage of CNN in image processing is reflected in its ability to automatically extract the spatial features of the image and identify the lesion area.

[0055] As Figure 2 shown, the specific steps are as follows:

[0056] The input image passes through multiple convolutional layers and pooling layers to extract high-level features, and finally obtains a probability map of the lesion area through a fully connected layer;

[0057] Combine edge detection and region growing strategies to assist in segmenting the boundary and improve the segmentation accuracy.

[0058] Finally, the output segmentation result is a binary image, where the lesion area is marked as 1 and other parts are marked as 0.

[0059] It should be noted that through convolutional operations, CNN can extract the edge information in the image, then reduce the computational complexity through pooling operations, and obtain the final classification result through a fully connected layer. This process enables CNN to gradually improve its accurate segmentation ability for the lesion area through parameter updates.

[0060] S2.2: Based on the image segmentation result, further analyze the identified lesion area, and generate the first recognition result through image recognition algorithms (such as object detection or image classification).

[0061] It should be noted that the first recognition result includes the location and size of each lesion type (such as inflammation, hyperplasia, tumor, etc.) and each lesion area.

[0062] Specifically, using the lesion areas in the segmentation result, perform an object detection task in the image to determine the specific type of the lesion (such as inflammation, hyperplasia, tumor, etc.);

[0063] Use a classification network (such as a deep network based on ResNet or VGG) to classify the type of each lesion area.

[0064] Among them, the input is the segmented sub-image, and the output is the lesion type and the corresponding probability value.

[0065] And by calculating the area and surrounding boundary of the segmentation region, obtain the location and size of each lesion area.

[0066] S2.3: Combine the first recognition result with the first environmental data, apply multi-modal data fusion technology, automatically perform lesion evaluation, and generate the first evaluation data.

[0067] It should be noted that through a weighted algorithm, a comprehensive analysis is performed on the lesion area, its type, and environmental data to calculate the degree of infection of the lesion area.

[0068] Specifically, an increase in local temperature, a change in pH value, and abnormal humidity are usually closely related to the infection process. Based on the changes in these parameters, it is possible to evaluate whether the area is in an infected state and the degree of infection (such as mild, moderate, severe).

[0069] Furthermore, combined with the lesion type in the first recognition result, by analyzing the changes in physiological parameters of the lesion area (such as an increase in temperature, acid-base changes in pH value, etc.), a machine learning algorithm (such as a support vector machine, decision tree, etc.) is used to evaluate the degree of inflammation.

[0070] The evaluation of the degree of inflammation not only considers the lesion area segmented from the image, but also needs to analyze the physiological response state of this area, such as an increase in local temperature, a change in pH value, etc., which can all be used as physiological markers of inflammation.

[0071] The first evaluation data includes the lesion type, degree of infection, degree of inflammation, and the size, distribution, and physiological state of the lesion area (such as an increase in local temperature, a change in pH value, etc.).

[0072] Preferably, multi-modal data fusion can further verify the nature and development state of the lesion area by combining environmental data.

[0073] S3: Based on the first evaluation data, according to the patient physiological data set, through the multi-level neural network model, conduct correlation analysis to evaluate the future development potential of the lesion (such as hyperplasia, diffusion, etc.), classify the risk level of the lesion area, and generate the second evaluation data.

[0074] Among them, the patient physiological data set includes various data such as the patient's genetic information and immune system status. Among them, genetic data can reveal the patient's susceptibility and the genetic trend of the lesion, and the immune system status can reflect the individual's immune response to the lesion, thereby affecting the spread and hyperplasia of the lesion.

[0075] Furthermore, use the multi-modal neural network to integrate different types of data.

[0076] It should be noted that the multi-level neural network model can process image data (high-definition hysteroscopy images) and structured data (genetic information, immune status, etc.).

[0077] Specifically, through multi-level network analysis, not only consider the hysteroscopy image data, but also combine multi-dimensional data such as genetic information and immune system status, design a dynamic evolution prediction model, and achieve accurate prediction of the future development potential of the lesion (such as hyperplasia, diffusion, deterioration, etc.).

[0078] Preferably, by analyzing information such as the patient's gene mutations and genetic susceptibility, evaluate the future development trend of the lesion. Through the deep learning model, combine these genetic information with the image data to infer the possible development path of the patient's lesion.

[0079] The immune system status can affect the spread speed and severity of the lesion. By analyzing the immune data and combining the lesion image data, it is possible to predict whether the lesion is likely to experience rapid hyperplasia or diffusion.

[0080] Preferably, the present invention introduces a graph attention network, which uses the attention mechanism to automatically learn the importance weights between different modal data, thereby enhancing the expression ability of the model.

[0081] At the same time, when fusing the features of structured data such as genetic information and immune status with the image data, introduce a multi-scale feature fusion mechanism to effectively process the association between the spatial structure features of the image data and the structured data. The specific formula is expressed as:

[0082]

[0083] where h i is the feature representation vector of the i-th gene node, a is the attention weight adaptively learned, representing the importance of the relationship between gene information, α i is the attention weight between gene information and immune information, F geneh is the gene information feature processed by the graph attention network. p h g is the feature representation vector of the p-th gene node. g h imm is the feature representation vector of the g-th gene node.

[0084] Similarly, the processing of the immune system state can be carried out in the same way to obtain the immune state feature F processed by the graph attention network. imm .

[0085] Using a multi-modal neural network to comprehensively model the above multi-dimensional data and using a fusion learning model to predict the evolution trend, where the output is the probability of future development potential (such as proliferation, diffusion), etc.

[0086] First of all, the feature extraction of image data should consider not only global features but also local features and multi-scale features.

[0087] Performing multi-scale processing on the image through a Pyramid Convolutional Neural Network (Pyramid CNN) to obtain a multi-level spatial feature representation:

[0088]

[0089] Among them, is the image feature obtained after processing by the pyramid convolution network. is the k-th scale feature of the image data, W k is the k-th layer convolution kernel, and K is the number of scales of the multi-scale convolution operation in the image data; k

[0090] Design a feature fusion layer to jointly model the image data and structured data (genes, immune system), and introduce a coupling loss function to ensure the semantic consistency between different modal data. The fusion process is as follows:

[0091]

[0092] Among them, F fu is the fused feature representation, f i (.) maps the feature of the i-th modality, β i is the importance weight of different modal features, λ is the adjustment coefficient of the coupling loss term, fu i (.) i is the coupling loss function, F gene,imm is the feature of gene information and immune state, and λ1 and λ2 are balance coefficients. gene,imm

[0093] It should be noted that the role of this coupling loss function is to minimize the difference between the gene-immune data features and the image features in the fused space, so that different modal data can jointly represent the relevant features of the lesion and ensure that important information is not lost during multi-modal fusion.

[0094] By further analyzing the features after multimodal fusion, a prediction model of lesion evolution can be established to predict the probability of future development potential (such as hyperplasia, diffusion).

[0095] The evolution trend prediction is as follows:

[0096] Specifically, first, the fused features are processed by a deep neural network (DNN) to output the probability of the future development potential of the lesion:

[0097] y pred = Softmax(W × F fu + b)

[0098] where y pred is the predicted probability of lesion development, W is the weight matrix of the neural network, and b is the bias term.

[0099] Furthermore, the local risk (such as subtle changes in the lesion area) and global risk (such as the possibility of overall lesion diffusion) of the lesion are evaluated separately.

[0100] Among them, the local risk can be obtained through image segmentation and analysis, while the global risk considers the impact of gene and immune data on lesion diffusion.

[0101] Specifically, the local risk mainly refers to the subtle changes in the lesion area, including the expansion of the lesion edge, lesion hyperplasia in the local area, etc. Through image segmentation and regional analysis, the changes within the lesion area can be accurately captured.

[0102] The local risk score R local can be evaluated through the results of image segmentation. Especially based on edge detection and region growing, the local risk is calculated using the change rate of the lesion area:

[0103]

[0104] where is the change in area of the e-th lesion area between two image acquisitions, is the total area of the e-th lesion area, γ e is the regional weight of the lesion area, is the image gradient of the e-th region, reflecting the degree of pixel change within the lesion area, and N is the total number of lesion areas.

[0105] Preferably, the global risk mainly considers the impact of gene information, immune status, and other physiological data on the possibility of lesion diffusion and development.

[0106] The global risk assessment considers the expansion trend of the lesion throughout the uterine cavity, rather than just a certain local area.

[0107] Global risk score R global can be evaluated through the combination of genetic data, immune status, and environmental factors. According to the gene mutation information, immune response, and environmental data together, they jointly affect the global risk:

[0108]

[0109] Among them, ω1 to ω3 are the coefficients in the global risk, β j , λ v and μ l are the weight coefficients of genetic data, immune system data, and environmental data respectively. G j is the jth genetic data feature, T v is the vth immune system state feature, E l is the lth environmental data feature, ρ is the influence coefficient of local risk on global risk, and M, p, and q represent the number of features of genetic, immune system, and environmental data respectively.

[0110] It can be seen that the influence of local risk in global risk reflects the role of local lesions on the overall lesion expansion trend. Especially when the local lesion area rapidly proliferates or changes, it may significantly affect the overall spread of the lesion.

[0111] It should be noted that when the local risk is high, it indicates that there may be a potential for rapid deterioration in the local lesion, which needs to be focused on; the global risk is usually closely related to factors such as the genetic susceptibility and immune response of the lesion. When the global risk is high, it means that the lesion may not only spread rapidly locally but also affect other regions or systems.

[0112] Combining local risk and global risk, classify the risk according to the lesion characteristics and clinical needs of different patients.

[0113] The specific classification can be carried out according to the following criteria:

[0114] Divide the global risk score and the global risk score into 4 score intervals according to 3 thresholds each:

[0115] When the local risk is lower than the third-level threshold and the global risk is lower than the third-level threshold, it is comprehensively evaluated as low risk;

[0116] When the local risk is lower than the third-level threshold and the global risk is higher than the first-level threshold, it is comprehensively evaluated as medium risk;

[0117] When the local risk is between the second-level threshold and the third-level threshold and the global risk is lower than the third-level threshold, it is comprehensively evaluated as high risk;

[0118] When the local risk is higher than the first-level threshold and the global risk is between the second-level threshold and the third-level threshold, it is comprehensively evaluated as a high risk;

[0119] When the local risk is higher than the first-level threshold and the global risk is also higher than the first-level threshold, it is comprehensively evaluated as an extremely high risk.

[0120] According to the above analysis results, the second evaluation data is output, including the prediction of the future development potential of the lesion area and the risk level data.

[0121] Optionally, during the real-time data acquisition process, based on the first recognition result, the acquisition frequency and image acquisition parameters are automatically adjusted. For example, when the lesion is in a high-risk state, the image resolution or acquisition frequency is automatically increased to ensure high-precision tracking of the lesion, and the time window for image acquisition is optimized through an intelligent algorithm to improve the detection efficiency and accuracy.

[0122] S4: Automatically generate a risk assessment report according to the second evaluation data.

[0123] Among them, the report includes the risk level division of the lesion area, the possible future evolution trend, and relevant risk warnings, providing a reference for subsequent clinical decisions.

[0124] The following introduces the device embodiments of the present invention, which can be used to execute the hysteroscopic image target detection method in the above embodiments of the present invention. It can be understood that the device can be a computer program (including program code) running in a computer device. For example, the device is an application software; the device can be used to execute the corresponding steps in the method provided by the embodiments of the present invention. For the details not disclosed in the device embodiments of the present invention, please refer to the embodiments of the above hysteroscopic image target detection method of the present invention.

[0125] Figure 3 The block diagram of a hysteroscopic image target detection system according to an embodiment of the present invention is shown.

[0126] Refer to Figure 3 As shown, a hysteroscopic image target detection system according to an embodiment of the present invention includes:

[0127] A hysteroscopic image acquisition module 310 inserts the hysteroscope into the patient's uterine cavity to collect high-definition images of the intrauterine tissue in real time to obtain the first image data; and collects the environmental data in the uterine cavity through the sensor installed on the hysteroscope to obtain the first environmental data;

[0128] The image recognition and analysis module 320 uses image recognition technology to automatically analyze the first image data, identify different types of diseased tissues, and obtain a first recognition result; the first recognition result is combined with the first environmental data to automatically perform a lesion assessment and obtain first assessment data; the first assessment data includes the infection degree, inflammation degree of the lesion, and the size and distribution of the lesion area;

[0129] The lesion assessment and prediction module 330, based on the first assessment data, according to the patient's physiological data set, analyzes and evaluates the future development potential of the lesion through a multi-level neural network model, divides the risk level of the lesion area, and generates second assessment data;

[0130] The assessment report generation module 340 automatically generates a risk assessment report according to the second assessment data.

[0131] In the present invention, based on the foregoing solution, the identification of different types of diseased tissues includes: using a convolutional neural network to automatically analyze the lesion area of the preprocessed first image data, identifying and segmenting the lesion area in the uterine cavity, and the steps are as follows: the input image passes through multiple convolutional layers and pooling layers to extract high-level features, and finally obtains a probability map of the lesion area through a fully connected layer; combining edge detection and region growing strategies to assist in segmenting the boundary and improving the segmentation accuracy; the output segmentation result is a binary image, where the lesion area is marked as 1 and other parts are marked as 0; using the lesion area in the segmentation result to perform an object detection task in the image to determine the specific type of the lesion; using a classification network to classify the type of each lesion area, and obtaining the position and size of each lesion area by calculating the area and surrounding boundary of the segmented area.

[0132] In the present invention, based on the foregoing solution, the patient's physiological data set includes the patient's genetic information and immune system status data.

[0133] In the present invention, based on the foregoing solution, the multi-level neural network model processes image data and structured data; the structured data includes the genetic information and immune system status data.

[0134] In the present invention, based on the foregoing solution, the construction of the multi-level neural network model includes the following operation steps: through a multi-scale feature fusion mechanism, fusing the structured data with the image data; the feature extraction of the image data should not only consider global features, but also local features and multi-scale features: performing multi-scale processing on the image data through a pyramid convolutional neural network to obtain a multi-level spatial feature representation:

[0135]

[0136] Among them, is the image feature obtained after being processed by the pyramid convolution network, is the k-scale feature of the image data, W k is the k-th layer convolution kernel, and K is the number of scales of the multi-scale convolution operation in the image data; a feature fusion layer is designed to jointly model the image data and the structured data to obtain the fused feature F fu , and a coupling loss function is introduced to ensure the semantic consistency between different modality data.

[0137] In the present invention, based on the foregoing solution, the assessment of the future development potential of the lesion through the correlation analysis of the multi-level neural network model includes: processing the fused feature F fu by a deep neural network to output the probability of the future development potential of the lesion, and the specific calculation formula is:

[0138] y pred = Softmax(W × F fu + b)

[0139] where y pred is the predicted probability of lesion development, W is the weight matrix of the neural network, and b is the bias term.

[0140] In the present invention, based on the foregoing solution, the risk level division of the lesion area includes the following steps: the risk includes the local risk and the global risk of the lesion; the local risk score R local is evaluated through the result of image segmentation. Based on edge detection and region growing, the local risk is calculated using the change rate of the lesion area:

[0141]

[0142] where, is the area change amount of the e-th lesion area between two image acquisitions, is the total area of the e-th lesion area, γ e is the regional weight of the lesion area, is the image gradient of the e-th area, reflecting the degree of pixel change within the lesion area, and N is the total number of lesion areas; the global risk score R global is evaluated through the combination of gene data, immune status, and environmental factors:

[0143]

[0144] where, ω1 to ω3 are the coefficients in the global risk, and β j , λ v and μ l are the weight coefficients of gene data, immune system data, and environmental data respectively, and Gj is the j-th gene data feature, T v is the v-th immune system status feature, E l is the l-th environmental data feature, ρ is the influence coefficient of local risk on global risk, and M, P, and Q respectively represent the number of features of gene, immune system, and environmental data; by integrating local risk and global risk, the risk is classified according to the lesion characteristics and clinical needs of different patients.

[0145] Based on the technical solution of the present invention, by collecting high-definition images and environmental data of intrauterine tissues in real time, and combining multi-level analysis of image recognition technology, sensor data, and neural network models, the present invention can achieve a comprehensive assessment, evolution trend analysis, and risk level classification of intrauterine lesion areas.

[0146] The present invention effectively integrates image data, environmental data, and patient physiological data, and significantly improves the accuracy of data processing and the reliability of evaluation results through multi-modal data fusion and feature modeling. Compared with traditional methods that rely on subjective judgment, the present invention can objectively and automatically output targeted risk assessment reports, providing reliable data support and technical guarantee for the scientific assessment and intelligent management of intrauterine lesions.

[0147] Figure 4 The structure diagram of a computer system of an electronic device suitable for implementing the embodiments of the present invention is shown.

[0148] It should be noted that the computer system of the electronic device in this embodiment is only an example, and should not bring any restrictions to the functions and usage scopes of the embodiments of the present invention.

[0149] The computer system in this embodiment includes a central processing unit 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory 402 or the program loaded from the storage part 408 into the random access memory 403, such as executing the hysteroscopy image target detection method described in the above embodiments. In the random access memory 403, various programs and data required for system operation are also stored. The central processing unit 401, the read-only memory 402, and the random access memory 403 are connected to each other through a bus 404. The input / output interface 405 is also connected to the bus 404.

[0150] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. The drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read from it can be installed into the storage section 408 as needed.

[0151] Specifically, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication section 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the system of the present invention are executed.

[0152] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. Among them, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0154] The units involved in the embodiments of the present invention can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation to the units themselves in some cases.

[0155] According to one aspect of the present invention, there is provided a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above various alternative implementation manners.

[0156] As another aspect, the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the hysteroscopy image target detection method described in the above embodiments.

[0157] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present invention, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0158] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (such as a personal computer, a server, a touch terminal, or a network device, etc.) to execute the methods according to the embodiments of the present invention.

[0159] After considering the specification and practicing the disclosed embodiments herein, those skilled in the art will readily conceive of other embodiments of the present invention. The present invention is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed by the present invention.

[0160] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A method for detecting a target in a hysteroscopic image, characterized in that: include: Inserting a hysteroscope into the patient's uterine cavity, collecting high-definition images of tissues in the uterine cavity in real time, and obtaining first image data; and collecting environmental data in the uterine cavity through a sensor installed in the hysteroscope to obtain first environmental data; The first image data is automatically analyzed using image recognition technology to identify different types of diseased tissues to obtain a first recognition result; the first recognition result is combined with the first environmental data to automatically perform a lesion assessment to obtain first assessment data; The first evaluation data includes the infection degree and inflammation degree of the lesion and the size and distribution of the lesion area; Based on the first assessment data, according to the patient's physiological data set, the future development potential of the lesion is assessed through a multi-level neural network model association analysis, and the risk level of the lesion area is divided, and the second assessment data is generated; A risk assessment report is automatically generated based on the second assessment data.

2. The hysteroscopic image target detection method according to claim 1, characterized in that: The identification of different types of diseased tissues includes: The convolutional neural network is used to automatically analyze the lesion area of ​​the preprocessed first image data to identify and segment the lesion area in the uterine cavity. The steps are as follows: The input image is passed through multiple convolutional layers and pooling layers to extract high-level features, and finally a probability map of the lesion area is obtained through a fully connected layer; Combine edge detection and region growing strategies to assist segmentation boundaries; The output segmentation result is a binary image, in which the lesion area is marked as 1 and the other parts are marked as 0; Using the lesion area in the segmentation result, perform the target detection task in the image to determine the specific type of the lesion; A classification network is used to classify the type of each lesion area, and the location and size of each lesion area are obtained by calculating the area and surrounding boundaries of the segmented region.

3. The hysteroscopic image target detection method according to claim 1, characterized in that: The patient physiological data set includes the patient's genetic information and immune system status data.

4. The method for detecting targets in hysteroscopic images according to claim 3, characterized in that: The multi-level neural network model processes image data and structured data; The structured data includes the gene information and immune system status data.

5. The method for detecting targets in hysteroscopic images according to claim 4, characterized in that: The construction of the multi-level neural network model includes the following steps: Through the multi-scale feature fusion mechanism, the structured data and image data are fused; The feature extraction of the image data should not only consider the global features, but also the local features and multi-scale features: The image data is processed at multiple scales through a pyramid convolutional neural network to obtain a multi-level spatial feature representation: in, is the image feature obtained after processing by the pyramid convolutional network. is the k-th scale feature of the image data, W k is the k-th convolution kernel, K is the number of scales of the multi-scale convolution operation in the image data; Design a feature fusion layer to jointly model image data and structured data to obtain the fused feature F fu , and introduces a coupling loss function to ensure the semantic consistency between different modal data.

6. The method for detecting targets in hysteroscopic images according to claim 1, characterized in that: The evaluation of the future development potential of the lesion by multi-level neural network model association analysis includes: The fused feature F is processed by a deep neural network fu Processing is performed to output the probability of the future development potential of the lesion. The specific calculation formula is: and pred =Softmax(W×F fu +b) Among them, y pred is the predicted probability of lesion development, W is the weight matrix of the neural network, and b is the bias term.

7. The method for detecting targets in hysteroscopic images according to claim 6, characterized in that: The risk level classification of the lesion area comprises the following steps: The risks include local risks and global risks of the lesion; The local risk score R local The evaluation is performed through the results of image segmentation. Based on edge detection and region growing, the change rate of the lesion area is used to calculate the local risk: in, is the area change of the e-th lesion area between two image acquisitions, is the total area of ​​the e-th lesion area, γ e is the regional weight of the lesion area, is the image gradient of the e-th region, reflecting the degree of pixel change in the lesion region, and N is the total number of lesion regions; The global risk score R global Evaluated through a combination of genetic data, immune status, and environmental factors: Among them, ω1~ω3 are the coefficients in the global risk, β j , v and μ l are the weight coefficients of gene data, immune system data and environmental data, G j is the jth gene data feature, T v is the vth immune system status feature, E l is the lth environmental data feature, ρ is the influence coefficient of local risk on global risk, M, P, and Q represent the number of features of gene, immune system, and environmental data, respectively; Based on the comprehensive consideration of local and global risks, the risks of different patients are classified into different levels according to their lesion characteristics and clinical needs.

8. A hysteroscopic image target detection system, characterized in that: include: A hysteroscopic image acquisition module inserts a hysteroscope into the patient's uterine cavity, acquires high-definition images of the tissues in the uterine cavity in real time, and obtains first image data; and collecting environmental data in the uterine cavity through a sensor installed in the hysteroscope to obtain first environmental data; an image recognition and analysis module, which uses image recognition technology to automatically analyze the first image data, identify different types of lesion tissues, and obtain a first recognition result; the first recognition result is combined with the first environmental data to automatically evaluate the lesion and obtain first evaluation data; the first evaluation data includes the infection degree and inflammation degree of the lesion and the size and distribution of the lesion area; A lesion assessment and prediction module, based on the first assessment data and the patient's physiological data set, assesses the future development potential of the lesion through a multi-level neural network model association analysis, divides the lesion area into risk levels, and generates second assessment data; An assessment report generating module automatically generates a risk assessment report based on the second assessment data.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hysteroscopic image target detection method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the hysteroscopic image target detection method as described in any one of claims 1 to 7.

Images