Digestive endoscopy image part recognition and tumor region instance segmentation method and system

Abstract

The present invention belongs to the technical field of medical auxiliary diagnosis, and provides a method for digestive endoscopy image part recognition and tumor region instance segmentation, including: inputting the acquired digestive endoscopy image into a pre-trained part recognition and tumor segmentation model to obtain part recognition results and tumor segmentation results; the part recognition and tumor segmentation model includes: a backbone network for extracting features extracted by the backbone network; a part recognition branch network for processing the features extracted by the backbone network to obtain attention features, and a fully connected unit for processing the attention features to obtain part recognition results; a tumor instance segmentation branch network for fusing the features extracted by the backbone network using a neck module to obtain fusion features, and obtaining tumor segmentation results through a detection head module. The present invention also provides an intelligent auxiliary diagnosis system, a computer program product, and an electronic device. The present invention improves the real-time and accuracy of intelligent auxiliary diagnosis of digestive endoscopy images.

Description

Technical Field

[0001] The present invention relates to the technical field of medical auxiliary diagnosis, and in particular to a method and system for digestive endoscopy image part recognition and tumor region instance segmentation. Background Art

[0002] In a digestive endoscopy system based on artificial intelligence assistance, it is usually necessary to simultaneously detect the part corresponding to the endoscopic image and whether there is a tumor. The existing methods mainly consider designing corresponding deep learning models for the two tasks for detection, but running two deep learning models at the same time requires more computational freedom, which limits the real-time performance of the detection process. In addition, the inventors have statistically analyzed a large amount of case data and found that the probability of lesions occurring in different parts of the digestive tract is not the same, and the statistical distribution of lesions occurring in different parts of the cases is different. The method based on two deep learning models is difficult to learn the correlation between lesions and parts, making it possible to optimize the detection accuracy. Summary of the invention

[0003] The present invention aims to at least solve the technical problems that in the existing artificial intelligence-assisted digestive endoscopy system, two deep learning models are simultaneously run to respectively detect the part corresponding to the endoscopic image and whether a tumor exists, which limits the real-time detection, and the part recognition and tumor lesion area detection are not associated. A method and system for digestive endoscopy image part recognition and tumor area instance segmentation are provided, which integrates the two tasks of part recognition and tumor area detection into a unified framework, realizes part recognition and tumor area detection simultaneously, reduces the resource requirements of the detection system, and improves the detection efficiency and accuracy.

[0004] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a method for digestive endoscopy image part recognition and tumor area instance segmentation, comprising: obtaining a digestive endoscopy image, inputting the digestive endoscopy image into a pre-trained part recognition and tumor segmentation model, and obtaining a part recognition result and a tumor segmentation result; wherein the part recognition and tumor segmentation model comprises: a backbone network, which is used to extract semantic information of the digestive endoscopy image to obtain backbone network extracted features; a part recognition branch network, which uses a classification attention mechanism module to process the backbone network extracted features to obtain attention features, and classifies the attention features through a fully connected unit to obtain a part recognition result; a tumor instance segmentation branch network, which uses a neck module to fuse the backbone network extracted features to obtain fused features, and uses a detection head module to process the fused features to obtain a tumor segmentation result.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides an intelligent auxiliary diagnosis system for implementing the method for digestive endoscopy image part recognition and tumor area instance segmentation described in the first aspect of the present invention, including: an image acquisition module, used to acquire digestive endoscopy images, and input the digestive endoscopy images into the part recognition and tumor segmentation module; the part recognition and tumor segmentation module inputs the digestive endoscopy images into a pre-trained part recognition and tumor segmentation model to obtain part recognition results and tumor segmentation results; wherein the part recognition and tumor segmentation model includes: a backbone network, used to extract semantic information of the digestive endoscopy images to obtain backbone network extracted features; a part recognition branch network, using a classification attention mechanism module to process the backbone network extracted features to obtain attention features, and classifying the attention features through a fully connected unit to obtain a part recognition result; a tumor instance segmentation branch network, using a neck module to fuse the backbone network extracted features to obtain fused features, and processing the fused features through a detection head module to obtain a tumor segmentation result.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for digestive endoscopy image part recognition and tumor area instance segmentation described in the first aspect of the present invention.

[0007] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for digestive endoscopy image part recognition and tumor area instance segmentation as described in the first aspect of the present invention.

[0008] The beneficial technical effect of the present invention is as follows: the input digestive endoscopy image is first subjected to semantic feature extraction through a backbone network, i.e., the backbone network extracted features are obtained, and the backbone network extracted features are then respectively input into a part recognition branch network and a tumor instance segmentation branch network, thereby simultaneously realizing part recognition and tumor lesion area detection. The part recognition and tumor segmentation model integrates the tasks of part recognition and tumor lesion area detection into a unified framework, thereby reducing the resource requirements of the detection system and improving the detection efficiency. The part recognition and tumor segmentation model are trained as a whole, so that the part recognition branch network and the tumor instance segmentation branch network can learn the correlation between tumor lesions and parts, associate tumor lesions with parts, and further improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flow chart of a method for digestive endoscopy image part recognition and tumor region instance segmentation in a preferred embodiment of the present invention;

[0010] Figure 2 It is a schematic diagram of the internal structure of a part recognition and tumor segmentation model in a preferred embodiment of the present invention;

[0011] Figure 3 It is a schematic diagram of the process of obtaining a part recognition and tumor segmentation model in an application scenario of the present invention;

[0012] Figure 4 It is a structural block diagram of an intelligent auxiliary diagnosis system in a preferred embodiment of the present invention;

[0013] Figure 5 It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0014] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0015] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0016] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0017] The present invention provides a method for digestive endoscopy image part recognition and tumor region instance segmentation, and the execution subject of the method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0018] In a preferred embodiment, Figure 1 As shown in FIG. 1 , the digestive endoscopy image part recognition and tumor region instance segmentation method includes:

[0019] Step S1, obtaining a digestive endoscopy image.

[0020] In this embodiment, the digestive endoscopic image may be derived from an image collected by an endoscope system used for the examination of a certain digestive tract organ. The digestive endoscopic image may be read from the memory of the endoscope system, or it may be extracted from a digestive endoscopic image database. The digestive tract organs include the pharynx, esophagus, stomach, small intestine, large intestine, etc., and the corresponding endoscope systems include esophagoscopic system, gastroscopic system, small enteroscope system, colonoscope system, duodenoscope system, etc. There are multiple anatomical parts (referred to as parts) for each organ in the digestive tract. For example, the pharynx is divided into three parts: the nasopharynx, oropharynx, and hypopharynx; the stomach is divided into four parts: the gastric cardia, the gastric fundus, the gastric body, and the pylorus; and the small intestine is divided into three parts: the duodenum, the jejunum, and the ileum. More than one part of the digestive tract organ may be located within the field of view of the endoscope system camera at the same time. Therefore, there may be more than one part that needs to be identified in a digestive endoscopic image.

[0021] Step S2, inputting the digestive endoscopy image into a pre-trained part recognition and tumor segmentation model to obtain a part recognition result and a tumor segmentation result.

[0022] In this embodiment, the part recognition result includes the predicted probability of all parts in a certain digestive tract organ. The predicted probability of a certain part indicates the probability of the part existing in the digestive endoscopic image, and the predicted probability value range can be 0 to 1. The tumor segmentation result includes the tumor type, tumor coordinate position (prediction box) and tumor mask area of ​​the tumor area in the digestive endoscopic image, or NULL (indicating that no tumor is identified in the digestive endoscopic image at this time). Preferably, after obtaining the digestive endoscopic image, a step of preprocessing it is also included, and the preprocessing is preferably but not limited to distortion correction, brightness adjustment, etc.

[0023] like Figure 2 As shown, the site recognition and tumor segmentation model includes:

[0024] (1) Backbone network, used to extract semantic information of digestive endoscopy images to obtain backbone network extraction features. The backbone network is mainly used to extract semantic features of digestive endoscopy images, and is not limited to selecting existing VGGNet, Darknet, Ghostnet, CSPDarkNet53, MobileNetV3 and other backbone feature extraction network structures. The CSPDarkNet53 model is preferably used as the backbone network. The processing process of the backbone network can be expressed as:

[0025] x=f(I)

[0026] Among them, x represents the features extracted by the backbone network, f(·) represents the processing process of the backbone network, and I represents the digestive endoscopy image input into the backbone network.

[0027] (2) The part recognition branch network uses the classification attention mechanism module to process the features extracted from the backbone network to obtain attention features, and classifies the attention features through the fully connected unit to obtain the part recognition result. The classification attention mechanism module improves the features extracted from the backbone network. The classification attention mechanism module is not limited to the existing SENet, CBAM, CSAM, TripletAttention, CoordAttention, ECA, EMAtattention, MHSA, ShuffleAttention, SiAM and other attention mechanism models. The fully connected unit may include more than one cascaded fully connected layer, and part recognition is realized through the fully connected unit, and the predicted probability of each part of the corresponding digestive tract organ is output.

[0028] Attention feature x Ca Expressed as: x Ca =Att(x), Att(·) represents the classification attention mechanism module processing process. Part recognition result c x Expressed as: c x =R(f c (Wx Ca+b)), W, b represent the weight matrix and bias parameter of the fully connected unit respectively, f c (·) represents a fully connected unit, and R(·) represents an activation function.

[0029] (3) Tumor instance segmentation branch network, using the neck module to fuse the features extracted by the backbone network to obtain fused features, and the fused features are processed by the detection head module to obtain the tumor segmentation result. The neck module is not limited to selecting existing models such as FPN, Gather-and-Distribute, Slim-neck and attention mechanism. The detection head module mainly realizes the detection, positioning and segmentation of the lesion area (tumor area) through decoupling based on the fused features output by the neck module. The detection head module is not limited to the existing FPN, ASFF, RT-DETR and other series of detection heads.

[0030] Fusion feature x Df Expressed as: x Df =Fu(x), Fu(·) represents the fusion operation of the neck module. The tumor segmentation result is: [Cls, Bbox, Mask] = D h (x Df ), Cls represents the tumor category, Bbox represents the tumor coordinate position, Mask represents the tumor mask area, D h (·) represents a detection head module or a decoupling module in the detection head module.

[0031] In this embodiment, the backbone network, classification attention mechanism module, neck module and detection head module all adopt the existing model structure, which will not be described in detail here. The part recognition branch network and the tumor instance segmentation branch network share the backbone network extraction features output by the backbone network as input features. On the one hand, the detection efficiency is improved, and on the other hand, it is convenient to train the two branch networks at the same time, so that the two branch networks can learn the correlation between the part and the tumor area during training, realize the unification of the two detection tasks, and improve the detection accuracy.

[0032] In a preferred embodiment, in order to enhance the semantic information of the digestive endoscopic image and further improve the accuracy of site recognition and tumor region segmentation, after obtaining the digestive endoscopic image in step S1, the digestive endoscopic image is preprocessed, and the preprocessed endoscopic image is input into a pre-trained site recognition and tumor segmentation model, wherein the preprocessing process includes:

[0033] Step S11, Fourier transform is performed on the pixels of the digestive endoscopy image to obtain a transformation result, and the phase component P is extracted from the transformation result. 0 .

[0034] Step S12: the phase component P 0 Perform inverse Fourier transform to obtain semantic image Ise :

[0035] I se =FFT -1 (P 0 )

[0036] FFT -1 represents the inverse Fourier transform.

[0037] Step S13, fusing the digestive endoscopic image and the semantic image to obtain a preprocessed digestive endoscopic image. Specifically, the fusion method may be to add pixel values ​​of corresponding pixel points in the digestive endoscopic image and the semantic image or to calculate the average value of the pixel values.

[0038] Further preferably, in order to highlight the semantic information and not lose the details of the original digestive endoscopic image, the fusion strength of the two images can be adjusted, and the digestive endoscopic image I is fused according to the following formula: 0 and semantic image I se , obtain the preprocessed digestive endoscopy image I:

[0039] I=I 0 +α se I se

[0040] Among them, α se It represents the semantic weighting coefficient, and its value range can be 0.1-0.5. The “+” in the above formula represents I 0 and I se The pixel values ​​of corresponding pixels in between are added.

[0041] In a preferred embodiment, the training process of the site recognition and tumor segmentation model includes:

[0042] Step A: Obtain a sample image set and set a part category label and tumor label box for each sample image.

[0043] In actual application, each digestive tract organ can correspond to a part recognition and tumor segmentation model. During training, multiple digestive endoscopic images taken by the endoscope system corresponding to the digestive tract organ are obtained as a sample image set. The part category label can be manually set for each sample image. Specifically, the part category label of the part that actually exists in the sample image is set to 1, and the part category label of the part that does not exist is set to 0; for digestive endoscopic images with tumors, the tumor area is manually and accurately selected, and the set bounding box is used as the tumor label box, and the position information of the tumor label box is retained. For digestive endoscopic images without tumors, the tumor label box is not marked.

[0044] Step B: construct the network structure of the site recognition and tumor segmentation model and initialize the network parameters.

[0045] Step C: Use the sample image set to train the part recognition and tumor segmentation model. During the training process, the loss function is calculated and the network parameters of the part recognition and tumor segmentation model are updated through the back propagation algorithm until the training stop condition is reached.

[0046] In this embodiment, the training stop condition is preferably but not limited to the number of training times reaching the preset maximum number of training times, or the loss function value is less than or equal to the preset loss threshold. Step C also includes dividing the sample image set into a training set, a test set and a validation set, using the training set to perform multiple batches of training on the network of the site recognition and tumor segmentation model, calculating the loss function after each batch of training, calculating the gradient of the loss function and updating the network parameters of the model using the back propagation algorithm. The trained model is tested and verified using the test set and the validation set. When the test and verification pass, the network parameters of the model are saved to obtain a trained site recognition and tumor segmentation model. The loss function can be a weighted sum of the site recognition loss and the tumor segmentation loss.

[0047] In a preferred embodiment, in order to enable the part recognition and tumor segmentation model to effectively and quickly learn the correlation between the tumor area (lesion area) and the part during training, the part recognition and tumor area segmentation detection are associated to further improve the accuracy of model detection. The loss function is Loss:

[0048] Loss = γ 1 L p +γ 2 L m +γ 3 L pm

[0049] Among them, γ 1 , γ 2 , γ 3 Respectively represent the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient, which can be preset as fixed values ​​or dynamically updated during the training process; L p represents the part recognition loss; L m represents the tumor segmentation loss; L pm represents the mixing loss, Among them, C represents the number of parts, that is, the number of part categories, N represents the number of sample images participating in the training, i represents the sample image index participating in the training, c represents the part index, g c It represents the prior probability of a tumor in site c. α represents the penalty coefficient, and the default value is 0.25. i,c g represents the intersection-over-union ratio of the predicted box of part c in the i-th sample image and the tumor label box. c It can be calculated according to the statistical results of the sample image set using the following formula:

[0050]

[0051] In this embodiment, N1 represents the number of sample images with tumors in the sample image set, n c Indicates the number of sample images where the tumor is located at position c. c The value can also be adjusted based on the above calculation formula and combined with the clinical experience of multiple experts to accurately characterize the relationship between the location and the tumor.

[0052] In this embodiment, preferably, Among them, g c ' indicates that according to all tumor segmentation results corresponding to the N sample images participating in the training output by the part recognition and tumor segmentation model during the training process, the ratio of the number of samples in which the part recognition and tumor segmentation model detects the presence of tumors in part c to the number of samples in which the part recognition and tumor segmentation model detects the presence of tumors (i.e., the actual training probability) is calculated. This is so that the part recognition and tumor segmentation model can effectively and quickly learn the correlation between the tumor area (lesion area) and the part during training, so as to more closely associate part recognition with tumor area segmentation detection.

[0053] In this embodiment, y i,c represents the part category label of the i-th sample image part c, which is 0 or 1; p i,c Represents the predicted probability of the u-th sample image part c.

[0054] In this embodiment, When the tumor label box of the i-th sample image at part c is NULL, that is, there is no tumor, and the prediction box of part c in the tumor segmentation result corresponding to the i-th sample image output by the part recognition and tumor segmentation model in training is NULL, that is, there is no tumor, let IOU i,c = 1. When the tumor label box of the i-th sample image at part c is not NULL, and / or the predicted box of part c in the tumor segmentation result corresponding to the i-th sample image output by the part recognition and tumor segmentation model during training is not NULL, IOU i,c The calculation formula is as follows:

[0055]

[0056] B i,c represents the prediction box of part c in the i-th sample image, Represents the tumor label box of part c in the i-th sample image.

[0057] In this embodiment, a mixed loss is introduced into the loss function, which combines the prior probability g of the tumor occurring at the site c. c , training actual probability g c ', the intersection of the predicted box of part c in the sample image and the tumor label box, so that the model is trained with g c 'Close to g c The maximum intersection-over-union ratio is set as the goal, so that the model can fully learn the correlation between the location and the tumor, and improve the detection accuracy of the model.

[0058] In a preferred embodiment, in order to accelerate model convergence and improve training efficiency, the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient are dynamically updated according to the training results. Specifically, in each training, the first weighting coefficient is updated according to the ratio coefficient of the site recognition loss calculated in the last training, the second weighting coefficient is updated according to the ratio coefficient of the tumor segmentation loss calculated in the last training, and the third weighting coefficient is updated according to the ratio coefficient of the mixed loss calculated in the last training.

[0059] In this implementation, during the t-th training, the updating process of the first weighting coefficient, the second weighting coefficient and the third weighting coefficient includes:

[0060] Calculate the ratio of the part recognition loss of the t-1th training and the t-2th training to obtain the ratio coefficient w of the part recognition loss of the t-1th training 1 (t-1), the ratio coefficient of the part recognition loss of the t-1th training is input into the exponential function to obtain the intermediate value of part recognition T is a constant, representing the smoothing parameter of weight update; e is a natural constant.

[0061]

[0062] Calculate the ratio of the tumor segmentation loss of the t-1th training and the t-2th training to obtain the ratio coefficient w of the tumor segmentation loss of the t-1th training 2 (t-1), the ratio coefficient of the tumor segmentation loss of the t-1th training is input into the exponential function to obtain the intermediate value of tumor segmentation

[0063]

[0064] Calculate the ratio of the mixed loss of the t-1th training and the t-2th training to obtain the ratio coefficient w of the mixed loss of the t-1th training 3 (t-1), input the ratio coefficient of the mixed loss of the t-1th training into the exponential function to obtain the mixed intermediate value

[0065]

[0066] The intermediate values ​​of site identification, tumor segmentation and mixed intermediate values ​​are normalized respectively to obtain the updated value of the first weighting coefficient, the updated value of the second weighting coefficient and the updated value of the third weighting coefficient in the t-th training; wherein t is a positive integer greater than 2.

[0067] In this embodiment, the normalization process obtains the updated value γ of the first weighting coefficient of the t-th training. 1 ', the updated value of the second weighting coefficient γ 2 ', the updated value of the third weighting coefficient γ 3 'Respectively:

[0068]

[0069] In this embodiment, in each iterative training, the updated value γ of the first weighting coefficient can be calculated according to the above steps. 1 ', the updated value of the second weighting coefficient γ 2 ', the updated value of the third weighting coefficient γ 3 ', and then use these updated values ​​to calculate the loss function.

[0070] In an application scenario of the method for digestive endoscopy image part recognition and tumor region instance segmentation provided by the present invention, the process of constructing the part recognition and tumor segmentation model is as follows: Figure 3 shown.

[0071] The present invention also discloses an intelligent auxiliary diagnosis system for realizing the above-mentioned digestive endoscopy image part recognition and tumor region instance segmentation method. In a preferred embodiment, Figure 4 As shown, including:

[0072] An image acquisition module is used to acquire digestive endoscopic images and input the digestive endoscopic images into a site recognition and tumor segmentation module;

[0073] The part recognition and tumor segmentation module inputs the digestive endoscopy image into the pre-trained part recognition and tumor segmentation model to obtain the part recognition result and the tumor segmentation result; wherein, the part recognition and tumor segmentation model includes:

[0074] The backbone network is used to extract the semantic information of digestive endoscopy images to obtain backbone network extraction features;

[0075] The part recognition branch network uses the classification attention mechanism module to process the features extracted by the backbone network to obtain the attention features, and classifies the attention features through the fully connected unit to obtain the part recognition results;

[0076] The tumor instance segmentation branch network uses the neck module to fuse the features extracted by the backbone network to obtain fused features, and then processes the fused features through the detection head module to obtain the tumor segmentation result.

[0077] In this embodiment, the image acquisition module and the part recognition and tumor segmentation module correspond one-to-one to steps S1 and S2 of the above-mentioned digestive endoscopy image part recognition and tumor region instance segmentation method, and will not be repeated here.

[0078] The present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned digestive endoscopy image part recognition and tumor area instance segmentation method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0079] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0080] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for digestive endoscopy image part recognition and tumor region instance segmentation provided by the present invention.

[0081] like Figure 5 , which is a schematic diagram of the structure of an electronic device for a method for digestive endoscopy image part recognition and tumor region instance segmentation provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a method for digestive endoscopy image part recognition and tumor region instance segmentation program.

[0082] Among them, the processor 10 can be composed of an integrated circuit in some embodiments, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing digestive endoscopy image part recognition and tumor area instance segmentation methods, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0083] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the digestive endoscopy image part recognition and tumor area instance segmentation method program, but also can be used to temporarily store data that has been output or is to be output.

[0084] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0085] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0086] Figure 5 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0087] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0088] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0089] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0090] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0091] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for digestive endoscopy image part recognition and tumor region instance segmentation, characterized in that: include: Obtaining a digestive endoscopic image, inputting the digestive endoscopic image into a pre-trained part recognition and tumor segmentation model, and obtaining a part recognition result and a tumor segmentation result; Among them, the site recognition and tumor segmentation models include: The backbone network is used to extract the semantic information of digestive endoscopy images to obtain backbone network extraction features; The part recognition branch network uses the classification attention mechanism module to process the features extracted by the backbone network to obtain the attention features, and classifies the attention features through the fully connected unit to obtain the part recognition results; The tumor instance segmentation branch network uses the neck module to fuse the features extracted by the backbone network to obtain fused features, and then processes the fused features through the detection head module to obtain the tumor segmentation result.

2. The method for digestive endoscopy image part recognition and tumor region instance segmentation according to claim 1, characterized in that: After obtaining the digestive endoscopic image, the digestive endoscopic image is preprocessed, and the preprocessed endoscopic image is input into a pre-trained part recognition and tumor segmentation model, wherein the preprocessing process includes: Performing Fourier transformation on the pixels of the digestive endoscopy image to obtain a transformation result, and extracting a phase component from the transformation result; Perform inverse Fourier transform on the phase component to obtain a semantic image; The digestive endoscopic image and the semantic image are fused to obtain the preprocessed digestive endoscopic image.

3. The method for digestive endoscopy image part recognition and tumor region instance segmentation according to claim 2, characterized in that: The digestive endoscopy image I0 and the semantic image I are fused according to the following formula se Obtain the preprocessed digestive endoscopy image I: I=I0+α se I se Among them, α se Represents the semantic weighting coefficient.

4. The method for digestive endoscopy image part recognition and tumor region instance segmentation according to any one of claims 1 to 3, characterized in that: The training process of the site recognition and tumor segmentation model includes: Obtain a sample image set, and set a part category label and a tumor label box for each sample image; Construct the network structure of the site recognition and tumor segmentation model; The sample image set is used to train the part recognition and tumor segmentation model. During the training process, the loss function is calculated and the network parameters of the part recognition and tumor segmentation model are updated through the back propagation algorithm until the training stop condition is reached.

5. The method for digestive endoscopy image part recognition and tumor region instance segmentation according to claim 4, characterized in that: The loss function is Loss: Loss=γ1L p +γ2L m +γ3L pm Wherein, γ1, γ2, and γ3 represent the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient respectively; L p represents the part recognition loss; L m represents the tumor segmentation loss; L pm represents the mixing loss, N represents the number of sample images involved in training, i represents the index of sample images involved in training, C represents the number of parts, c represents the part index, g represents the c represents the prior probability of a tumor in site c, α represents the penalty coefficient, and IOU i,c Represents the intersection-over-union ratio of the predicted box of part c in the i-th sample image and the tumor label box.

6. The method for digestive endoscopy image part recognition and tumor region instance segmentation according to claim 5, characterized in that: In each training, the first weighting coefficient is updated according to the ratio coefficient of the site recognition loss calculated in the previous training, the second weighting coefficient is updated according to the ratio coefficient of the tumor segmentation loss calculated in the previous training, and the third weighting coefficient is updated according to the ratio coefficient of the mixed loss calculated in the previous training.

7. The method for digestive endoscopy image part recognition and tumor region instance segmentation according to claim 6, characterized in that: During the t-th training, the updating process of the first weighting coefficient, the second weighting coefficient and the third weighting coefficient includes: Calculate the ratio of the part recognition loss of the t-1th training and the t-2th training to obtain the ratio coefficient of the part recognition loss of the t-1th training, and input the ratio coefficient of the part recognition loss of the t-1th training into the exponential function to obtain the intermediate value of the part recognition; Calculate the ratio of the tumor segmentation loss of the t-1th training and the t-2th training to obtain the ratio coefficient of the tumor segmentation loss of the t-1th training, and input the ratio coefficient of the tumor segmentation loss of the t-1th training into the exponential function to obtain the intermediate value of the tumor segmentation; Calculate the ratio of the mixed loss of the t-1th training and the t-2th training to obtain the ratio coefficient of the mixed loss of the t-1th training, and input the ratio coefficient of the mixed loss of the t-1th training into the exponential function to obtain the mixed intermediate value; Normalizing the intermediate values ​​of site identification, tumor segmentation, and mixed intermediate values ​​respectively to obtain updated values ​​of the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient in the t-th training; Wherein, t is a positive integer greater than 2.

8. An intelligent auxiliary diagnosis system, used to implement the method for digestive endoscopy image part recognition and tumor region instance segmentation according to any one of claims 1 to 7, characterized in that: include: An image acquisition module is used to acquire digestive endoscopic images and input the digestive endoscopic images into a site recognition and tumor segmentation module; The part recognition and tumor segmentation module inputs the digestive endoscopy image into the pre-trained part recognition and tumor segmentation model to obtain the part recognition result and the tumor segmentation result; wherein, the part recognition and tumor segmentation model includes: The backbone network is used to extract the semantic information of digestive endoscopy images to obtain backbone network extraction features; The part recognition branch network uses the classification attention mechanism module to process the features extracted by the backbone network to obtain the attention features, and classifies the attention features through the fully connected unit to obtain the part recognition results; The tumor instance segmentation branch network uses the neck module to fuse the features extracted by the backbone network to obtain fused features, and then processes the fused features through the detection head module to obtain the tumor segmentation result.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for digestive endoscopy image part recognition and tumor region instance segmentation as described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for digestive endoscopy image site recognition and tumor region instance segmentation as described in any one of claims 1 to 7.

Images