A method, apparatus, and device for recognizing colposcopy image regions based on a two-stage deep learning network.

Abstract

This invention discloses a method for recognizing colposcopy image regions based on a two-stage deep learning network. The method includes: acquiring different image data using a colposcope; constructing a U-shaped transform convolutional neural network (HCNN) model based on the image data; training the constructed HCNN model using the HCNN; outputting the training results of predicting atypical small regions in the image data based on the trained HCNN model; constructing a YOLO neural network model based on the HCNN training results; training a YOLO neural network model using the YOLO neural network; and using the two-stage deep learning model (HCNN-YOLO neural network model) to predict atypical small regions and monitoring points corresponding to the colposcopy image data. This invention enables the identification of the location of atypical small regions and monitoring points using colposcopy images.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, and device for recognizing colposcopy image regions based on a two-stage deep learning network. Background Technology

[0002] A colposcope is an instrument used clinically to examine the cervix, vagina, and vulva. It mainly consists of a microscope, a cannula, a microscope stage, and a shelf. It typically comprises an observation system and an illumination system. The observation system is a stereoscopic optical microscopy system with eyepieces and objectives, operating at a short working distance, and can be connected to an external image acquisition and display system. It utilizes the principle of microscopic magnification to observe the details of objects. Current technology combining colposcopy with image analysis algorithms plays an important role in the clinical examination of the cervix, vagina, and vulva, but it has two main problems:

[0003] (1) Colposcopy images contain many complex elements. Some areas in the images are not typical and are small. Existing technical means are not sensitive enough to the details of atypical small areas and the selection of features is highly uninterpretable. As a result, the current technical solutions can only output the approximate area and cannot obtain the monitoring point.

[0004] (2) The colposcopy images are limited by the light, and the images are often greatly affected by the ambient light. In addition, the colposcopy lens is constantly moving during the colposcopy examination, which can easily cause the loss of detailed information in atypical small areas, making them difficult to identify. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a colposcopy image region recognition method, device, and equipment based on a two-stage deep learning network, which can identify the location and monitoring points of atypical small regions through colposcopy images.

[0006] According to one aspect of the present invention, a method for recognizing colposcopy image regions based on a two-stage deep learning network is provided, comprising: acquiring different image data using a colposcope; constructing a U-shaped transform convolutional neural network model based on the image data; training the constructed U-shaped transform convolutional neural network model using a U-shaped transform convolutional neural network; outputting training results for predicting atypical small regions in the image data based on the trained U-shaped transform convolutional neural network model; constructing a YOLO neural network model based on the training results of the U-shaped transform convolutional neural network; training the YOLO neural network model using a YOLO neural network; and predicting atypical small regions and monitoring points corresponding to the colposcopy image data based on the two-stage deep learning model of the trained U-shaped transform convolutional neural network model and YOLO neural network model.

[0007] According to another aspect of the present invention, a colposcopy image region recognition device based on a two-stage deep learning network is provided, comprising: an acquisition module, a construction module, a training module, and a prediction module; the acquisition module is used to acquire different image data using a colposcope; the construction module is used to construct a two-stage deep learning model of a U-transform convolutional neural network model-YOLO neural network model; the training module is used to train the two-stage deep learning model of the U-transform convolutional neural network model-YOLO neural network model; and the prediction module is used to predict atypical small regions and monitoring points corresponding to the colposcopy image data based on the trained two-stage deep learning model of the U-transform convolutional neural network model-YOLO neural network model.

[0008] According to another aspect of the present invention, a colposcopy image region recognition device based on a two-stage deep learning network is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the colposcopy image region recognition method based on a two-stage deep learning network as described in any of the preceding claims.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the colposcopy image region recognition method based on a two-stage deep learning network as described in any of the preceding claims.

[0010] It can be observed that the above scheme, employing a two-stage deep learning model, makes atypical small regions highly interpretable and sufficiently sensitive to their detailed information, enabling the extraction of atypical small regions and monitoring points. The two components of the two-stage deep learning algorithm (atypical small region segmentation in colposcopy images and biopsy location guidance) are trained sequentially. The first network is trained and used consistently for the training of the second network. Atypical small regions in colposcopy images are often atypical and small, their locations are difficult to detect, exhibiting high concealment. Furthermore, different types of atypical small regions often have different levels of attention. Therefore, directly using existing techniques for training would result in uninterpretable output results for atypical small regions. This invention adopts a two-stage approach: in the first stage, biopsy locations are manually delineated to ensure high interpretability; a U-shaped transform convolutional neural network is used to learn its features and output atypical small regions; and in the second stage, YOLO is used to select and output highly suspicious biopsy regions, thereby generating a model that is both high-performance and highly interpretable. To address the resolution loss in atypical small regions, a U-shaped transform convolutional neural network is proposed in the first stage of the two-stage deep learning model. This network compensates for the resolution loss in atypical small regions caused by the transform neural network, achieving accurate localization. To address the issue of lost detail information in atypical small regions due to the continuous movement of the colposcope during colposcopy, a target detection algorithm is used in the second stage of the two-stage deep learning model to locate monitoring points within these regions. This embodiment addresses three technical problems of existing colposcopy imaging algorithms: lack of interpretability, insufficient sensitivity to detail information in atypical small regions, and loss of detail information in atypical small regions. It enables highly accurate predictions for both atypical small regions and monitoring points. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating an embodiment of the colposcopy image region recognition method based on a two-stage deep learning network according to the present invention.

[0013] Figure 2 This is a schematic diagram of the U-shaped transform convolutional neural network framework structure of an embodiment of the colposcopy image region recognition method based on a two-stage deep learning network according to the present invention.

[0014] Figure 3This is a prediction result diagram output by the two-stage deep learning model of the colposcopy image region recognition method based on a two-stage deep learning network according to an embodiment of the present invention. The model is a U-shaped transform convolutional neural network model-YOLO neural network model. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] This invention provides a colposcopy image region recognition method based on a two-stage deep learning network, which can identify the location and monitoring points of atypical small regions through colposcopy images.

[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the colposcopy image region recognition method based on a two-stage deep learning network according to the present invention. It should be noted that if substantially the same result is obtained, the method of the present invention is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the method includes the following steps:

[0018] S101: Acquiring different image data using a colposcope;

[0019] In this embodiment, the image data can be acquired under conditions of acetic acid hot compress or iodine solution hot compress; this embodiment is not limited to either.

[0020] S102: Construct a U-shaped transform convolutional neural network model based on image data;

[0021] In this embodiment, the constructed U-shaped transform convolutional neural network model can predict atypical small regions in the image data based on the differences in image features between different colposcopy image data; at the same time, it makes up for the loss of feature resolution of atypical small regions caused by the transform neural network, and achieves accurate localization.

[0022] S103: The constructed U-shaped transform convolutional neural network model is trained using a U-shaped transform convolutional neural network.

[0023] In this embodiment, a U-shaped transform convolutional neural network is constructed, and the edge of each atypical small region in the image data is marked by the U-shaped transform convolutional neural network. The image features of each atypical small region are extracted from the image data after the edge of the atypical small region is marked by the U-shaped transform convolutional neural network, and color features, texture features, shape features and spatial relationship features are extracted from the image features as training inputs of the U-shaped transform convolutional neural network. The constructed U-shaped transform convolutional neural network model is trained by the U-shaped transform convolutional neural network with color features, texture features, shape features and spatial relationship features as training inputs.

[0024] Specifically, to address the detection of atypical small regions, a model more sensitive to detailed information is needed to extract feature information from colposcopy images, and transform neural networks excel in extracting detailed features. However, directly using transform neural network segmentation models for atypical small region detection easily leads to loss of feature resolution in atypical small regions, resulting in insufficient accuracy. Therefore, this invention proposes a U-shaped transform convolutional neural network, namely the medical image segmentation framework—the transform neural network segmentation network. Please refer to [link to relevant documentation]. Figure 2 Transform neural networks (CNNs) can serve as encoders for medical image segmentation tasks, enhancing finer details by recovering local spatial information. They establish a self-attention mechanism from a sequence-to-sequence prediction perspective. To compensate for the feature resolution loss introduced by CNNs, the CNN segmentation model employs a hybrid convolutional neural network (CNN)-CNN architecture to utilize detailed high-resolution spatial information from CNN features and the global context encoded by the CNN. Inspired by the U-Net architecture, the features encoded by the CNN's self-attention mechanism are subsequently upsampled and combined with different high-resolution CNN features skipped from the encoding path to overcome the feature resolution loss and improve accuracy. On one hand, the CNN encodes tokenized image patches from CNN feature maps into input sequences for extracting global context. On the other hand, the decoder upsamples the encoded features and then combines them with high-resolution CNN feature maps for precise localization. The trained CNN segmentation model is then manually labeled with bounding boxes to indicate the locations of atypical small regions in each image.

[0025] To train and validate the atypical small region detection network, the locations of atypical small regions in each image were manually labeled by drawing bounding boxes around them. The transform neural network segmentation model has an encoder-decoder architecture. The encoder primarily extracts visual feature representations from the colposcopy images, while the decoder is responsible for decomposing the encoded features and generating pixel-by-pixel segmentation results. The transform neural network segmentation model takes each frame of still colposcopy image data as input and outputs a semantic segmentation image of the atypical small region based on the segmentation result of each input colposcopy image data.

[0026] S104: Based on the trained U-shaped transform convolutional neural network model, output the training results of atypical small regions in the predicted image data.

[0027] In this embodiment, the training results are as follows: Figure 3 The darker-colored irregular area in the center of the image represents atypical small areas predicted by the model.

[0028] S105: Construct a YOLO neural network model based on the training results of the U-shaped transform convolutional neural network.

[0029] In this embodiment, the constructed YOLO neural network model can predict the monitoring points in atypical small regions based on the differences in image features between atypical small regions in different colposcopy image data; it solves the technical problem that the colposcopy lens is constantly moving during the colposcopy examination, which easily leads to the loss of detailed information in atypical small regions.

[0030] S106: The YOLO neural network model is trained using the YOLO neural network.

[0031] In this embodiment, a YOLO neural network is constructed using the training results of a U-shaped transform convolutional neural network. The YOLO neural network is then used to mark the edges of corresponding monitoring points in each image from the U-shaped transform convolutional neural network training results. The YOLO neural network extracts image features of each monitoring point from the image data after the edges of the monitoring points are marked. Color features, texture features, shape features, and spatial relationship features are extracted from the image features and used as training inputs for the YOLO neural network. The constructed YOLO neural network model is then trained using the YOLO neural network with color features, texture features, shape features, and spatial relationship features as training inputs.

[0032] Specifically, after the transformation neural network segmentation model was trained, the monitoring points in its output atypical small regions were manually annotated. The labelImg annotation tool was used, and the annotated data was in the YOLO format for a single-stage object detection algorithm. Because the colposcope lens is constantly moving during a colposcopy, detailed information in atypical small regions can be easily lost, making them difficult to identify. Therefore, a detection method is needed that can dynamically move with the target region in dynamic video, and object detection algorithms perform well in this regard. Within the atypical small regions, object detection algorithms are used to locate the monitoring points. Please refer to [link to relevant documentation]. Figure 3 Since the object detection algorithm generates bounding boxes for object detection, this embodiment uses a circular bounding box centered on the monitoring point as the monitoring signal. Predicted atypical small regions are used to limit the range of monitoring point localization. The backbone network of the object detection algorithm model is VGG16, and its architecture includes convolutional, pooling, fully connected, and output layers. The features obtained by VGG16 are enhanced using a compression activation module and a convolutional attention mechanism module to obtain feature layers containing more effective information. The object detection algorithm model takes the semantic segmentation results of the transformed neural network segmentation model as input and is trained based on the monitoring points annotated by the doctor, ultimately locating the monitoring points in atypical small regions, such as... Figure 3 As shown.

[0033] S107: Based on the two-stage deep learning model of the trained U-shaped transform convolutional neural network model-YOLO neural network model, predict the atypical small regions and monitoring points corresponding to the colposcopy image data.

[0034] In this embodiment, the perfect world reflection algorithm is used to preprocess the image data acquired by colposcopy. Based on the trained U-shaped transform convolutional neural network model, color features, texture features, shape features, and spatial relationship features associated with the atypical small regions are extracted from the preprocessed image data. These extracted features are then input into the trained U-shaped transform convolutional neural network model to identify the atypical small regions in the image. The trained U-shaped transform convolutional neural network model yields the predicted bounding boxes of the atypical small regions. Based on the output atypical small region results, color features, texture features, shape features, and spatial relationship features associated with the monitoring points are extracted. These features are then input into the trained YOLO neural network model to identify the atypical small regions in the image. The trained YOLO neural network model yields the predicted monitoring point locations.

[0035] Specifically, due to the limited lighting conditions in colposcopy images, and the significant impact of ambient light on different images due to variations in colposcopy instruments, detailed information in atypical small areas is easily lost. Therefore, this embodiment performs a uniform size transformation on the cervical images collected by colposcopy. To ensure successful feature extraction in the deep neural network model, the input images need to be uniformly resized. To ensure that the resized images do not destroy the original data features, this embodiment uses a distortion-free approach: scaling the original image proportionally and then adding grayscale bars to areas that do not meet the size requirements. This approach ensures that the original data is not distorted, thus avoiding damage to important data features. Because of the limited lighting conditions in colposcopy images, and the significant impact of ambient light on different images due to variations in colposcopy instruments, detailed information in atypical small areas is easily lost. Before inputting the image data into the model, white balance processing is performed to eliminate color difference artifacts. This embodiment uses a perfect world reflection algorithm for this process. As mentioned above, due to the limited lighting conditions in colposcopy images, and the significant impact of ambient light on different images due to variations in colposcopy instruments, detailed information in atypical small areas is easily lost. This embodiment tested the mean white balance method, perfect reflection, gray-world hypothesis, image analysis-based color cast detection and color correction method, and dynamic thresholding algorithm from three perspectives: processing speed, processing effect, and data requirements for the aforementioned two-stage deep learning model. Ultimately, it was found that the perfect world reflection algorithm performed better in the narrow, moist, and highly variable light environment of the vagina, and in meeting the data requirements of the aforementioned two-stage deep learning model.

[0036] Secondly, a U-shaped transform convolutional neural network is used to segment atypical small regions;

[0037] The specific steps are as follows:

[0038] (1) Image serialization. Given an image a∈R C×K×T The spatial resolution is C×K, and the number of channels is T. Tokenization is performed by reshaping the input image into a series of flattened two-dimensional patches. Each image is decomposed into patches with a spatial resolution of X×X, and the number of patches per image is T.

[0039] (2) Patch embedding, which embeds the vectorized patch a x Mapping to the latent W dimension uses a trainable linear projection embedding space. To encode the patch space information, specific positional embeddings are used, which are added to the patch embeddings to preserve the positional information y0, as shown in the following equation.

[0040]

[0041] Where Q∈R (X×X·T) It is a patch embedded projection, Qloc ∈R N×W This indicates positional embedding.

[0042] The transform neural network encoder consists of a B-layer multi-head self-attention mechanism and a multi-layer perceptron module. Therefore, the output of the b-th layer can be written as follows:

[0043] y′ b =MultiheadSelfAttention(Normalization(y b-1 ))+r b-1

[0044] y b =MultiLayerPerceptron(Normalization(y′ b ))+y′ b

[0045] Where Normalization(·) represents the layer normalization operator, which is y b Encoded image representation.

[0046] (3) A hybrid convolutional neural network-transformer neural network model as the encoder. The transformer neural network segmentation model does not use a transformer neural network as the encoder, but instead uses a hybrid convolutional neural network-transformer neural network model, in which the convolutional neural network is first used as a feature extractor to generate feature maps for the input. Patch embedding is applied to the extracted 1×1 patches from the convolutional neural network feature maps instead of from the original image.

[0047] (4) Cascaded Upsampler. We introduce a Cascaded Upsampler (CUP), which consists of multiple upsampling steps used to decode hidden layer features to output the final segmentation mask. This involves upsampling the original hidden layer features from... Remodeling Then, we instantiate the cascaded upsampler by cascading multiple upsampling blocks to achieve the desired result from... The algorithm extends to C×K, where each block consists of a 2×2 upsampling operator, a 3×3 convolutional layer, and an activation function layer. The cascaded upsampling operators, together with the hybrid encoder, form a U-shaped architecture that allows for feature aggregation at different resolution levels through skip connections.

[0048] Secondly, the YOLO target detection algorithm is used to locate monitoring points in atypical small areas in real time.

[0049] Finally, through the preceding steps, we can obtain the locations of atypical small regions in the colposcopy image data using the predicted bounding boxes, and these predicted bounding boxes have been filtered. These filtered boxes can be directly plotted on the image to obtain the results.

[0050] It can be observed that in this embodiment, a two-stage deep learning model combining a U-shaped transform convolutional neural network (UTC) model and a YOLO neural network model is adopted. In the first stage, the biopsy location is manually delineated, making the delineation result highly interpretable. The U-shaped transform convolutional neural network learns its features and outputs atypical small regions. Then, in the second stage, YOLO selects and outputs highly suspicious biopsy regions, thereby generating a model that is both high-performance and highly interpretable. To address the loss of feature resolution in atypical small regions, a U-shaped transform convolutional neural network is proposed in the first stage of the two-stage deep learning model. This compensates for the loss of feature resolution in atypical small regions caused by the transform neural network, achieving accurate localization. To address the issue of lost detail in atypical small areas due to the constant movement of the colposcope during colposcopy, a target detection algorithm is used in the second stage of the two-stage deep learning model to locate monitoring points in these areas. Furthermore, to address the problem of lost detail in atypical small areas caused by limited lighting in colposcopy images and the significant influence of ambient light on images from different colposcopes, and to meet the data requirements of the two-stage deep learning model, a perfect world reflection algorithm is used for image preprocessing. This embodiment addresses three technical problems of existing colposcopy image algorithms: lack of interpretability, insufficient sensitivity to detail in atypical small areas, and loss of detail in atypical small areas. It enables highly accurate predictions for atypical small areas and monitoring points.

[0051] The present invention also provides a colposcopy image region recognition device based on a two-stage deep learning network, which can identify the location and monitoring points of atypical small regions through colposcopy images.

[0052] This invention also proposes a colposcopy image region recognition device based on a two-stage deep learning network. In this embodiment, the colposcopy image region recognition device based on a two-stage deep learning network includes an acquisition module, a construction module, a training module, and a prediction module;

[0053] The acquisition module is used to collect different image data using a colposcope;

[0054] The building module is used to construct a two-stage deep learning model of the U-shaped transform convolutional neural network model - YOLO neural network model;

[0055] The training module is used to train a two-stage deep learning model of the U-shaped transform convolutional neural network model - YOLO neural network model.

[0056] The prediction module is used to predict atypical small regions and monitoring points in colposcopy image data based on the two-stage deep learning model of the trained U-shaped transform convolutional neural network model-YOLO neural network model.

[0057] Optionally, this training module can be specifically used for:

[0058] A U-shaped transform convolutional neural network is constructed, and the edge of each atypical small region in the image data is marked by the U-shaped transform convolutional neural network. The image features of each atypical small region are extracted from the image data after the edge of the atypical small region is marked by the U-shaped transform convolutional neural network. Color features, texture features, shape features and spatial relationship features are extracted from the image features and used as training inputs for the U-shaped transform convolutional neural network. The constructed U-shaped transform convolutional neural network model is trained by the U-shaped transform convolutional neural network with color features, texture features, shape features and spatial relationship features as training inputs.

[0059] Optionally, the training module is specifically used for:

[0060] A YOLO neural network is constructed using the training results of a U-shaped transform convolutional neural network. The YOLO neural network is then used to label the edges of corresponding monitoring points in each image from the U-shaped transform convolutional neural network training results. The YOLO neural network extracts image features of each monitoring point from the image data after the edges of the monitoring points are labeled. Color features, texture features, shape features, and spatial relationship features are extracted from the image features and used as training inputs for the YOLO neural network. The constructed YOLO neural network model is then trained using the YOLO neural network with color features, texture features, shape features, and spatial relationship features as training inputs.

[0061] The training module is specifically used for:

[0062] The perfect world reflection algorithm is used to preprocess the image data acquired by colposcopy. Based on the trained U-shaped transform convolutional neural network model, color features, texture features, shape features, and spatial relationship features associated with the atypical small regions are extracted from the preprocessed image data. These extracted features are then input into the trained U-shaped transform convolutional neural network model to identify the atypical small regions in the image. The trained U-shaped transform convolutional neural network model yields the predicted bounding boxes of the atypical small regions. Based on the output atypical small region results, color features, texture features, shape features, and spatial relationship features associated with the monitoring points are extracted. These features are then input into the trained YOLO neural network model to identify the atypical small regions in the image. The trained YOLO neural network model yields the predicted monitoring point locations.

[0063] Each unit module of the colposcopy image region recognition device based on a two-stage deep learning network can execute the corresponding steps in the above method embodiments, respectively or correspondingly. Therefore, each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0064] The present invention also provides a colposcopy image region recognition device based on a two-stage deep learning network, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described colposcopy image region recognition method based on a two-stage deep learning network.

[0065] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0066] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0067] The present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.

[0068] It can be observed that the above scheme employs a two-stage deep learning model combining a U-shaped transform convolutional neural network (UTC) model and a YOLO neural network model. In the first stage, the biopsy location is manually delineated, ensuring high interpretability. The U-shaped transform convolutional neural network learns its features and outputs atypical small regions. Then, in the second stage, YOLO selects and outputs highly suspicious biopsy regions, thus generating a model that is both high-performance and highly interpretable. To address the loss of feature resolution in atypical small regions, a U-shaped transform convolutional neural network is proposed in the first stage of the two-stage deep learning model. This compensates for the loss of feature resolution in atypical small regions caused by the transform neural network, achieving accurate localization. To address the issue of lost detail in atypical small areas due to the constant movement of the colposcope during colposcopy, a target detection algorithm is used in the second stage of the two-stage deep learning model to locate monitoring points in these areas. Furthermore, to address the problem of lost detail in atypical small areas caused by limited lighting in colposcopy images and the significant influence of ambient light on images from different colposcopes, and to meet the data requirements of the two-stage deep learning model, a perfect world reflection algorithm is used for image preprocessing. This embodiment addresses three technical problems of existing colposcopy image algorithms: lack of interpretability, insufficient sensitivity to detail in atypical small areas, and loss of detail in atypical small areas. It enables highly accurate predictions for atypical small areas and monitoring points.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection of apparatuses or units, and may be electrical, mechanical, or other forms.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0071] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for recognizing colposcopy image regions based on a two-stage deep learning network, characterized in that, include: Different imaging data were collected using a colposcope; Construct a U-shaped transform convolutional neural network model based on image data; The architecture of the U-shaped transform convolutional neural network model is as follows: Image serialization involves splitting an image into a sequence of patches. Patch Embedding: Encoding Patch Features and Location Information A hybrid encoder is constructed based on convolutional neural networks (CNNs) and transform neural networks (TNs). The CNNs serve as feature extractors, extracting multi-scale local feature maps from the image and patching these feature maps. The transform neural networks consist of several coding layers, each composed of multi-head self-attention and a multilayer perceptron. Combined with layer normalization and residual connections, the global dependencies between patches are captured. By decoding the hidden features of the transform neural network through cascaded upsampling blocks, the image resolution is restored, forming a U-shaped architecture; The constructed U-shaped transform convolutional neural network model is trained using a U-shaped transform convolutional neural network; Based on the trained U-shaped transform convolutional neural network model, the training results of predicting atypical small regions in the image data are output. Construct a YOLO neural network model based on the training results of the U-shaped transform convolutional neural network; The YOLO neural network model is trained using a YOLO neural network. Based on the two-stage deep learning model of the trained U-shaped transform convolutional neural network model-YOLO neural network model, the atypical small regions and monitoring points corresponding to the colposcopy image data are predicted. The two-stage deep learning model includes atypical small region segmentation of colposcopy images and biopsy location guidance; the training uses the first network trained and fixed for the training of the second network; a U-shaped transform convolutional neural network is used in the first stage of the two-stage deep learning model; and an object detection algorithm is used in the second stage of the two-stage deep learning model to locate monitoring points in atypical small regions.

2. The colposcopy image region recognition method based on a two-stage deep learning network as described in claim 1, characterized in that, The step of training the constructed U-transform convolutional neural network model using a U-transform convolutional neural network includes: A U-shaped transform convolutional neural network is constructed, and the edge of each atypical small region in the image data is marked by the U-shaped transform convolutional neural network. The image features of each atypical small region are extracted from the image data after the edge of the atypical small region is marked by the U-shaped transform convolutional neural network. Color features, texture features, shape features and spatial relationship features are extracted from the image features and used as training inputs of the U-shaped transform convolutional neural network. The constructed U-shaped transform convolutional neural network model is trained by the U-shaped transform convolutional neural network with color features, texture features, shape features and spatial relationship features as training inputs.

3. The colposcopy image region recognition method based on a two-stage deep learning network as described in claim 1, characterized in that, The step of training the YOLO neural network model using a YOLO neural network includes: A YOLO neural network is constructed using the training results of a U-shaped transform convolutional neural network. The YOLO neural network is then used to label the edges of corresponding monitoring points in each image from the U-shaped transform convolutional neural network training results. The YOLO neural network extracts image features of each monitoring point from the image data after the edges of the monitoring points are labeled. Color features, texture features, shape features, and spatial relationship features are extracted from the image features and used as training inputs for the YOLO neural network. The constructed YOLO neural network model is then trained using the YOLO neural network with the color features, texture features, shape features, and spatial relationship features as training inputs.

4. The colposcopy image region recognition method based on a two-stage deep learning network as described in claim 1, characterized in that, The method of using a two-stage deep learning model based on the trained U-shaped transform convolutional neural network model-YOLO neural network model to predict atypical small regions and monitoring points in colposcopy image data includes: The image data acquired by colposcopy is preprocessed, and color features, texture features, shape features, and spatial relationship features associated with the atypical small regions are extracted from the preprocessed image data according to a trained U-shaped transform convolutional neural network model. The extracted color features, texture features, shape features, and spatial relationship features associated with the atypical small regions are input into the trained U-shaped transform convolutional neural network model to identify the atypical small regions in the image. The bounding boxes of the predicted atypical small regions are obtained through the trained U-shaped transform convolutional neural network model. Based on the output atypical small region results, color features, texture features, shape features, and spatial relationship features associated with the monitoring points are extracted. The extracted color features, texture features, shape features, and spatial relationship features associated with the monitoring points are input into a trained YOLO neural network model to identify the atypical small regions in the image. The predicted monitoring point positions are obtained through the trained YOLO neural network model.

5. A colposcopy image region recognition device based on a two-stage deep learning network, characterized in that, The apparatus is used to implement the method according to any one of claims 1-4, the apparatus comprising: Acquisition module, construction module, training module, and prediction module; The acquisition module is used to collect different image data using a colposcope; The building module is used to construct a two-stage deep learning model of the U-shaped transform convolutional neural network model - YOLO neural network model; The training module is used to train a two-stage deep learning model of the U-shaped transform convolutional neural network model - YOLO neural network model. The prediction module is used to predict atypical small regions and monitoring points in colposcopy image data based on the two-stage deep learning model of the trained U-shaped transform convolutional neural network model-YOLO neural network model.

6. The colposcopy image region recognition device based on a two-stage deep learning network as described in claim 5, characterized in that, The training module is specifically used for: A U-shaped transform convolutional neural network is constructed, and the edge of each atypical small region in the image data is marked by the U-shaped transform convolutional neural network. The image features of each atypical small region are extracted from the image data after the edge of the atypical small region is marked by the U-shaped transform convolutional neural network. Color features, texture features, shape features and spatial relationship features are extracted from the image features and used as training inputs of the U-shaped transform convolutional neural network. The constructed U-shaped transform convolutional neural network model is trained by the U-shaped transform convolutional neural network with color features, texture features, shape features and spatial relationship features as training inputs.

7. The colposcopy image region recognition device based on a two-stage deep learning network as described in claim 5 or 6, characterized in that, The training module is specifically used for: A YOLO neural network is constructed using the training results of a U-shaped transform convolutional neural network. The YOLO neural network is then used to label the edges of corresponding monitoring points in each image from the U-shaped transform convolutional neural network training results. The YOLO neural network extracts image features of each monitoring point from the image data after the edges of the monitoring points are labeled. Color features, texture features, shape features, and spatial relationship features are extracted from the image features and used as training inputs for the YOLO neural network. The constructed YOLO neural network model is then trained using the YOLO neural network with the color features, texture features, shape features, and spatial relationship features as training inputs.

8. The colposcopy image region recognition device based on a two-stage deep learning network as described in claim 5, characterized in that, The training module is specifically used for: After preprocessing the image data, color features, texture features, shape features, and spatial relationship features associated with the atypical small regions are extracted from the preprocessed image data according to the trained U-shaped transform convolutional neural network model. These extracted features are then input into the trained U-shaped transform convolutional neural network model to identify the atypical small regions in the image. The trained U-shaped transform convolutional neural network model yields the predicted bounding boxes of the atypical small regions. Based on the output atypical small region results, color features, texture features, shape features, and spatial relationship features associated with the monitoring points are extracted. These features are then input into the trained YOLO neural network model to identify the atypical small regions in the image. The trained YOLO neural network model yields the predicted monitoring point locations.

9. A colposcopy image region recognition device based on a two-stage deep learning network, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the colposcopy image region recognition method based on a two-stage deep learning network as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the colposcopy image region recognition method based on a two-stage deep learning network as described in any one of claims 1 to 4.

Images