Methods, apparatus, equipment and media for determining regions of interest in medical images

Abstract

This application discloses a method, apparatus, device, and medium for determining a region of interest (ROI) in a medical image. The method is applicable to the field of smart healthcare technology. The method includes: determining n edge key points of the medical image, where the n edge key points are used to locate the edges of the ROI, and n is an integer greater than 1; determining a bounding box for the ROI based on the n edge key points; acquiring semantic and detail features of the medical image within the bounding box; and identifying the irregular contour of the ROI based on the semantic and detail features. This method determines the ROI through edge key points, yielding relatively accurate results.

Description

Technical Field

[0001] This application relates to the field of smart medical technology, and in particular to a method, apparatus, device, and medium for determining a region of interest in a medical image. Background Technology

[0002] The Response Evaluation Criteria in Solid Tumors (RECIST) sets out standards for tumor evaluation and applies these standards to areas such as tumor research, disease assessment, evaluation of tumor treatment efficacy, and tumor progression.

[0003] The relevant technique utilizes an attention mechanism and a Co-seg (Co-segmentation) model to compare the similarity between the target region and normal regions in a medical image. When the similarity exceeds a preset value, the target region is considered to be a lesion. The segmentation supervision information for the Co-seg model comes from binary masks in an existing dataset.

[0004] However, the accuracy of the edges of the lesion area obtained by the related technology is not high enough. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for determining regions of interest in medical images. It enables the generation of a dynamic bounding box from edge key points in the medical image, and then the precise irregular edges of the region of interest are determined within the bounding box. The technical solution is as follows:

[0006] According to one aspect of this application, a method for determining a region of interest in a medical image is provided, the method comprising:

[0007] Determine n edge key points of the medical image, the n edge key points are used to locate the region edge of the region of interest, and n is an integer greater than 1;

[0008] The location box of the region of interest is determined based on the n edge key points;

[0009] Obtain the semantic and detail features of the medical image within the positioning bounding box;

[0010] Based on the semantic features and the detailed features, the irregular contours of the region of interest are identified.

[0011] According to another aspect of this application, an apparatus for determining a region of interest in a medical image is provided, the apparatus comprising:

[0012] A determination module is used to determine n edge key points of the medical image, wherein the n edge key points are used to locate the region edge of the region of interest, and n is an integer greater than 1;

[0013] The determining module is further configured to determine the location box of the region of interest based on the n edge key points;

[0014] The extraction module is used to obtain the semantic and detail features of the medical image within the positioning frame;

[0015] The recognition module is used to identify the irregular contour of the region of interest based on the semantic features and the detailed features.

[0016] In an optional design of this application, the recognition module is further configured to fuse the semantic features and the detail features to obtain fused features; and invoke a segmentation network to identify the irregular contour of the region of interest from the fused features.

[0017] In an optional design of this application, the recognition module is further configured to invoke the segmentation network, perform convolution and normalization on the fused features to obtain a segmentation probability map, wherein the pixel values ​​of the segmentation probability map are used to represent the probability that the corresponding pixel belongs to the region of interest; binarize the segmentation probability map to obtain a binary mask; and determine the irregular contour of the region of interest based on the intersection of the first value and the second value in the binary mask.

[0018] In an optional design of this application, the extraction module is further configured to: select a first enclosing region from the semantic feature map of the medical image using the positioning box to obtain the semantic features in the positioning box; select a second enclosing region from the medical image using the positioning box to obtain a region image; and input the region image into a residual network to obtain the detail features in the positioning box.

[0019] In an optional design of this application, the segmentation network and / or the residual network are obtained by weakly supervised training on the foreground and background of the region of interest; the foreground is a quadrilateral region composed of two major axis keypoints and two minor axis keypoints; the background is the outer region of the minimum circumcircle of the foreground, the two major axis keypoints are the two endpoints of the longest first line segment in the region of interest, and the two minor axis keypoints are the two endpoints of the longest second line segment in the region of interest that is perpendicular to the first line segment.

[0020] In an optional design of this application, the determining module is further configured to invoke the hourglass backbone network to downsample and upsample the medical image to restore it, and output the semantic feature map of the medical image; and invoke the prediction network to identify the n edge key points from the semantic feature map of the medical image.

[0021] In an optional design of this application, the extraction module is further configured to downsample the medical image through a downsampling layer in the hourglass backbone network to obtain a downsampled medical image, wherein the downsampling layer includes cascaded a-layer network layers, where a is an integer greater than 1; and to upsample and restore the downsampled medical image through an upsampling layer in the hourglass backbone network to obtain the semantic feature map of the medical image, wherein the upsampling layer includes cascaded a-layer network layers corresponding to the downsampling layer; wherein the input of the i-th layer in the upsampling layer is the upsampling and restoration result of the (i-1)-th layer in the upsampling layer and the downsampling result of the corresponding i-th layer in the downsampling layer, where i is an integer greater than 1 and less than a+1.

[0022] In an optional design of this application, the determining module is further configured to determine the bounding rectangle of the n edge key points as the positioning box of the region of interest.

[0023] In an optional design of this application, the determining module is further configured to determine the upper boundary of the circumscribed rectangle based on the uppermost edge key point among the n edge key points; determine the lower boundary of the circumscribed rectangle based on the lowermost edge key point among the n edge key points; determine the left boundary of the circumscribed rectangle based on the leftmost edge key point among the n edge key points; determine the right boundary of the circumscribed rectangle based on the rightmost edge key point among the n edge key points; and determine the rectangle formed by the upper boundary, the lower boundary, the left boundary, and the right boundary as the positioning box of the region of interest.

[0024] According to another aspect of this application, a method for training an image processing model is provided, the method comprising:

[0025] Obtain a sample training set, which includes sample medical images and ground truth annotations, wherein the ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1;

[0026] The hourglass backbone network and the prediction network are invoked to extract and determine n predicted edge key points of the sample medical image. The n predicted edge key points are used to locate the edge of the region of interest, where n is an integer greater than 1.

[0027] The predicted location box of the region of interest is determined based on the n predicted edge key points.

[0028] Obtain the predicted semantic features and predicted detail features of the sample medical image within the predicted localization box;

[0029] The segmentation network is invoked to identify the predicted irregular contours of the predicted region of interest based on the predicted semantic features and the predicted detail features.

[0030] Based on the loss difference between the n predicted keypoints and the actual annotations, the hourglass backbone network and the segmentation network are subjected to weakly supervised training.

[0031] According to another aspect of this application, a training apparatus for an image processing model is provided, the apparatus comprising:

[0032] The acquisition module is used to acquire a sample training set, which includes sample medical images and ground truth annotations. The ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1.

[0033] The determination module is used to call the hourglass backbone network and the prediction network to extract and determine n predicted edge key points of the sample medical image. The n predicted edge key points are used to locate the edge of the region of interest, where n is an integer greater than 1.

[0034] The determination module is also used to determine the predicted location box of the region of interest based on the n predicted edge key points;

[0035] The extraction module is used to obtain the predicted semantic features and predicted detail features of the sample medical image within the predicted localization box;

[0036] The identification module is also used to invoke the segmentation network to identify the predicted irregular contour of the predicted region of interest based on the predicted semantic features and the predicted detail features.

[0037] The training module is used to perform weakly supervised training on the hourglass backbone network and the segmentation network based on the loss difference between the n predicted edge key points and the real annotations.

[0038] In an optional design of this application, the recognition module is further configured to fuse the predicted semantic features and the predicted detail features to obtain predicted fusion features; and to invoke the segmentation network to identify the predicted irregular contour of the predicted region of interest from the predicted fusion features.

[0039] In an optional design of this application, the recognition module is further configured to invoke the segmentation network to convolve and normalize the predicted fusion features to obtain a predicted segmentation probability map, wherein the pixel values ​​of the predicted segmentation probability map are used to represent the probability that the corresponding pixel belongs to the predicted region of interest; to binarize the predicted segmentation probability map to obtain a predicted binary mask; and to determine the predicted irregular contour of the predicted region of interest based on the intersection of the first value and the second value in the predicted binary mask.

[0040] In an optional design of this application, the training module is further configured to determine two predicted major axis keypoints and two predicted minor axis keypoints based on the n predicted edge keypoints, wherein the two predicted major axis keypoints are the two endpoints of the longest first line segment in the predicted region of interest, and the two predicted minor axis keypoints are the two endpoints of the longest second line segment in the predicted region of interest that is perpendicular to the first line segment; define the quadrilateral region formed by the two predicted major axis keypoints and the two predicted minor axis keypoints as the foreground; define the region outside the smallest circumcircle of the foreground as the background; calculate the cross-entropy loss of the foreground and the background to obtain the partial cross-entropy loss; and perform weakly supervised training on the hourglass backbone network and the segmentation network based on the partial cross-entropy loss.

[0041] In an optional design of this application, the training module is further configured to generate a predicted segmentation probability map based on the predicted semantic feature map, wherein the pixel values ​​in the predicted segmentation probability map are used to represent the probability that the corresponding pixel belongs to the predicted region of interest; perform gradient enhancement on the predicted segmentation probability map and the sample medical image to obtain a first gradient intensity map of the predicted segmentation probability map and a second gradient intensity map of the sample medical image, respectively; calculate the edge loss based on the sum of the products of each element in the first gradient intensity map and the second gradient intensity map; and perform weakly supervised training on the hourglass backbone network and the segmentation network based on the edge loss.

[0042] According to another aspect of this application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the method for determining a region of interest in a medical image or the training method for an image processing model as described above.

[0043] According to another aspect of this application, a computer storage medium is provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the program code being loaded and executed by a processor to implement the method for determining regions of interest in medical images or the training method for image processing models as described above.

[0044] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for determining a region of interest in a medical image or the method for training an image processing model as described above.

[0045] The beneficial effects of the technical solutions provided in this application include at least the following:

[0046] In the first stage, bounding boxes surrounding the region of interest (ROI) are dynamically determined based on edge key points in the medical image. In the second stage, a more accurate irregular contour of the ROI is obtained based on semantic and detail features. On one hand, the bounding boxes determine the approximate range of the ROI, allowing the second-stage calculation to be performed only on the image region within the bounding boxes, thus simplifying subsequent computations. On the other hand, incorporating detail features and using a fusion of semantic and detail features yields a more accurate irregular contour of the ROI. Furthermore, when training the image processing model, only edge key point annotations are needed as supervision information; specific irregular contour annotations are not required. Moreover, edge key point annotations are easier to obtain than specific irregular contour annotations, reducing the cost of manual annotation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the structure of an image processing model provided in an exemplary embodiment of this application;

[0049] Figure 2 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application;

[0050] Figure 3 This is a flowchart illustrating a method for determining a region of interest in a medical image according to an exemplary embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of an image processing model provided in an exemplary embodiment of this application;

[0052] Figure 5 This is a flowchart illustrating a method for determining a region of interest in a medical image according to an exemplary embodiment of this application;

[0053] Figure 6 This is a flowchart illustrating a method for determining a region of interest in a medical image according to an exemplary embodiment of this application;

[0054] Figure 7 This is a flowchart illustrating a training method for an image processing model provided in an exemplary embodiment of this application;

[0055] Figure 8 This is a flowchart illustrating an exemplary embodiment of the present application;

[0056] Figure 9 This is a schematic diagram of gradient enhancement provided in an exemplary embodiment of this application;

[0057] Figure 10 This is a flowchart illustrating a training method for an image processing model provided in an exemplary embodiment of this application;

[0058] Figure 11 This is a schematic diagram of the structure of a device for determining a region of interest in a medical image provided in an exemplary embodiment of this application;

[0059] Figure 12 This is a schematic diagram of the structure of a training device for an image processing model provided in an exemplary embodiment of this application;

[0060] Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0062] First, let's introduce the terms used in the embodiments of this application:

[0063] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0064] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0065] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0066] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0067] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0068] The Response Evaluation Criteria in Solid Tumours (RECIST) is a set of standards for measuring tumor lesions and evaluating the effectiveness of tumor treatment, published in 1999. RECIST proposes a one-dimensional measurement method that uses the length of the longest diameter of the tumor instead of its area to represent tumor size. RECIST endpoints refer to the two endpoints of the longest line segment within the tumor lesion area, as well as the two endpoints of the longest line segment perpendicular to the longest diameter within the tumor lesion area.

[0069] Picture Archiving and Communication Systems (PACS): These systems are used in hospital radiology departments. Their primary task is to digitally store massive amounts of various medical images generated daily (including images from MRI, Computed Tomography (CT), ultrasound, X-ray machines, various infrared instruments, microscopes, etc.) through various interfaces. These images can be quickly retrieved and used when needed, while also providing auxiliary diagnostic and management functions. PACS plays a crucial role in data transmission between various imaging devices and in organizing data storage.

[0070] Heatmaps: These display data in a dataset using special highlighting techniques. Heatmaps include at least one of the following: click heatmaps, attention heatmaps, analytics heatmaps, comparison heatmaps, sharing heatmaps, overlay heatmaps, and historical heatmaps. For example, in an attention heatmap, the total time a user spends at different locations on the website is displayed; the longer a user spends viewing a particular page area, the closer that page area's color will be to the infrared spectrum.

[0071] Mask: A bitmap used for AND operations. The pixel values ​​of the pixels in this bitmap are only 0 and 1. A mask can extract the part of the image that the user is interested in from the original image.

[0072] The Prewitt operator is a first-order differential operator for edge detection. It utilizes the gray-level differences between the top and bottom, left and right neighbors of a pixel to detect edges by reaching extreme values, removing some false edges, and smoothing noise. Its principle involves performing neighborhood convolution with the image using two directional templates in the image space; one template detects horizontal edges, and the other detects vertical edges.

[0073] ROI (Region of Interest): In fields such as machine vision and image processing, the region that needs to be processed from the image to be processed is called the region of interest.

[0074] Figure 1 A schematic diagram of an image processing model provided in an exemplary embodiment of this application is shown. The image processing model includes an hourglass backbone network 102, a prediction network 104, a segmentation network 110, and a residual network 107.

[0075] The hourglass backbone network 102 is used to extract semantic features from medical images and generate a semantic feature map. The input of the hourglass backbone network 102 is the medical image 101, and the output is the semantic feature map 103. The hourglass backbone network 102 has a symmetrical structure. For example, the hourglass backbone network 102 downsamples and then upsamples the medical image 101 to restore it and obtain the semantic feature map.

[0076] Prediction network 104 extracts edge keypoints from the semantic feature map and generates bounding boxes based on these keypoints. The bounding boxes are the circumscribed rectangles in the semantic feature map that enclose the edge keypoints. The input to prediction network 104 is the semantic feature map 103, and the output is the edge keypoints 105. Furthermore, the bounding boxes generated from the edge keypoints limit the computational scope of segmentation network 110, thereby reducing its computational load. Figure 1 As shown, after the semantic feature map 103 is combined with the positioning box, the semantic feature 106 in the positioning box within the semantic feature map 103 is obtained.

[0077] The residual network 107 is used to extract detail features from the input image. The input to the residual network 107 is the region image 112 in the medical image 101 corresponding to the region within the bounding box, and the output is detail features 108. For example, detail features include, but are not limited to, at least one of edges, corners, and textures.

[0078] After obtaining semantic feature 106 and detail feature 108, feature fusion is performed on semantic feature 106 and detail feature 108 to obtain fused feature 109. Since semantic feature 106 is a high-level abstract feature, while detail feature 108 is processed at the original resolution of the medical image, it has richer details.

[0079] The segmentation network 110 is used to identify the region of interest (ROI) from the input features. The input to the segmentation network 110 is the fused feature 109, and the output is the ROI 111. Further, the ROI 111 is the portion within the white box in the medical image 113. Optionally, the segmentation network 110 convolves and normalizes the fused feature 109 to obtain a segmentation probability map; it binarizes the segmentation probability map to obtain a binary mask; and it determines the region contour of the ROI 111 based on the intersection of the first and second values ​​in the binary mask.

[0080] Figure 2A structural block diagram of a computer system provided in an exemplary embodiment of this application is shown. The computer system 200 includes: a first terminal 220, a server 240, and a second terminal 260.

[0081] The first terminal 220 has an application installed related to determining the region of interest. This application can be a small app within a web application, a dedicated application, or a web client. The first terminal is at least one of a smartphone, tablet, e-book reader, MP3 player, MP4 player, laptop, and desktop computer. For example, a user sends medical images to a server 240 via the first terminal.

[0082] The first terminal is connected to the server 240 via a wireless or wired network.

[0083] Server 240 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, server 240 undertakes the main computing work, and the first terminal undertakes the secondary computing work; or, server 240 undertakes the secondary computing work, and the first terminal undertakes the main computing work; or, server 240 and the first terminal use a distributed computing architecture for collaborative computing. In the embodiments of this application, each step is executed by the first terminal alone, or by server 240 alone, or by the first terminal and server 240 jointly. For example, server 240 receives a medical image sent by the first terminal, and server 240 determines the region of interest in the medical image.

[0084] The second terminal 260 also has an application related to tumor evaluation installed. This application can be a small program within an app, a dedicated application, or a web client. The first terminal is at least one of a smartphone, tablet, e-book reader, MP3 player, MP4 player, laptop, and desktop computer. Optionally, the first terminal 220 and the second terminal 260 can be the same terminal or different terminals. For example, the first terminal 220 is the terminal used by the patient, and the second terminal 260 is the terminal used by the doctor. The patient uploads medical images using the first terminal 220. After receiving the medical images sent by the first terminal 220, the server 240 determines the region of interest in the medical images. Then, the server 240 sends the image of the region of interest to the second terminal 260, allowing the doctor to view the patient's condition on the second terminal 260.

[0085] Optionally, the clients installed on the first terminal 220 and the second terminal 260 may be the same, or the same type of application on different platforms. The first terminal 220 can refer to one of multiple terminals, and the second terminal 260 can refer to one of multiple terminals; this embodiment only uses the first terminal 220 and the second terminal 260 as examples. The device types of the first terminal 220 and the second terminal 260 may be the same or different, and these device types include at least one of the following: smartphones, tablets, e-book readers, MP3 players, MP4 players, laptops, and desktop computers.

[0086] Figure 3 This illustration shows a schematic diagram of the process for determining a region of interest in a medical image according to an exemplary embodiment of this application. The method comprises... Figure 2 The computer system 200 shown executes a method comprising the following steps:

[0087] Step 302: Determine n edge key points of the medical image. The n edge key points are used to locate the region edge of the region of interest, where n is an integer greater than 1.

[0088] Edge keypoints are points on the edge of the region of interest.

[0089] Medical images include images produced by at least one of the following medical devices: MRI, computed tomography (CT), ultrasound, X-ray machine, various infrared spectrometers, and microscope. For example, a medical image is a CT image of a user's lymph nodes.

[0090] Optionally, edge keypoints are also used to evaluate the region of interest. For example, when the region of interest is a tumor lesion in a medical image, and the edge keypoints are the two major endpoints and two minor endpoints in the RECIST criteria, the two major endpoints and two minor endpoints can be used to evaluate the treatment status of the tumor lesion.

[0091] Optionally, the n edge critical points are the two major axis endpoints and two minor axis endpoints in the RECIST standard.

[0092] Optionally, the region of interest is the lesion area of ​​the tumor, and the marginal key points are RECIST endpoints.

[0093] Step 304: Determine the bounding box of the region of interest based on n edge key points.

[0094] Bounding boxes can select candidate regions from an image. Since only the irregular contours of the region of interest need to be identified within the bounding box, the computational load on computer equipment can be reduced.

[0095] Optionally, the positioning frame includes, but is not limited to, at least one of a rectangle, triangle, circle, and pentagon.

[0096] Optionally, the positioning box is the smallest bounding rectangle of n edge key points.

[0097] Optionally, when the n edge keypoints are two major axis keypoints and two minor axis keypoints, the bounding rectangle of the two major axis keypoints and two minor axis keypoints is determined; the bounding rectangle is defined as the location box of the region of interest. Optionally, the position of the bounding rectangle is determined based on the coordinates of the two major axis keypoints and two minor axis keypoints. For example, if the coordinates of the two major axis keypoints are (1, 1) and (10, 15), and the coordinates of the two minor axis keypoints are (4, 7) and (5, 4), and the bounding rectangle is required to contain the two major axis keypoints and two minor axis keypoints, then the bounding rectangle is a rectangle formed by the lines x = 1, y = 1, x = 10, and y = 15.

[0098] Step 306: Obtain the semantic and detail features of the medical image within the bounding box.

[0099] Semantic features are used to represent the semantics in medical images. For example, a semantic feature might indicate that region A in a medical image is a lesion.

[0100] Detail features include, but are not limited to, at least one of the following in medical images: edges, corners, and textures.

[0101] Optionally, a semantic recognition model can be invoked to extract semantic features from the medical image. Optionally, when the medical image is a CT image, multiple sets of continuous axial slices of the CT image can be used as a set of multi-channel images, and this set of multi-channel images can be used as input for the medical image.

[0102] Step 308: Identify the irregular contours of the region of interest based on semantic and detail features.

[0103] Optionally, a segmentation network is invoked to identify the irregular contours of the region of interest. Specifically, the method includes the following sub-steps:

[0104] 1. By fusing semantic features and detail features, the fused features of medical images are obtained.

[0105] Optionally, semantic features and detail features can be combined using an adder to obtain fused features of the medical image.

[0106] Among them, the fusion feature is used to determine the general outline of the region of interest, and the detail feature is used to determine the edges of the specific irregular outline of the region of interest.

[0107] 2. Call the segmentation network to identify the irregular contours of the region of interest from the fused features.

[0108] Segmentation networks are used to extract regions of interest from fused features.

[0109] In medical images, regions of interest (ROIs) often exhibit irregular shapes. For example, when the ROI is the lesion area of ​​a tumor, the lesion area will appear irregular in shape.

[0110] In summary, this embodiment dynamically determines the bounding box surrounding the region of interest (ROI) in the first stage based on the edge key points of the medical image; in the second stage, it obtains a more accurate irregular contour of the ROI based on semantic and detail features. On the one hand, the bounding box determines the approximate range of the ROI, and the second stage calculation is performed only on the image region within the bounding box, thus simplifying the computational load of subsequent calculations. On the other hand, by incorporating detail features and using a fusion of semantic and detail features, a more accurate irregular contour of the ROI can be obtained. Furthermore, when training the image processing model, only the annotations of the edge key points are needed as supervision information; specific irregular contour annotations are not required. Moreover, edge key point annotations are easier to obtain than specific irregular contour annotations, reducing the cost of manual annotation.

[0111] In the following embodiments, a residual network is introduced to identify the irregular contours of the region of interest (ROI) through semantic features within the bounding box, and detailed features from medical images are incorporated to improve the segmentation accuracy of the ROI. For the specific structure of the image processing model, please refer to [reference needed]. Figure 1 The model structure shown is illustrated. On the other hand, the hourglass backbone network is used to extract semantic features from medical images, which yields good extraction results and high accuracy of semantic features.

[0112] Figure 4 This illustration shows a schematic diagram of the process for determining a region of interest in a medical image according to an exemplary embodiment of this application. The method comprises... Figure 2 The computer system 200 shown executes a method comprising the following steps:

[0113] Step 401: Call the hourglass backbone network to downsample and upsample the medical image to restore it, and output the semantic feature map of the medical image.

[0114] The hourglass backbone network package includes at least one of a downsampling layer and an upsampling restoration layer.

[0115] Optionally, the medical image is downsampled through a downsampling layer in the hourglass backbone network to obtain a downsampled medical image.

[0116] The downsampling layer is used to reduce the resolution of medical images and extract semantic features from them. The downsampling layer consists of cascaded 'a' network layers, where 'a' is an integer greater than 1. Each network layer is used for downsampling, and the downsampling layer consists of cascaded 'a' network layers, where 'a' is an integer greater than 1.

[0117] A semantic feature map is an image composed of the semantic features of a medical image. The pixel value of each pixel in the semantic feature map represents the semantic features of the medical image.

[0118] Optionally, the downsampled medical image can be upsampled and restored using an upsampling layer in the hourglass backbone network to obtain the semantic feature map of the medical image.

[0119] Upsampling restoration is used to restore the downsampled medical image to its original size. The upsampling layer includes a cascaded network layer a corresponding to the downsampling layer. The input of the i-th network layer in the upsampling layer is the upsampling restoration result of the (i-1)-th network layer in the upsampling layer and the downsampling result of the i-th network layer in the downsampling layer. i is an integer greater than 1 and less than a+1.

[0120] Optionally, the upsampling restoration method includes at least one of the nearest neighbor method, bilinear interpolation method, and cubic interpolation method.

[0121] Step 402: Determine the n edge key points of the medical image.

[0122] Optionally, a prediction network is invoked to identify n edge keypoints from the semantic feature map of the medical image. The prediction network is used to extract edge keypoints from the semantic feature map.

[0123] Optionally, when the n edge keypoints are two major axis keypoints and two minor axis keypoints, the prediction network is invoked to identify the two major axis keypoints and two minor axis keypoints from the semantic feature map of the medical image.

[0124] Optionally, a predicted offset map is obtained; the predicted offsets corresponding to the two major axis keypoints and the two minor axis keypoints are determined in the predicted offset map; and the two major axis keypoints and the two minor axis keypoints are corrected using the predicted offsets.

[0125] Optionally, when the region of interest is the lesion area of ​​the tumor and the marginal key points are RECIST endpoints, this step includes the following sub-steps:

[0126] 1. Call the prediction network to generate heatmaps of two major path keypoints, two minor path keypoints, and the center point based on the semantic feature map.

[0127] The prediction network includes a heatmap generation network layer, which generates heatmaps of edge keypoints and center point based on the semantic feature map. Optionally, the heatmap generation network layer in the prediction network is invoked to generate heatmaps of two major axis keypoints, two minor axis keypoints, and the center point based on the semantic feature map.

[0128] The center point refers to the intersection of the major and minor axes, and it is both the midpoint of the major axis and the midpoint of the minor axis.

[0129] 2. Group the pixels at the first k peak positions in the heatmaps of the two major axis keypoints and the two minor axis keypoints to obtain keypoint combinations.

[0130] In the heatmaps of two major axis keypoints and two minor axis keypoints, the pixel value of a pixel represents the probability that the pixel is an edge keypoint. Therefore, the larger the pixel value, the more likely the corresponding pixel is to be an edge keypoint.

[0131] k is a positive integer, and its value is set by the technical personnel. For example, k is 40.

[0132] Optionally, a pixel is randomly selected from the pixels at the first k peak positions in the heatmaps of the two major axis keypoints and the two minor axis keypoints, and the four selected pixels are combined to achieve keypoint combination.

[0133] For example, there are four keypoint heatmaps. From each of the four heatmaps, 40 candidate keypoints are extracted. Figure 1 Randomly select a candidate keypoint from the corresponding candidate keypoints, and then select the keypoint hotspot. Figure 2 Randomly select a candidate keypoint from the corresponding candidate keypoints, and then select the keypoint hotspot. Figure 3 Randomly select a candidate keypoint from the corresponding candidate keypoints, and then select the keypoint hotspot. Figure 4 Randomly select one candidate key point from the corresponding candidate key points, and combine the randomly selected candidate key points to obtain the key point combination.

[0134] 3. Based on the combination of key points, determine the center point corresponding to the combination of key points in the center point heatmap.

[0135] For example, let the center point be denoted as c, the two endpoints of the major axis be denoted as l and r, and the two endpoints of the minor axis be denoted as t and b. Then the coordinates of the center point are: in and The x-coordinates of endpoints l and r, and The ordinates represent the endpoints t and b.

[0136] 4. Calculate the combined score of key point combinations and their corresponding center points.

[0137] For example, a method for calculating the combined score is given, which is to calculate the sum of the probability corresponding to the candidate key point and twice the probability of the center point to obtain the combined score.

[0138] 5. In response to the combined score of the keypoint combination reaching a preset threshold, the pixels in the keypoint combination are identified as two major axis keypoints and two minor axis keypoints.

[0139] If the combined score of a key point combination does not reach the preset threshold, the key point combination is discarded.

[0140] Optionally, in this sub-step, multiple keypoint combinations are obtained, and the pixels in the keypoint combination with the highest combination score among the aforementioned multiple keypoint combinations are determined as two major axis keypoints and two minor axis keypoints.

[0141] Step 403: Determine the bounding rectangle of the n edge key points as the location box of the region of interest.

[0142] For example, when the n edge keypoints are two major axis keypoints and two minor axis keypoints, the bounding rectangle of the quadrilateral formed by the two major axis keypoints and the two minor axis keypoints is determined as the location box of the region of interest.

[0143] Optionally, the upper boundary of the bounding rectangle is determined based on the topmost edge keypoint among the n edge keypoints; the lower boundary is determined based on the bottommost edge keypoint among the n edge keypoints; the left boundary is determined based on the leftmost edge keypoint among the n edge keypoints; and the right boundary is determined based on the rightmost edge keypoint among the n edge keypoints. The rectangle formed by the upper, lower, left, and right boundaries is then defined as the bounding box of the region of interest. For example, if the coordinates of the topmost edge keypoint are (3, 8), the coordinates of the bottommost edge keypoint are (2, 1), the coordinates of the leftmost edge keypoint are (1, 4), and the coordinates of the rightmost edge keypoint are (5, 4), and the bounding rectangle is required to contain two major axis keypoints and two minor axis keypoints, then the bounding box is a rectangle formed by the lines x = 1, y = 1, x = 5, and y = 8.

[0144] Step 404: Select the first enclosing region from the semantic feature map using a bounding box to obtain the semantic features within the bounding box.

[0145] The first enclosing region refers to the area within the location box in the semantic feature map.

[0146] Step 405: Select the second enclosed region from the medical image using a bounding box to obtain the region image.

[0147] A region image is the image in a medical image that corresponds to a bounding box.

[0148] The second enclosed region refers to the area within the positioning frame in a medical image.

[0149] Step 406: Input the region image into the residual network to obtain the detailed features within the localization box.

[0150] Residual networks are used to extract detailed features from region images. These detailed features include, but are not limited to, at least one of edges, corners, and textures.

[0151] The role of convolution is to classify each pixel within the bounding box and obtain a classification score. Normalization maps the classification scores of the pixels to the (0, 1) interval, and the result of the mapping is the probability that the pixel belongs to the region of interest.

[0152] The segmentation network layer and / or residual network layer are obtained by weakly supervising the training of the foreground and background of the region of interest. The foreground is a quadrilateral region consisting of two major axis keypoints and two minor axis keypoints; the background is the outer region of the smallest circumcircle of the foreground. The two major axis keypoints are the two endpoints of the longest first line segment in the region of interest, and the two minor axis keypoints are the two endpoints of the longest second line segment in the region of interest that is perpendicular to the first line segment.

[0153] Step 407: Fuse semantic features and detail features to obtain the fused features of the medical image.

[0154] The fusion features include semantic features and detailed features.

[0155] Optionally, semantic features and detail features can be combined using an adder to obtain fused features of the medical image.

[0156] Step 408: Call the segmentation network layer to perform convolution and normalization on the fused features to obtain the segmentation probability map.

[0157] The pixel values ​​in the segmentation probability map represent the probability that the corresponding pixel belongs to the region of interest. For example, if the pixel value of pixel 1 in the segmentation probability map is 80%, it means that the probability that pixel 1 belongs to the region of interest is 80%.

[0158] Since the regions corresponding to the fused features represent features of a medical image, these features need to be converted into probabilities. Convolution classifies each pixel within the bounding box, yielding a classification score. Normalization maps the classification scores of the pixels to the (0, 1) interval, resulting in the probability that a pixel belongs to the region of interest.

[0159] Alternatively, normalization can be performed using the softmax function (normalization exponential function) or the sigmoid function (often used as the activation function of neural networks, mapping variables to the (0, 1) interval).

[0160] Step 409: Binarize the segmentation probability map to obtain a binary mask.

[0161] A binary mask refers to an image where each pixel has exactly two possible values; therefore, in this embodiment, it can also be referred to as a binary image. For example, the pixel values ​​in a binary mask are only 0 and 1.

[0162] Optionally, a segmentation threshold is obtained; the segmentation probability map is binarized according to the segmentation threshold to obtain a binary mask. For example, if the segmentation threshold is set to 80%, then when the probability of a pixel is less than 80%, the value of the corresponding pixel is set to 0; when the probability of a pixel is not less than 80%, the value of the corresponding pixel is set to 1.

[0163] Step 410: Determine the irregular contour of the region of interest based on the intersection of the first and second values ​​in the binary mask.

[0164] A binary mask has only two possible values: a first value and a second value. Since the irregular contour of the region of interest is continuous, the pixels on the first side of the irregular contour are assigned the first value, and the pixels on the second side are assigned the second value. Similarly, at the boundary between the first and second values, the pixels on the first side are assigned the first value, and the pixels on the second side are assigned the second value.

[0165] Optionally, the region composed of the first value in the binary mask can be determined as the region of interest, or the region composed of the second value in the binary mask can be determined as the region of interest.

[0166] In summary, this embodiment dynamically determines the bounding box surrounding the region of interest (ROI) in the first stage based on the edge key points of the medical image; in the second stage, it obtains a more accurate irregular contour of the ROI based on semantic and detail features. On the one hand, the bounding box determines the approximate range of the ROI, and the second stage calculation is performed only on the image region within the bounding box, thus simplifying the computational load of subsequent calculations. On the other hand, by incorporating detail features and using a fusion of semantic and detail features, a more accurate irregular contour of the ROI can be obtained. Furthermore, when training the image processing model, only the annotations of the edge key points are needed as supervision information; specific irregular contour annotations are not required. Moreover, edge key point annotations are easier to obtain than specific irregular contour annotations, reducing the cost of manual annotation.

[0167] On the other hand, since the fused features include the detailed features from the original medical image, and these detailed features are processed at the original image resolution, the details are richer, resulting in more accurate segmentation results and better performance.

[0168] In the following embodiments, on the one hand, the entire image processing model can remove the branches of the residual network and directly identify the irregular contours of the region of interest through the semantic features in the localization box, resulting in faster recognition speed; on the other hand, the hourglass backbone network is used to extract semantic features from medical images, achieving good extraction results and high feature accuracy. When the segmentation network only includes segmentation network layers, please refer to the structural diagram of the image processing model. Figure 5 .

[0169] exist Figure 5 In the image processing model, there are an hourglass backbone network 102, a prediction network 104, and a segmentation network 110.

[0170] The hourglass backbone network 102 is used to extract semantic features from medical images and generate semantic feature maps. The input of the hourglass backbone network 102 is the medical image 101, and the output is the semantic feature map 103.

[0171] Prediction network 104 extracts edge keypoints from the semantic feature map and generates bounding boxes based on these keypoints. The region within each bounding box is the area of ​​the bounding rectangle in the semantic feature map that surrounds the edge keypoints. The input to prediction network 104 is the semantic feature map 103, and the output is the edge keypoints 105. Furthermore, bounding boxes are generated based on the edge keypoints, and these bounding boxes are combined with the semantic feature map 103 to achieve the first enclosing region 114 in the semantic feature map. The purpose of the bounding boxes is to limit the computational range of segmentation network 110, thereby reducing the computational load of segmentation network 110.

[0172] The segmentation network 110 is used to identify the region of interest from the input semantic features. The input to the segmentation network 110 is the semantic features 106, and the output is the region of interest 111. Further, the region of interest 111 is the part within the white box in the medical image 113.

[0173] Figure 6 This illustration shows a flowchart of a method for determining a region of interest in a medical image according to an exemplary embodiment of this application. The method comprises... Figure 2 The computer system 200 shown executes a method comprising the following steps:

[0174] Step 601: Call the hourglass backbone network to downsample and upsample the medical image to restore it, and output the semantic feature map of the medical image.

[0175] The hourglass backbone network package includes at least one of a downsampling layer and an upsampling restoration layer.

[0176] Optionally, the medical image is downsampled through a downsampling layer in the hourglass backbone network to obtain a downsampled medical image.

[0177] Optionally, the downsampled medical image can be upsampled and restored using an upsampling layer in the hourglass backbone network to obtain the semantic feature map of the medical image.

[0178] For example, for a medical image of size M*N, it is downsampled by a factor of s to obtain a medical image with a resolution of (M / s)*(N / s). The image within the original medical image window s*s becomes a single pixel, and the value of this pixel is the mean or maximum value of all pixels within the window s*s.

[0179] Optionally, the upsampling restoration method includes at least one of the nearest neighbor method, bilinear interpolation method, and cubic interpolation method.

[0180] Step 602: Call the prediction network to identify n edge key points from the semantic feature map of the medical image.

[0181] The prediction network is used to extract edge key points from the semantic feature map.

[0182] Optionally, a predicted offset map is obtained; the predicted offsets corresponding to the two major axis keypoints and the two minor axis keypoints are determined in the predicted offset map; and the two major axis keypoints and the two minor axis keypoints are corrected using the predicted offsets.

[0183] Optionally, the region of interest is the lesion area of ​​the tumor, and the marginal key points are RECIST endpoints.

[0184] Step 603: Determine the bounding rectangle of the n edge key points as the location box of the region of interest.

[0185] For example, when the n edge keypoints are two major axis keypoints and two minor axis keypoints, the bounding rectangle of the quadrilateral formed by the two major axis keypoints and the two minor axis keypoints is determined as the location box of the region of interest.

[0186] Optionally, the upper boundary of the bounding rectangle is determined based on the uppermost edge keypoint among the n edge keypoints; the lower boundary of the bounding rectangle is determined based on the lowermost edge keypoint among the n edge keypoints; the left boundary of the bounding rectangle is determined based on the leftmost edge keypoint among the n edge keypoints; the right boundary of the bounding rectangle is determined based on the rightmost edge keypoint among the n edge keypoints; and the rectangle formed by the upper boundary, lower boundary, left boundary, and right boundary is determined as the bounding box of the region of interest.

[0187] Step 604: Select the first enclosing region from the semantic feature map using a bounding box to obtain the semantic features within the bounding box.

[0188] For example, Figure 5 As shown, the first enclosing region 114 is obtained by determining the location box in the semantic feature map 106.

[0189] Step 605: Call the segmentation network layer to convolve and normalize the semantic features in the region within the localization box to obtain a segmentation probability map.

[0190] The role of convolution is to classify each pixel within the bounding box and obtain a classification score. Normalization maps the classification scores of the pixels to the (0, 1) interval, and the result of the mapping is the probability that the pixel belongs to the region of interest.

[0191] Alternatively, normalization can be performed using the softmax function (normalization exponential function) or the sigmoid function (often used as the activation function of neural networks, mapping variables to the (0, 1) interval).

[0192] Step 606: Binarize the segmentation probability map to obtain a binary mask.

[0193] A binary mask refers to an image where each pixel has exactly two possible values; therefore, in this embodiment, it can also be referred to as a binary image. For example, the pixel values ​​in a binary mask are only 0 and 1.

[0194] Optionally, a segmentation threshold is obtained; the segmentation probability map is binarized according to the segmentation threshold to obtain a binary mask. For example, if the segmentation threshold is set to 80%, then when the probability of a pixel is less than 80%, the value of the corresponding pixel is set to 0; when the probability of a pixel is not less than 80%, the value of the corresponding pixel is set to 1.

[0195] Step 607: Determine the irregular contour of the region of interest based on the intersection of the first and second values ​​in the binary mask.

[0196] A binary mask has only two possible values: a first value and a second value. Since the irregular contour of the region of interest is continuous, the pixels on the first side of the irregular contour are assigned the first value, and the pixels on the second side are assigned the second value. Similarly, at the boundary between the first and second values, the pixels on the first side are assigned the first value, and the pixels on the second side are assigned the second value.

[0197] Optionally, the region composed of the first value in the binary mask can be determined as the region of interest, or the region composed of the second value in the binary mask can be determined as the region of interest.

[0198] In summary, this embodiment dynamically determines the bounding box surrounding the region of interest (ROI) in the first stage based on the edge key points of the medical image; in the second stage, it obtains a more accurate irregular contour of the ROI based on semantic and detail features. On the one hand, the bounding box determines the approximate range of the ROI, and the second stage calculation is performed only on the image region within the bounding box, thus simplifying the computational load of subsequent calculations. On the other hand, by incorporating detail features and using a fusion of semantic and detail features, a more accurate irregular contour of the ROI can be obtained. Furthermore, when training the image processing model, only the annotations of the edge key points are needed as supervision information; specific irregular contour annotations are not required. Moreover, edge key point annotations are easier to obtain than specific irregular contour annotations, reducing the cost of manual annotation.

[0199] On the other hand, this embodiment adds a segmentation network to the detection network, and the structure of the segmentation network can be flexibly changed according to actual needs to obtain different regions of interest. The obtained regions of interest are relatively accurate and the effect is good.

[0200] Figure 7 This illustration shows a schematic diagram of the training method for an image processing model provided in an exemplary embodiment of this application. The method consists of... Figure 2 The computer system 200 shown executes a method comprising the following steps:

[0201] Step 702: Obtain the sample training set.

[0202] The training set includes sample medical images and ground truth annotations. Ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1.

[0203] Optionally, the image archiving and communication system can be invoked to obtain the sample training set.

[0204] Optionally, when the medical image is a CT image, multiple sets of continuous axial slices of the CT image are taken as a set of multi-channel images, and this set of multi-channel images is used as the medical image input.

[0205] Optionally, the region of interest is the lesion area of ​​the tumor, and the marginal key points are RECIST endpoints.

[0206] Step 704: Call the hourglass backbone network and prediction network to extract n predicted edge key points of the determined sample medical image.

[0207] Optionally, the sample medical image is downsampled through the downsampling layer in the hourglass backbone network to obtain the downsampled sample medical image; the downsampled sample medical image is then upsampled and restored through the upsampling layer in the hourglass backbone network to obtain the predicted semantic feature map of the sample medical image.

[0208] Optionally, the n predicted keypoints include two predicted major axis keypoints and two predicted minor axis keypoints. Optionally, a prediction offset map is obtained; the prediction offsets corresponding to the two major axis keypoints and the two minor axis keypoints are determined in the prediction offset map; and the two major axis keypoints and the two minor axis keypoints are corrected using the prediction offsets.

[0209] Step 706: Determine the predicted localization box of the region of interest based on n predicted edge key points.

[0210] n predicted edge key points are used to locate the predicted region of interest.

[0211] Optionally, the predicted location box includes, but is not limited to, at least one of rectangle, triangle, circle, and pentagon.

[0212] Optionally, the predicted bounding box is the smallest bounding rectangle of n edge keypoints.

[0213] Step 708: Obtain the predicted semantic features and predicted detail features of the sample medical image within the predicted localization box.

[0214] Predicted semantic features are used to represent the semantics in a sample medical image. For example, a predicted semantic feature indicates that region A in a sample medical image is a lesion region.

[0215] Predicting detailed features includes, but is not limited to, at least one of the edges, corners, and textures in the sample medical image.

[0216] Step 710: Invoke the segmentation network to identify the predicted irregular contours of the region of interest based on the predicted semantic features and predicted detail features.

[0217] Optionally, a segmentation network is invoked to identify the predicted irregular contours of the region of interest. Specifically, the method includes the following sub-steps:

[0218] 1. By fusing predicted semantic features and predicted detail features, the predicted fusion features of the sample medical image are obtained.

[0219] Optionally, semantic features and detail features can be combined using an adder to obtain fused features of the medical image.

[0220] Among them, the fusion feature is used to determine the general irregular contour of the region of interest, and the detail feature is used to determine the edge of the irregular contour of the region of interest.

[0221] 2. Call the segmentation network to identify the predicted irregular contours of the region of interest from the predicted fusion features.

[0222] Segmentation networks are used to extract regions of interest from fused features.

[0223] In medical images, regions of interest (ROIs) often exhibit irregular shapes. For example, when the ROI is the lesion area of ​​a tumor, the lesion area will appear irregular in shape.

[0224] Step 712: Based on the loss difference between the n predicted edge key points and the actual annotations, perform weakly supervised training on the hourglass backbone network and the segmentation network.

[0225] Optionally, this step includes the following sub-steps:

[0226] 1. Determine two predicted major axis key points and two predicted minor axis key points based on n predicted edge key points.

[0227] The two predicted major axis keypoints are the two endpoints of the longest first line segment in the region of interest, and the two predicted minor axis keypoints are the two endpoints of the longest second line segment in the region of interest that is perpendicular to the first line segment.

[0228] For example, Figure 8 As shown in the CT image of the tumor, the long axis 801 and the short axis 802 are determined based on two predicted long axis key points and two predicted short axis key points in medical image 81.

[0229] 2. The quadrilateral region formed by the two predicted major axis key points and the two predicted minor axis key points is defined as the foreground.

[0230] For example, Figure 8 As shown in the schematic diagram 82, a quadrilateral region is obtained by connecting the two endpoints of the major axis 801 and the two endpoints of the minor axis 802 end to end. To clearly represent this quadrilateral region, please refer to the schematic diagram 83, which clearly shows the region represented by the foreground 803.

[0231] 3. Define the region outside the smallest circumcircle of the foreground as the background.

[0232] For example, Figure 8 As shown in the schematic diagram 84, the smallest circumcircle of the foreground is drawn, and the outer region of the circumcircle is defined as the background 804.

[0233] 4. Calculate the cross-entropy loss of the foreground and background to obtain the partial cross-entropy loss.

[0234] Let the output probability graph be denoted as Foreground and background are recorded separately. and Let the partial cross-entropy loss be denoted as Then we have:

[0235]

[0236] in, This indicates the number of pixels in the foreground. N represents the number of pixels in the background. and The sum of , i is less than an integer, or i is less than Positive integers.

[0237] 5. Weakly supervised training of the hourglass backbone network and the segmentation network based on partial cross-entropy loss.

[0238] Based on partial cross-entropy loss, the hourglass backbone network and segmentation network are trained under weak supervision using the error backpropagation algorithm.

[0239] Alternatively, another weakly supervised training method is provided, with the following specific steps:

[0240] 1. Generate a predicted segmentation probability map based on the predicted semantic feature map.

[0241] The pixel values ​​in the predicted segmentation probability map represent the probability that the corresponding pixel belongs to the predicted region of interest. Optionally, the predicted segmentation probability map is generated based on the predicted semantic feature map.

[0242] 2. Gradient enhancement is performed on the predicted segmentation probability map and the sample medical image to obtain the first gradient intensity map of the predicted segmentation probability map and the second gradient intensity map of the sample medical image, respectively.

[0243] Optionally, the Previtt operator is used to perform gradient enhancement on the predicted segmentation probability map to obtain a first gradient intensity map of the predicted segmentation probability map; the Previtt operator is used to perform gradient enhancement on the sample medical image to obtain a second gradient intensity map of the sample medical image.

[0244] For example, such as Figure 9 As shown, the sample medical image 901 and the predicted segmentation probability map 902 are images after gradient enhancement. Gradient enhancement is performed on the sample medical image 901 and the predicted segmentation probability map 902 to obtain the first gradient intensity map 904 and the second gradient intensity map 903. From the visualized first gradient intensity map 904 and second gradient intensity map 903, it can be seen that the edges of the predicted segmentation result are consistent with the image edges.

[0245] 3. The edge loss is calculated based on the sum of the products of each element in the first and second gradient intensity maps.

[0246] Optionally, the edge loss is denoted as Then we have:

[0247]

[0248] Where N represents the number of elements in the predicted segmentation probability map. The expression indicates element-wise multiplication, G(I) represents the second gradient intensity map, and G(S) represents the first gradient intensity map.

[0249] 4. Weakly supervised training of the hourglass backbone network and the segmentation network based on edge loss.

[0250] Based on edge loss, the hourglass backbone network and segmentation network are trained under weak supervision using the error backpropagation algorithm.

[0251] In summary, this embodiment provides a training method for an image processing model. During the training process, only the annotation of edge key points is needed as supervision information to train the image processing model. Specific irregular contour annotations are not required. Moreover, compared with specific irregular contour annotations, the annotation of edge key points is easier to obtain, which can reduce the cost of manual annotation.

[0252] Figure 10 This application provides a schematic diagram illustrating the training method of an image processing model according to an exemplary embodiment. The method comprises... Figure 2 The computer system 200 shown executes a method comprising the following steps:

[0253] Step 1001: Obtain the sample training set.

[0254] The training set includes sample medical images and ground truth annotations. Ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1.

[0255] Optionally, the image archiving and communication system can be invoked to obtain the sample training set.

[0256] Optionally, when the medical image is a CT image, multiple sets of continuous axial slices of the CT image are taken as a set of multi-channel images, and this set of multi-channel images is used as the medical image input.

[0257] Optionally, the region of interest is the lesion area of ​​the tumor, and the marginal key points are RECIST endpoints.

[0258] Step 1002: Call the hourglass backbone network and prediction network to extract n predicted edge key points of the determined sample medical image.

[0259] Optionally, the sample medical image is downsampled through the downsampling layer in the hourglass backbone network to obtain the downsampled sample medical image; the downsampled sample medical image is then upsampled and restored through the upsampling layer in the hourglass backbone network to obtain the predicted semantic feature map of the sample medical image.

[0260] Optionally, the n predicted keypoints include two predicted major axis keypoints and two predicted minor axis keypoints. Optionally, a prediction offset map is obtained; the prediction offsets corresponding to the two major axis keypoints and the two minor axis keypoints are determined in the prediction offset map; and the two major axis keypoints and the two minor axis keypoints are corrected using the prediction offsets.

[0261] Step 1003: Determine the predicted localization box of the region of interest based on n predicted edge key points.

[0262] n predicted edge key points are used to locate the predicted region of interest.

[0263] Optionally, the predicted location box includes, but is not limited to, at least one of rectangle, triangle, circle, and pentagon.

[0264] Optionally, the predicted bounding box is the smallest bounding rectangle of n edge keypoints.

[0265] Optionally, the upper boundary of the predicted bounding rectangle is determined based on the uppermost predicted edge keypoint among the n predicted edge keypoints; the lower boundary of the predicted bounding rectangle is determined based on the lowermost predicted edge keypoint among the n predicted edge keypoints; the left boundary of the predicted bounding rectangle is determined based on the leftmost predicted edge keypoint among the n predicted edge keypoints; and the right boundary of the predicted bounding rectangle is determined based on the rightmost predicted edge keypoint among the n predicted edge keypoints. The rectangle formed by the upper boundary, lower boundary, left boundary, and right boundary is determined as the predicted bounding box of the region of interest.

[0266] Step 1004: Obtain the predicted semantic features and predicted detail features of the sample medical image within the predicted localization box.

[0267] Predicted semantic features are used to represent the semantics in a sample medical image. For example, a predicted semantic feature indicates that region A in a sample medical image is a lesion region.

[0268] Predicting detailed features includes, but is not limited to, at least one of the edges, corners, and textures in the sample medical image.

[0269] Step 1005: Call the segmentation network to perform convolution and normalization on the predicted fusion features to obtain the predicted segmentation probability map.

[0270] The predicted region image is the image in the sample medical image that corresponds to the region within the predicted bounding box.

[0271] Step 1006: Binarize the predicted segmentation probability map to obtain the predicted binary mask.

[0272] Predicted detail features are used to represent image details within the predicted bounding box region.

[0273] Residual network layers are used to extract detailed features from region images. The detailed features extracted during training include, but are not limited to, at least one of edges, corners, and textures.

[0274] Step 1007: Determine the predicted irregular contour of the region of interest based on the boundary between the first and second values ​​in the predicted binary mask.

[0275] The predicted binary mask has only two possible values: a first value and a second value. Since the irregular contour of the predicted region of interest is continuous, the pixels on the first side of the irregular contour are assigned the first value, and the pixels on the second side are assigned the second value. Similarly, at the boundary between the first and second values, the pixels on the first side are assigned the first value, and the pixels on the second side are assigned the second value.

[0276] Optionally, the region composed of the first value in the predicted binary mask is determined as the region of interest, or the region composed of the second value in the predicted binary mask is determined as the region of interest.

[0277] Step 1008: Based on the loss difference between the n predicted edge key points and the actual annotations, perform weakly supervised training on the hourglass backbone network and the segmentation network.

[0278] Optionally, based on the loss difference between n predicted edge keypoints and the actual annotations, the hourglass backbone network and the segmentation network can be weakly supervised and trained using the error backpropagation algorithm.

[0279] In summary, this embodiment provides a training method for an image processing model. During the training process, only the annotation of edge key points is needed as supervision information to train the image processing model. Specific irregular contour annotations are not required. Moreover, compared with specific irregular contour annotations, the annotation of edge key points is easier to obtain, which can reduce the cost of manual annotation.

[0280] Furthermore, since the fused features include the detailed features from the original medical image, and these detailed features are processed at the original image resolution, the details are richer, resulting in more accurate segmentation results and better performance.

[0281] When detecting tumor regions, the method provided in this application can achieve better segmentation results and more accurate edge key points. Table 1 shows a comparison table between existing technical solutions and the method of this application, where MR-CNN refers to MultiRegion-Convolutional Neural Network. MR-CNN* indicates that MR-CNN is a model trained using partial cross-entropy loss and edge loss. ARUNet is a type of U-Net used for image segmentation. HN indicates the use of an hourglass network for detection. GrabCut is an image segmentation algorithm. GrabCut-k is an improved form of GrabCut. Co-Seg (Co-Segmentation) is an image segmentation algorithm.

[0282] Table 1 Comparison of Existing Technical Solutions and the Method of This Application

[0283]

[0284] As shown in Table 1, the method of this application performs well on the segmentation evaluation index, outperforming the other technical solutions in Table 1. Furthermore, this technical solution differs from the label loss generated by pseudo-label transformation using GrabCut, instead employing a weakly supervised segmentation loss constructed from partial labels and edge information.

[0285] On the other hand, using the KITS19 dataset (a dataset for segmenting and annotating kidney tumors), with the region of interest from medical images as input and the segmentation network fixed as U-Net, the segmentation performance of different segmentation losses is compared, as shown in Table 2:

[0286] Table 2 Comparison of segmentation results under different technical solutions

[0287]

[0288]

[0289] As shown in Table 2, the proposed solution outperforms the methods in Table 2 in both the Dice coefficient (used to calculate the similarity between two samples) and the Hausdorff distance (a measure of the distance between proper subsets in the space, which can be applied to edge matching algorithms), thus verifying the effectiveness of the loss function used in this application.

[0290] Optionally, embodiments of this application are combined with blockchain technology. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0291] Figure 11 A schematic diagram of an apparatus for determining a region of interest in a medical image according to an exemplary embodiment of this application is shown. The system can be implemented as all or part of a computer device through software, hardware, or a combination of both. The apparatus 1100 includes:

[0292] The determination module 1101 is used to determine n edge key points of the medical image, wherein the n edge key points are used to locate the region edge of the region of interest, and n is an integer greater than 1;

[0293] The determining module 1101 is further configured to determine the location box of the region of interest based on the n edge key points;

[0294] Extraction module 1102 is used to obtain the semantic features and detail features of the medical image within the positioning box;

[0295] The recognition module 1103 is used to identify the irregular contour of the region of interest based on the semantic features and the detailed features.

[0296] In an optional design of this application, the recognition module 1103 is further configured to fuse the semantic features and the detail features to obtain fused features; and invoke a segmentation network to identify the irregular contour of the region of interest from the fused features.

[0297] In an optional design of this application, the recognition module 1103 is further configured to invoke the segmentation network, perform convolution and normalization on the fused features to obtain a segmentation probability map, wherein the pixel values ​​of the segmentation probability map are used to represent the probability that the corresponding pixel belongs to the region of interest; binarize the segmentation probability map to obtain a binary mask; and determine the irregular contour of the region of interest based on the intersection of the first value and the second value in the binary mask.

[0298] In an optional design of this application, the extraction module 1102 is further configured to: select a first enclosing region from the semantic feature map of the medical image using the positioning box to obtain the semantic features in the positioning box; select a second enclosing region from the medical image using the positioning box to obtain a region image; and input the region image into a residual network to obtain the detail features in the positioning box.

[0299] In an optional design of this application, the segmentation network and / or the residual network are obtained by weakly supervised training on the foreground and background of the region of interest; the foreground is a quadrilateral region composed of two major axis keypoints and two minor axis keypoints; the background is the outer region of the minimum circumcircle of the foreground, the two major axis keypoints are the two endpoints of the longest first line segment in the region of interest, and the two minor axis keypoints are the two endpoints of the longest second line segment in the region of interest that is perpendicular to the first line segment.

[0300] In an optional design of this application, the determining module 1101 is further configured to invoke the hourglass backbone network to downsample and upsample the medical image to restore it, and output the semantic feature map of the medical image; and invoke the prediction network to identify the n edge key points from the semantic feature map of the medical image.

[0301] In an optional design of this application, the extraction module 1102 is further configured to downsample the medical image through a downsampling layer in the hourglass backbone network to obtain a downsampled medical image, wherein the downsampling layer includes cascaded a-layer network layers, where a is an integer greater than 1; and to upsample and restore the downsampled medical image through an upsampling layer in the hourglass backbone network to obtain the semantic feature map of the medical image, wherein the upsampling layer includes cascaded a-layer network layers corresponding to the downsampling layer; wherein the input of the i-th layer in the upsampling layer is the upsampling and restoration result of the (i-1)-th layer in the upsampling layer and the downsampling result of the corresponding i-th layer in the downsampling layer, where i is an integer greater than 1 and less than a+1.

[0302] In an optional design of this application, the determining module 1101 is further configured to determine the bounding rectangle of the n edge key points as the positioning box of the region of interest.

[0303] In an optional design of this application, the determining module 1101 is further configured to determine the upper boundary of the circumscribed rectangle based on the uppermost edge key point among the n edge key points; determine the lower boundary of the circumscribed rectangle based on the lowermost edge key point among the n edge key points; determine the left boundary of the circumscribed rectangle based on the leftmost edge key point among the n edge key points; determine the right boundary of the circumscribed rectangle based on the rightmost edge key point among the n edge key points; and determine the rectangle formed by the upper boundary, the lower boundary, the left boundary, and the right boundary as the positioning box of the region of interest.

[0304] In summary, this embodiment dynamically determines the bounding box surrounding the region of interest (ROI) in the first stage based on the edge key points of the medical image; in the second stage, it obtains a more accurate irregular contour of the ROI based on semantic and detail features. On the one hand, the bounding box determines the approximate range of the ROI, and the second stage calculation is performed only on the image region within the bounding box, thus simplifying the computational load of subsequent calculations. On the other hand, by incorporating detail features and using a fusion of semantic and detail features, a more accurate irregular contour of the ROI can be obtained. Furthermore, when training the image processing model, only the annotations of the edge key points are needed as supervision information; specific irregular contour annotations are not required. Moreover, edge key point annotations are easier to obtain than specific irregular contour annotations, reducing the cost of manual annotation.

[0305] Figure 12A schematic diagram of a training apparatus for an image processing model provided in an exemplary embodiment of this application is shown. The system can be implemented as all or part of a computer device through software, hardware, or a combination of both. The apparatus 1200 includes:

[0306] The acquisition module 1201 is used to acquire a sample training set, which includes sample medical images and ground truth annotations. The ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1.

[0307] The determination module 1202 is used to call the hourglass backbone network and the prediction network to extract and determine n predicted edge key points of the sample medical image. The n predicted edge key points are used to locate the edge of the region of interest, where n is an integer greater than 1.

[0308] The determining module 1202 is further configured to determine the predicted location box of the region of interest based on the n predicted edge key points;

[0309] Extraction module 1203 is used to obtain the predicted semantic features and predicted detail features of the sample medical image in the predicted localization box;

[0310] The identification module 1204 is also used to call the segmentation network to identify the predicted irregular contour of the predicted region of interest based on the predicted semantic features and the predicted detail features.

[0311] Training module 1205 is used to perform weakly supervised training on the hourglass backbone network and the segmentation network based on the loss difference between the n predicted key points and the real annotations.

[0312] In an optional design of this application, the recognition module 1204 is further configured to fuse the predicted semantic features and the predicted detail features to obtain predicted fusion features; and invoke the segmentation network to identify the predicted irregular contour of the predicted region of interest from the predicted fusion features.

[0313] In an optional design of this application, the recognition module 1204 is further configured to invoke the segmentation network, perform convolution and normalization on the predicted fusion features to obtain a predicted segmentation probability map, wherein the pixel values ​​of the predicted segmentation probability map are used to represent the probability that the corresponding pixel belongs to the predicted region of interest; binarize the predicted segmentation probability map to obtain a predicted binary mask; and determine the predicted irregular contour of the predicted region of interest based on the intersection of the first value and the second value in the predicted binary mask.

[0314] In an optional design of this application, the training module 1205 is further configured to determine two predicted major axis keypoints and two predicted minor axis keypoints based on the n predicted edge keypoints, wherein the two predicted major axis keypoints are the two endpoints of the longest first line segment in the predicted region of interest, and the two predicted minor axis keypoints are the two endpoints of the longest second line segment in the predicted region of interest that is perpendicular to the first line segment; define the quadrilateral region formed by the two predicted major axis keypoints and the two predicted minor axis keypoints as the foreground; define the region outside the smallest circumcircle of the foreground as the background; calculate the cross-entropy loss of the foreground and the background to obtain the partial cross-entropy loss; and perform weakly supervised training on the hourglass backbone network and the segmentation network based on the partial cross-entropy loss.

[0315] In an optional design of this application, the training module 1205 is further configured to generate a predicted segmentation probability map based on the predicted semantic feature map, wherein the pixel values ​​in the predicted segmentation probability map are used to represent the probability that the corresponding pixel belongs to the predicted region of interest; perform gradient enhancement on the predicted segmentation probability map and the sample medical image to obtain a first gradient intensity map of the predicted segmentation probability map and a second gradient intensity map of the sample medical image, respectively; calculate the edge loss based on the sum of the products of each element in the first gradient intensity map and the second gradient intensity map; and perform weakly supervised training on the hourglass backbone network and the segmentation network based on the edge loss.

[0316] In summary, this embodiment provides a training method for an image processing model. During the training of the image processing model, only the ground truth annotations of edge key points are needed as supervision information. The training of the image processing model can be completed without segmentation annotations. Obtaining the ground truth annotations of edge key points is relatively simple and low-cost.

[0317] Figure 13 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 1300 includes a Central Processing Unit (CPU) 1301, a system memory 1304 including Random Access Memory (RAM) 1302 and Read-Only Memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 to facilitate information transfer between various devices within the computer device, and a mass storage device 1307 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0318] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0319] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer device-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0320] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1307 described above can be collectively referred to as memory.

[0321] According to various embodiments of this disclosure, the computer device 1300 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1300 can be connected to the network 1311 via a network interface unit 1312 connected to the system bus 1305, or it can use the network interface unit 1312 to connect to other types of networks or remote computer device systems (not shown).

[0322] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1301 executes the one or more programs to implement all or part of the steps of the above-mentioned face liveness detection method.

[0323] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method for determining a region of interest in a medical image, or the method for training an image processing model, provided in the above-described method embodiments.

[0324] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for determining a region of interest in a medical image, or the method for training an image processing model, as provided in the above-described method embodiments.

[0325] Optionally, this application also provides a computer program product containing instructions that, when run on a computer device, cause the computer device to perform the methods for determining regions of interest in medical images, or the methods for training image processing models, as described in the above aspects.

[0326] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0327] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0328] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a region of interest in a medical image, characterized in that, The method includes: Determine n edge key points of the medical image, the n edge key points are used to locate the region edge of the region of interest, and n is an integer greater than 1; The location box of the region of interest is determined based on the n edge key points; The first bounding region is selected from the semantic feature map of the medical image using the bounding box to obtain the semantic features within the bounding box; the second bounding region is selected from the medical image using the bounding box to obtain the region image; the region image is input into the residual network to obtain the detail features within the bounding box. Based on the semantic features and the detailed features, the irregular contours of the region of interest are identified.

2. The method according to claim 1, characterized in that, The step of identifying the irregular contour of the region of interest based on the semantic features and the detailed features includes: By fusing the semantic features and the detail features, the fused features of the medical image are obtained; The segmentation network is invoked to identify the irregular contour of the region of interest from the fused features.

3. The method according to claim 2, characterized in that, The invocation of the segmentation network to identify the irregular contour of the region of interest from the fused features includes: The segmentation network is invoked to perform convolution and normalization on the fused features to obtain a segmentation probability map. The pixel values ​​of the segmentation probability map are used to represent the probability that the corresponding pixel belongs to the region of interest. The segmentation probability map is binarized to obtain a binary mask; The irregular contour of the region of interest is determined based on the boundary between the first and second values ​​in the binary mask.

4. The method according to claim 1, characterized in that, The segmentation network and / or the residual network are obtained by weakly supervised training on the foreground and background of the region of interest. The segmentation network is used to identify the irregular contour of the region of interest from the fused features, which are obtained by fusing the semantic features and the detail features. The foreground is a quadrilateral region consisting of two major axis key points and two minor axis key points; The background is the outer region of the smallest circumcircle of the foreground, the two major axis keypoints are the two endpoints of the longest first line segment in the region of interest, and the two minor axis keypoints are the two endpoints of the longest second line segment in the region of interest that is perpendicular to the first line segment.

5. The method according to any one of claims 1 to 4, characterized in that, Determining the n edge key points of the medical image includes: The hourglass backbone network is invoked to downsample and upsample the medical image to restore it, and the semantic feature map of the medical image is output. The prediction network is invoked to identify the n edge key points from the semantic feature map of the medical image.

6. The method according to claim 5, characterized in that, The step of calling the hourglass backbone network to downsample and upsample the medical image and output the semantic feature map of the medical image includes: The medical image is downsampled by the downsampling layer in the hourglass backbone network to obtain the downsampled medical image. The downsampling layer includes a cascaded network layers, where a is an integer greater than 1. The upsampling layer in the hourglass backbone network is used to restore the downsampled medical image by upsampling to obtain the semantic feature map of the medical image. The upsampling layer includes a cascaded a-layer network layer corresponding to the downsampling layer. The input to the i-th network layer in the upsampling layer is the upsampling restoration result of the (i-1)-th network layer in the upsampling layer and the downsampling result of the corresponding i-th network layer in the downsampling layer, where i is an integer greater than 1 and less than a+1.

7. The method according to any one of claims 1 to 4, characterized in that, The bounding box of the region of interest is the bounding rectangle of the n edge key points; The step of determining the bounding box of the region of interest based on the n edge key points includes: The upper boundary of the circumscribed rectangle is determined based on the uppermost edge key point among the n edge key points; The lower boundary of the circumscribed rectangle is determined based on the lowermost edge key point among the n edge key points; The left boundary of the circumscribed rectangle is determined based on the leftmost edge key point among the n edge key points. The right boundary of the circumscribed rectangle is determined based on the rightmost edge key point among the n edge key points. The rectangle formed by the upper boundary, the lower boundary, the left boundary, and the right boundary is defined as the positioning box of the region of interest.

8. A training method for an image processing model, characterized in that, The image processing model includes an hourglass backbone network, a prediction network, and a segmentation network; the method includes: Obtain a sample training set, which includes sample medical images and ground truth annotations, wherein the ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1; The hourglass backbone network and the prediction network are invoked to extract and determine n predicted edge key points of the sample medical image. The n predicted edge key points are used to locate the edge of the region of interest, where n is an integer greater than 1. The predicted location box of the predicted region of interest is determined based on the n predicted edge key points; The predicted semantic features in the predicted semantic feature map of the sample medical image are obtained by selecting the bounding region from the predicted localization box; the bounding region in the sample medical image is obtained by selecting the bounding region from the predicted localization box; the sample region image is input into the residual network to obtain the predicted detail features in the predicted localization box. The segmentation network is invoked to identify the predicted irregular contours of the predicted region of interest based on the predicted semantic features and the predicted detail features. Based on the loss difference between the n predicted edge keypoints and the actual annotations, the hourglass backbone network and the segmentation network are trained under weak supervision.

9. The method according to claim 8, characterized in that, The loss difference includes a portion of the cross-entropy loss; The step of performing weakly supervised training on the hourglass backbone network and the segmentation network based on the loss difference between the n predicted keypoints and the actual annotations includes: Based on the n predicted edge key points, two predicted major axis key points and two predicted minor axis key points are determined. The two predicted major axis key points are the two endpoints of the longest first line segment in the predicted region of interest, and the two predicted minor axis key points are the two endpoints of the longest second line segment in the predicted region of interest that is perpendicular to the first line segment. The quadrilateral region formed by the two predicted major axis key points and the two predicted minor axis key points is defined as the foreground. The region outside the smallest circumcircle of the foreground is defined as the background; Calculate the cross-entropy loss between the foreground and the background to obtain the partial cross-entropy loss; The hourglass backbone network and the segmentation network are trained under weak supervision based on the partial cross-entropy loss.

10. The method according to claim 8, characterized in that, The loss difference also includes edge loss; The method further includes: A predicted segmentation probability map is generated based on the predicted semantic feature map, and the pixel values ​​in the predicted segmentation probability map are used to represent the probability that the corresponding pixel belongs to the predicted region of interest. Gradient enhancement is performed on the predicted segmentation probability map and the sample medical image to obtain a first gradient intensity map of the predicted segmentation probability map and a second gradient intensity map of the sample medical image, respectively. The edge loss is calculated based on the sum of the products of each element in the first gradient intensity map and the second gradient intensity map. The hourglass backbone network and the segmentation network are trained under weak supervision based on the edge loss.

11. An apparatus for determining a region of interest in a medical image, characterized in that, The device includes: A determination module is used to determine n edge key points of the medical image, wherein the n edge key points are used to locate the region edge of the region of interest, and n is an integer greater than 1; The determining module is further configured to determine the location box of the region of interest based on the n edge key points; An extraction module is used to select a first enclosing region from the semantic feature map of the medical image using the positioning box to obtain the semantic features in the positioning box; to select a second enclosing region from the medical image using the positioning box to obtain a region image; and to input the region image into a residual network to obtain the detail features in the positioning box. The recognition module is used to identify the irregular contour of the region of interest based on the semantic features and the detailed features.

12. A training device for an image processing model, characterized in that, The image processing model includes an hourglass backbone network, a prediction network, and a segmentation network; the device includes: The acquisition module is used to acquire a sample training set, which includes sample medical images and ground truth annotations. The ground truth annotations refer to n ground truth edge key points of the sample medical images, where n is an integer greater than 1. The determination module is used to call the hourglass backbone network and the prediction network to extract and determine n predicted edge key points of the sample medical image. The n predicted edge key points are used to locate the edge of the region of interest, where n is an integer greater than 1. The determination module is also used to determine the predicted localization box of the predicted region of interest based on the n predicted edge key points; An extraction module is used to select an enclosing region from the predicted semantic feature map of the sample medical image using the predicted localization box to obtain the predicted semantic features in the predicted localization box; to select an enclosing region from the sample medical image using the predicted localization box to obtain a sample region image; and to input the sample region image into a residual network to obtain the predicted detail features in the predicted localization box. The identification module is also used to call the segmentation network to identify the predicted irregular contour of the predicted region of interest based on the predicted semantic features and the predicted detail features; The training module is used to perform weakly supervised training on the hourglass backbone network and the segmentation network based on the loss difference between the n predicted edge key points and the real annotations.

13. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for determining a region of interest in a medical image as claimed in any one of claims 1 to 7, or the method for training an image processing model as claimed in any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the method for determining a region of interest in a medical image as described in any one of claims 1 to 7, or the method for training an image processing model as described in any one of claims 8 to 10.

Images