Image labeling method and device, and electronic device

Abstract

This invention discloses an image annotation method, apparatus, and electronic device. The method includes: acquiring a target ultrasound image to be annotated; responding to an annotation operation on the boundary of the target ultrasound image and displaying boundary annotation information; determining the target image region based on the boundary annotation information; extracting regional feature information of the target image region and extracting global feature information of the target ultrasound image; and outputting lesion annotation information based on the regional and global feature information, wherein the lesion annotation information is used to distinguish lesion regions and non-lesion regions in the target ultrasound image. This invention solves the technical problem in related technologies where lesion delineation requires doctors to spend a significant amount of time annotating, resulting in a cumbersome and time-consuming process.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image annotation method, apparatus, and electronic device. Background Technology

[0002] Among related technologies, with the rapid development of machine learning, it has begun to be applied in various industries, especially in the field of medical imaging, where it has received widespread attention. Deep learning, as a technology for implementing machine learning, with its multiple hidden layers, strong fitting power, and automatic learning and feature extraction capabilities, has gradually become the mainstream application algorithm in many fields due to its significantly superior performance compared to traditional machine learning algorithms. Currently, deep learning algorithms applied to medical imaging mainly use supervised learning to enable the model to learn how to make correct predictions and judgments on image data in specific tasks. Therefore, the quantity and quality of data annotation are crucial factors determining the performance of deep learning algorithms.

[0003] In the field of medical imaging, lesion delineation (lesion segmentation) is a key preprocessing method for automatically measuring and diagnosing lesions using machine learning. It is currently widely used in various medical tasks. However, the current lesion delineation method has obvious drawbacks: manual annotation requires experienced doctors to judge and record the boundaries, which is a tedious and time-consuming process. Doctors find it difficult to take time out of their already heavy workload to complete the annotation work, which poses a great challenge to the data collection work.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides an image annotation method, apparatus, and electronic device to at least solve the technical problem in related technologies that doctors need to spend a lot of time annotating lesions, which is cumbersome and time-consuming.

[0006] According to one aspect of the present invention, an image annotation method is provided, comprising: acquiring a target ultrasound image to be annotated; displaying boundary annotation information in response to an annotation operation on the boundary of the target ultrasound image; determining a target image region based on the boundary annotation information; extracting regional feature information of the target image region and extracting global feature information of the target ultrasound image; and outputting lesion annotation information based on the regional feature information and the global feature information, wherein the lesion annotation information is used to distinguish lesion regions and non-lesion regions in the target ultrasound image.

[0007] Optionally, the step of responding to the annotation operation on the boundary of the target ultrasound image and displaying boundary annotation information includes: receiving multiple point selection operation instructions, wherein the point selection operation instructions are used to indicate information for single-point annotation of the lesion boundary in the target ultrasound image; determining multiple boundary annotation points of the lesion region in the target ultrasound image based on the multiple point selection operation instructions, wherein the multiple boundary annotation points include: lesion boundary vertex, lesion boundary bottom point, lesion boundary left point, and lesion boundary right point; and determining the boundary annotation information based on the multiple boundary annotation points.

[0008] Optionally, the step of determining the target image region based on the boundary annotation information includes: determining a boundary point information map and a focal point information map based on the plurality of boundary annotation points, wherein the boundary point information map is used to indicate the boundary of the initial lesion region in the target ultrasound image, and the focal point information map is used to indicate the coverage of the initial lesion region; and determining the target image region based on the boundary point information map and the focal point information map.

[0009] Optionally, the step of determining the boundary point information map and the focus information map based on the plurality of boundary annotation points includes: constructing a plurality of boundary information sub-maps centered on each of the boundary annotation points; forming the boundary point information map based on the plurality of boundary information sub-maps; determining a first annotation line based on the vertex and bottom point of the lesion boundary; determining a second annotation line based on the left and right points of the lesion boundary; determining the lesion focus based on the first annotation line and the second annotation line; and constructing the focus information map centered on the lesion focus.

[0010] Optionally, the step of forming the boundary point information map based on the plurality of boundary information sub-maps includes: obtaining the pixel value of each boundary information sub-map at each pixel point; taking the maximum pixel value for each pixel point at the same pixel position of the plurality of boundary information sub-maps to obtain the boundary point information map.

[0011] Optionally, the step of determining the target image region based on the boundary point information map and the focal point information map includes: extracting the abscissa and ordinate with the smallest coordinate value among all boundary marker points based on the boundary point information map; constructing the target image region based on the extracted abscissa and ordinate, wherein the target image region includes: a lesion region.

[0012] Optionally, the step of responding to the boundary annotation operation of the target ultrasound image and displaying boundary annotation information includes: receiving multiple border selection instructions, wherein the border selection instructions are used to indicate information for border selection of lesion regions in the target ultrasound image; determining multiple boundary selection boxes of lesion regions in the target ultrasound image based on the multiple border selection instructions; and determining the boundary annotation information based on the multiple border selection instructions.

[0013] Optionally, the step of outputting lesion annotation information based on the regional feature information and global feature information includes: scaling the target ultrasound image so that the size of the processed target ultrasound image is a preset segmentation size; segmenting the lesion region and non-lesion region in the processed target ultrasound image using a preset classification model to obtain the lesion annotation information, wherein the preset classification model is trained on multiple sets of data using a preset loss function, and the input data in the multiple sets of data includes at least: ultrasound image, boundary information map and focus information map, and the output data in the multiple sets of data includes: lesion delineation area and boundary identification information of lesion region and non-lesion region.

[0014] Optionally, when training the preset classification model, the difference between the lesion segmentation result and the user-annotated result is determined, and then the segmentation weights are adjusted through back gradient propagation. The segmentation weights are used to adjust the boundary values ​​when segmenting lesion regions and non-lesion regions.

[0015] Optionally, the step of outputting lesion annotation information based on the regional feature information and the global feature information further includes: determining the weights in each network channel of the regional feature information and the global feature information; performing weighted fusion on the weights in each network channel to obtain an image feature map; analyzing the regional difference information and regional association information between the lesion area and the non-lesion area based on the image feature map; and outputting lesion annotation information based on the regional difference information and the regional association information.

[0016] Optionally, after outputting lesion annotation information based on the regional feature information and global feature information, the image annotation method further includes: fitting the lesion region to obtain the region's major axis and minor axis; and calculating the ratio between the region's major axis and minor axis to obtain the region's aspect ratio.

[0017] According to another aspect of the present invention, an image annotation apparatus is also provided, comprising: an acquisition unit for acquiring a target ultrasound image to be annotated; a response unit for responding to an annotation operation on the boundary of the target ultrasound image and displaying boundary annotation information; a first determination unit for determining a target image region based on the boundary annotation information; an extraction unit for extracting regional feature information of the target image region and extracting global feature information of the target ultrasound image; and an output unit for outputting lesion annotation information based on the regional feature information and the global feature information, wherein the lesion annotation information is used to distinguish lesion regions and non-lesion regions in the target ultrasound image.

[0018] Optionally, the response unit includes: a first receiving module, configured to receive multiple point selection operation instructions, wherein the point selection operation instructions are used to indicate information for single-point annotation of the lesion boundary in the target ultrasound image; a first determining module, configured to determine multiple boundary annotation points of the lesion region in the target ultrasound image based on the multiple point selection operation instructions, wherein the multiple boundary annotation points include: lesion boundary vertex, lesion boundary bottom point, lesion boundary left point, and lesion boundary right point; and a second determining module, configured to determine the boundary annotation information based on the multiple boundary annotation points.

[0019] Optionally, the first determining unit includes: a third determining module, used to determine a boundary point information map and a focal point information map based on the plurality of boundary markers, wherein the boundary point information map is used to indicate the boundary of the initial lesion region in the target ultrasound image, and the focal point information map is used to indicate the coverage of the initial lesion region; and a fourth determining module, used to determine the target image region based on the boundary point information map and the focal point information map.

[0020] Optionally, the third determining module includes: a first construction submodule, used to construct multiple boundary information sub-graphs centered on each of the boundary annotation points; a second construction submodule, used to form the boundary point information graph based on the multiple boundary information sub-graphs; a first determining submodule, used to determine a first annotation line based on the vertex and bottom point of the lesion boundary; a second determining submodule, used to determine a second annotation line based on the left and right points of the lesion boundary; a third determining submodule, used to determine the lesion focus based on the first and second annotation lines; and a third construction submodule, used to construct the focus information graph centered on the lesion focus.

[0021] Optionally, the second construction submodule includes: a first acquisition submodule, used to acquire the pixel value of each of the boundary information sub-images at each pixel point; and a fourth determination submodule, used to take the maximum pixel value of each pixel point at the same pixel position of multiple boundary information sub-images to obtain the boundary point information map.

[0022] Optionally, the fourth determining module includes: a first extraction submodule, used to extract the abscissa and ordinate with the smallest coordinate value among all boundary marker points based on the boundary point information map; and a fourth construction submodule, used to construct a target image region based on the extracted abscissa and ordinate, wherein the target image region includes: a lesion region.

[0023] Optionally, the response unit includes: a second receiving module, configured to receive multiple border selection instructions, wherein the border selection instructions are used to indicate information for border selection of the lesion region in the target ultrasound image; a fifth determining module, configured to determine multiple boundary selection boxes of the lesion region in the target ultrasound image based on the multiple border selection instructions; and a sixth determining module, configured to determine the boundary annotation information based on the multiple border selection instructions.

[0024] Optionally, the output unit includes: a first scaling module, used to scale the target ultrasound image so that the size of the processed target ultrasound image is a preset segmentation size; and a first segmentation module, used to segment the lesion region and non-lesion region in the processed target ultrasound image using a preset classification model to obtain the lesion annotation information, wherein the preset classification model is obtained by training multiple sets of data using a preset loss function, and the input data in the multiple sets of data includes at least: ultrasound image, boundary information map and focus information map, and the output data in the multiple sets of data includes: lesion delineation area and boundary identification information of lesion region and non-lesion region.

[0025] Optionally, when training the preset classification model, the difference between the lesion segmentation result and the user-annotated result is determined, and then the segmentation weights are adjusted through back gradient propagation. The segmentation weights are used to adjust the boundary values ​​when segmenting lesion regions and non-lesion regions.

[0026] Optionally, the output unit further includes: a seventh determining module, used to determine the weights in each network channel of the regional feature information and the global feature information; a first fusion module, used to perform weighted fusion of the weights in each network channel to obtain an image feature map; a first analysis module, used to analyze the regional difference information and regional association information between lesion areas and non-lesion areas based on the image feature map; and a first output module, used to output lesion annotation information based on the regional difference information and the regional association information.

[0027] Optionally, the image annotation device further includes: a fitting unit, used to fit the lesion region after outputting lesion annotation information based on the regional feature information and global feature information, to obtain the region's major axis and minor axis; and a calculation unit, used to calculate the ratio between the region's major axis and minor axis to obtain the region's aspect ratio.

[0028] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image annotation method described in any of the preceding embodiments by executing the executable instructions.

[0029] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the image annotation method described in any one of the above embodiments.

[0030] In this embodiment of the invention, when annotating lesions in pathological images, the target ultrasound image to be annotated is first acquired. In response to the boundary annotation operation of the target ultrasound image, boundary annotation information is displayed. Based on the boundary annotation information, the target image region is determined, and regional feature information of the target image region and global feature information of the target ultrasound image are extracted. Then, based on the regional and global feature information, lesion annotation information is output. The lesion annotation information is used to distinguish between lesion regions and non-lesion regions in the target ultrasound image. In this embodiment, only a small amount of boundary annotation information is needed to return accurate lesion delineation (segmentation) results to the graphical interactive interface based on the boundary points and the image's own characteristics, quickly completing the annotation of the ultrasound image. This solves the technical problem in related technologies where lesion delineation requires doctors to spend a significant amount of time annotating, a process that is cumbersome and time-consuming. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of an optional image annotation method according to an embodiment of the present invention;

[0033] Figure 2 This is an optional schematic diagram of lesion delineation according to an embodiment of the present invention;

[0034] Figure 3This is a schematic diagram of an optional image annotation device according to an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] The embodiments of this invention can be applied to various pathological image analysis systems and image analysis devices, and the pathologies targeted include, but are not limited to, breast cancer. To address the difficulty of delineating and marking lesions in the pathological image analysis process, an interactive annotation technology is proposed. After collecting a small amount of manually annotated data in the early stage, the algorithm model trained based on the small amount of annotation and the early data is used to analyze the newly input ultrasound images to obtain accurate lesion annotation results.

[0038] When receiving a labeling task, traditional manual labeling methods require meticulously drawing each point along the lesion boundary, consuming considerable time to complete the labeling of a single image. However, with AI-assisted labeling using the results of this invention, doctors simply need to open the pre-installed pathology image analysis system / software, access the image to be labeled, and select boundary points on the graphical interface. The AI ​​algorithm then performs inference calculations based on the selected boundary points and the ultrasound image itself, quickly returning accurate lesion delineation (segmentation) results to the graphical interface. Clicking "save" completes the labeling of the image. Compared to manual labeling, which requires accurately depicting every labeling point, this invention only requires doctors to select a small number of boundary points, significantly reducing the effort required for labeling. Furthermore, this invention requires far less interactive information than manual labeling, greatly reducing labeling time.

[0039] The present invention will now be described in conjunction with various embodiments.

[0040] Example 1

[0041] According to an embodiment of the present invention, an image annotation method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] Figure 1 This is a flowchart of an optional image annotation method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0043] Step S102: Obtain the target ultrasound image to be labeled;

[0044] Step S104: In response to the boundary annotation operation of the target ultrasound image, display the boundary annotation information;

[0045] Step S106: Determine the target image region based on the boundary annotation information;

[0046] Step S108: Extract the regional feature information of the target image region and extract the global feature information of the target ultrasound image;

[0047] Step S110: Based on regional feature information and global feature information, output lesion annotation information, wherein the lesion annotation information is used to distinguish lesion areas and non-lesion areas in the target ultrasound image.

[0048] Through the above steps, when annotating lesions in pathological images, the target ultrasound image to be annotated is first acquired. The boundary annotation operation of the target ultrasound image is then performed, displaying the boundary annotation information. Based on the boundary annotation information, the target image region is determined, and regional feature information of the target image region and global feature information of the target ultrasound image are extracted. Then, based on the regional and global feature information, lesion annotation information is output. This lesion annotation information is used to distinguish between lesion and non-lesion regions in the target ultrasound image. In this embodiment, only a small amount of boundary annotation information is needed to return accurate lesion delineation (segmentation) results to the graphical interactive interface based on the boundary points and the image's own characteristics, quickly completing the annotation of the ultrasound image. This solves the technical problem in related technologies where lesion delineation requires doctors to spend a significant amount of time annotating, a process that is cumbersome and time-consuming.

[0049] This invention, based on machine learning / artificial intelligence analysis (AI) technology, can learn how to correctly classify ultrasound image features pixel by pixel from a small amount of interactive information input from external terminal devices. In particular, it can extract effective image features for various pathological lesions (e.g., breast cancer) that have different scales, large intra-class differences, and high inter-class similarity, thereby improving the performance of machine learning / artificial intelligence analysis (AI) technology.

[0050] The present invention will now be described in detail with reference to the above-described implementation steps.

[0051] Step S102: Obtain the target ultrasound image to be labeled.

[0052] In this embodiment of the invention, the ultrasound scanned images can be transmitted through an external terminal device (e.g., a computer, iPad, mobile terminal, etc.). The number of ultrasound images can be one or multiple.

[0053] After viewing the ultrasound image through the display interface of an external terminal device, doctors or other staff can interact with / annotate the image boundaries using peripheral devices such as a mouse. This includes random point annotation, doodle annotation, border annotation, and boundary point annotation. Random point annotation refers to marking multiple prominent points on the ultrasound image, for example, using obvious colors (such as red, yellow, green, white, etc.). Doodle annotation can refer to using a specific virtual marker to draw on the boundaries of the ultrasound image. Border annotation can refer to selecting the boundaries of lesions in the ultrasound image, for example, using rectangular or curved borders. Boundary point annotation can refer to marking the boundary extreme points in the ultrasound image. In this embodiment of the invention, boundary point annotation is used for illustration; simply selecting the highest, lowest, leftmost, and rightmost points of the lesion provides all the necessary interactive information. Compared to other interaction methods, boundary points can more accurately indicate the approximate location of the lesion area, making it easier for subsequent AI algorithms to distinguish between lesion and non-lesion areas, achieving better segmentation results.

[0054] Step S104: In response to the annotation operation on the boundary of the target ultrasound image, display the boundary annotation information.

[0055] Optionally, the step of displaying boundary annotation information in response to the annotation operation on the boundary of the target ultrasound image includes: receiving multiple point selection operation instructions, wherein the point selection operation instructions are used to indicate information for single-point annotation of the lesion boundary in the target ultrasound image; determining multiple boundary annotation points of the lesion region in the target ultrasound image based on the multiple point selection operation instructions, wherein the multiple boundary annotation points include: lesion boundary vertex, lesion boundary bottom point, lesion boundary left point, and lesion boundary right point; and determining boundary annotation information based on the multiple boundary annotation points.

[0056] After obtaining the content of the annotation operation, it can be displayed according to the annotation method, such as displaying boundary point annotation information.

[0057] In addition, in this embodiment of the invention, border annotation information can also be displayed.

[0058] Optionally, the step of displaying boundary annotation information in response to the annotation operation of the boundary of the target ultrasound image includes: receiving multiple bounding box selection instructions, wherein the bounding box selection instructions are used to indicate information for selecting a bounding box for the lesion region in the target ultrasound image; determining multiple boundary selection boxes for the lesion region in the target ultrasound image based on the multiple bounding box selection instructions; and determining boundary annotation information based on the multiple bounding box selection instructions.

[0059] Step S106: Determine the target image region based on the boundary annotation information.

[0060] In this embodiment of the invention, the step of determining the target image region based on boundary annotation information includes: determining a boundary point information map and a focal point information map based on multiple boundary annotation points, wherein the boundary point information map is used to indicate the boundary of the initial lesion region in the target ultrasound image, and the focal point information map is used to indicate the coverage of the initial lesion region; and determining the target image region based on the boundary point information map and the focal point information map.

[0061] After obtaining the annotation information, the interactive information needs to be transformed into a form that the AI ​​algorithm can understand, and then the AI ​​algorithm is trained in conjunction with ultrasound images. In this invention, the interactive information, namely the four boundary points, is transformed into boundary point information maps and focus information maps (or intersection point information maps) through two transformation methods, serving as guidance information for the model to perform segmentation.

[0062] Optionally, the steps of determining the boundary point information map and the focus information map based on multiple boundary annotation points include: constructing multiple boundary information sub-maps centered on each boundary annotation point; forming a boundary point information map based on the multiple boundary information sub-maps; determining a first annotation connection line based on the lesion boundary vertex and lesion boundary bottom point; determining a second annotation connection line based on the lesion boundary left point and lesion boundary right point; determining the lesion focus based on the first annotation connection line and the second annotation connection line; and constructing a focus information map centered on the lesion focus.

[0063] Another optional step of forming a boundary point information map based on multiple boundary information sub-maps includes: obtaining the pixel value of each boundary information sub-map at each pixel point; taking the maximum pixel value for each pixel point at the same pixel position in multiple boundary information sub-maps to obtain the boundary point information map.

[0064] The boundary point information map is generated by applying a two-dimensional Gaussian function to all pixels in the entire map, with each boundary point as the center. Each boundary point forms a corresponding Gaussian map. Finally, the boundary point information map is formed by taking the maximum value of each pixel at the same position in the four Gaussian maps.

[0065] In this embodiment of the invention, a first labeled line is determined based on the vertex and bottom point of the lesion boundary, and a second labeled line is determined based on the left and right points of the lesion boundary. The focal point is the intersection of the line connecting the highest and lowest points and the line connecting the leftmost and rightmost points. In most cases, the focal point falls within the lesion and can be used as additional interactive information for the AI ​​algorithm. Each pixel in the focal point information map represents the Euclidean distance from that pixel to the focal point. The boundary point information map informs the AI ​​algorithm of the approximate boundary of the lesion, and the focal point information map informs the AI ​​algorithm of the approximate coverage area of ​​the lesion.

[0066] Figure 2This is an optional lesion delineation diagram according to an embodiment of the present invention, such as... Figure 2 As shown, in the interaction and results display area, by selecting four lesion boundary points: vertex, lowest point, leftmost point, and rightmost point, the focus of the lesion region can be analyzed, obtaining information maps of each boundary point and focus information. Then, the lesion region (i.e., the lesion region is delineated using an AI model) can be further analyzed. Figure 2 (The area indicated by the irregular dashed line).

[0067] The AI ​​algorithm used in this embodiment of the invention employs supervised learning using pre-labeled ultrasound image data. During training, binary cross-entropy loss and Dice loss are combined to determine the difference between the network segmentation result and the labeled result. Then, the network weights are adjusted via backpropagation. Thus, when training ends, the deep convolutional network can correctly distinguish between lesion areas and non-lesion features, completing the segmentation task. Since the network input is a concatenation of ultrasound grayscale images, boundary point information maps, and focal point information maps, the network is adjusted during training through loss function calculation and backpropagation. The network can learn that the area encompassed by the four boundary points roughly corresponds to the lesion area, and the focal point is highly likely to fall within the lesion area, while the probability of a pixel belonging to the lesion area decreases with distance as the focal point moves further away. Therefore, during inference, the network will also utilize this information and the trained weights to perform a better lesion delineation task on unseen ultrasound images.

[0068] During the training process described above, the network structure used for inference is completely consistent with the network structure used for training. The main component of a deep convolutional neural network—the convolutional layer—is a layer with parameters. During the inference process, because the layers with parameters in the network are continuously updated through backpropagation during training, the parameters are adjusted to be optimal for the current task. The trained parameters are loaded into the network to ensure that the network can correctly identify the features of unseen ultrasound images, so as to successfully complete the task of delineating lesion regions.

[0069] Optionally, the step of determining the target image region based on the boundary point information map and the focus information map includes: extracting the abscissa and ordinate with the smallest coordinate value among all boundary marker points based on the boundary point information map; constructing the target image region based on the extracted abscissa and ordinate, wherein the target image region includes: the lesion region.

[0070] Due to the varying scales, large intra-class differences, and high inter-class similarities among different types of pathological lesions, existing AI algorithms struggle to accurately distinguish between lesion areas and non-lesion areas. In this embodiment of the invention, an image region (or region of interest (RoI)) focus module and an image region feature and global feature fusion module are introduced based on the mainstream network structure UNet in the field of medical image segmentation, thus improving the UNet structure. The image region focus module infers a bounding box slightly larger than the lesion area based on the boundary point interaction information provided by experts [taking the minimum sum of the four boundary points and the minimum x and y coordinates]. min y min x max y max With (x min y min ) and (x max y max The rectangular region defined by the two coordinate points is the RoI region, and the feature information corresponding to this region is extracted from the feature map.

[0071] Step S108: Extract the regional feature information of the target image region and extract the global feature information of the target ultrasound image.

[0072] Step S110: Based on regional feature information and global feature information, output lesion annotation information, wherein the lesion annotation information is used to distinguish lesion areas and non-lesion areas in the target ultrasound image.

[0073] Optionally, the step of outputting lesion annotation information based on regional feature information and global feature information includes: scaling the target ultrasound image so that the size of the processed target ultrasound image is a preset segmentation size; using a preset classification model to segment the lesion region and non-lesion region in the processed target ultrasound image to obtain lesion annotation information. The preset classification model is trained on multiple sets of data using a preset loss function. The input data in the multiple sets of data includes at least: ultrasound image, boundary information map and focus information map. The output data in the multiple sets of data includes: lesion delineation area and boundary identification information of lesion region and non-lesion region.

[0074] Regardless of the size of the lesion region, it will be scaled to a specific size (i.e., a preset segmentation size) and learned on the same network branch. Extracting RoI features from the image region and scaling them to a fixed size enables the model to achieve scale invariance, meaning it can make correct segmentation judgments for lesions of different sizes. Since RoI region features focus more on feature extraction within the lesion area, they often lack contextual information, i.e., information on the differences and relationships between lesions and non-lesion areas; while global feature information contains feature information of the entire image, including information on the differences and relationships between lesions and non-lesion areas, but lacks sufficient attention to RoI regions. Therefore, fusing the two types of features maximizes effective information, enabling the classifier to more accurately distinguish between lesions and non-lesion areas.

[0075] In this embodiment of the invention, when training the preset classification model, the difference between the lesion segmentation result and the user annotation result is determined, and then the segmentation weight is adjusted through back gradient propagation. The segmentation weight is used to adjust the boundary value when segmenting the lesion region and the non-lesion region.

[0076] Optionally, the step of outputting lesion annotation information based on regional feature information and global feature information further includes: determining the weights in each network channel of the regional feature information and global feature information; performing weighted fusion on the weights in each network channel to obtain an image feature map; analyzing the regional difference information and regional association information between the lesion area and the non-lesion area based on the image feature map; and outputting lesion annotation information based on the regional difference information and regional association information.

[0077] To address this, this invention proposes an image region feature and global feature fusion module. This module uses a set of parameters that can be learned and adjusted during model training as weights for each channel of image region features and global features. The module performs weighted fusion of each channel information, and the fused feature map is then input into a classifier for discrimination to intelligently determine lesion areas and non-lesion areas.

[0078] Alternatively, after outputting lesion annotation information based on regional and global feature information, the image annotation method may further include: fitting the lesion region to obtain the region's major axis and minor axis; and calculating the ratio between the region's major axis and minor axis to obtain the region's aspect ratio.

[0079] In this embodiment of the invention, ultrasound image lesion identification can be performed through a graphical interactive interface. The graphical interactive interface includes several parts, such as a preview list of images to be annotated, an interactive and result display area, an aspect ratio display area, and a scoring and lesion delineation (segmentation) result saving area. Users can determine boundary points by clicking on peripheral devices such as a mouse. Figure 2(The black boundary dots in the main body) can be deselected by right-clicking. Once the four boundary points are determined, the graphical interface activates the AI ​​algorithm and transmits the current image and the selected boundary points to the AI ​​algorithm. The AI ​​algorithm then performs inference calculations based on the ultrasound image and interactive information to obtain the lesion delineation (segmentation) result, which is returned to the graphical interface for display. Figure 2 (The irregular dashed lines in the main body). The aspect ratio is calculated by approximating the ratio of the major axis to the minor axis of an ellipse based on the lesion delineation (segmentation) results from the AI ​​algorithm. This value is displayed in the aspect ratio display area.

[0080] Optionally, after outlining the lesion area and calculating the aspect ratio, the embodiments of the present invention can perform a quality assessment on the results of the AI ​​algorithm, and save the lesion outlining (segmentation) results, aspect ratio, and score values ​​obtained by the AI ​​algorithm in the scoring and lesion outlining (segmentation) result saving area of ​​the graphical interactive interface.

[0081] Through the above embodiments, lesion area delineation, aspect ratio calculation, result scoring, and result saving can be performed via terminal devices. After loading the ultrasound image for AI-assisted lesion delineation, based on the boundary point information selected by the power supply, the AI ​​model (which infers and calculates the lesion area mask image (lesion area pixels are labeled 1, non-lesion area pixels are labeled 0) based on the input ultrasound image and boundary point information, and performs aspect ratio fitting based on the inference results) infers the lesion area and non-lesion areas, effectively shortening the time for users to delineate lesion areas and improving annotation efficiency.

[0082] The invention will now be described in conjunction with another alternative embodiment.

[0083] Example 2

[0084] The present invention relates to an image annotation device, which includes multiple implementation units, each of which corresponds to a specific implementation step in the above embodiment one.

[0085] Figure 3 This is a schematic diagram of an optional image annotation device according to an embodiment of the present invention, such as... Figure 3 As shown, the image annotation device may include: an acquisition unit 31, a response unit 33, a first determination unit 35, an extraction unit 37, and an output unit 39, wherein...

[0086] Acquisition unit 31 is used to acquire the target ultrasound image to be labeled;

[0087] The response unit 33 is used to respond to the annotation operation on the boundary of the target ultrasound image and display the boundary annotation information;

[0088] The first determining unit 35 is used to determine the target image region based on boundary annotation information;

[0089] Extraction unit 37 is used to extract regional feature information of the target image region and extract global feature information of the target ultrasound image;

[0090] Output unit 39 is used to output lesion annotation information based on regional feature information and global feature information, wherein the lesion annotation information is used to distinguish lesion areas and non-lesion areas in the target ultrasound image.

[0091] The aforementioned image annotation device, when annotating lesions in medical images, first acquires the target ultrasound image to be annotated through the acquisition unit 31, responds to the boundary annotation operation of the target ultrasound image through the response unit 33, displays the boundary annotation information, determines the target image region based on the boundary annotation information through the first determination unit 35, extracts the regional feature information of the target image region and the global feature information of the target ultrasound image through the extraction unit 37, and then outputs the lesion annotation information based on the regional feature information and the global feature information through the output unit 39. The lesion annotation information is used to distinguish between lesion regions and non-lesion regions in the target ultrasound image. In this embodiment, only a small amount of boundary annotation information is needed to return accurate lesion delineation (segmentation) results to the graphical interactive interface based on the boundary points and the image's own characteristics, quickly completing the annotation of the ultrasound image. This solves the technical problem in related technologies where lesion delineation requires doctors to spend a lot of time annotating, resulting in a cumbersome and time-consuming process.

[0092] Optionally, the response unit includes: a first receiving module for receiving multiple point selection operation instructions, wherein the point selection operation instructions are used to indicate information for single-point annotation of the lesion boundary in the target ultrasound image; a first determining module for determining multiple boundary annotation points of the lesion region in the target ultrasound image based on the multiple point selection operation instructions, wherein the multiple boundary annotation points include: lesion boundary vertex, lesion boundary bottom point, lesion boundary left point, and lesion boundary right point; and a second determining module for determining boundary annotation information based on the multiple boundary annotation points.

[0093] Optionally, the first determining unit includes: a third determining module, used to determine a boundary point information map and a focal information map based on multiple boundary markers, wherein the boundary point information map is used to indicate the boundary of the initial lesion region in the target ultrasound image, and the focal information map is used to indicate the coverage of the initial lesion region; and a fourth determining module, used to determine the target image region based on the boundary point information map and the focal information map.

[0094] Optionally, the third determining module includes: a first construction submodule, used to construct multiple boundary information sub-graphs centered on each boundary annotation point; a second construction submodule, used to form a boundary point information graph based on the multiple boundary information sub-graphs; a first determining submodule, used to determine a first annotation connection line based on the lesion boundary vertex and lesion boundary bottom point; a second determining submodule, used to determine a second annotation connection line based on the lesion boundary left point and lesion boundary right point; a third determining submodule, used to determine the lesion focus based on the first and second annotation connection lines; and a third construction submodule, used to construct a focus information graph centered on the lesion focus.

[0095] Optionally, the second construction submodule includes: a first acquisition submodule, used to acquire the pixel value of each boundary information submap at each pixel point; and a fourth determination submodule, used to take the maximum pixel value of each pixel point at the same pixel position of multiple boundary information submaps to obtain a boundary point information map.

[0096] Optionally, the fourth determining module includes: a first extraction submodule, used to extract the abscissa and ordinate with the smallest coordinate values ​​among all boundary marker points based on the boundary point information map; and a fourth construction submodule, used to construct the target image region based on the extracted abscissa and ordinate, wherein the target image region includes: the lesion region.

[0097] Optionally, the response unit includes: a second receiving module for receiving multiple border selection instructions, wherein the border selection instructions are used to indicate information for border selection of the lesion region in the target ultrasound image; a fifth determining module for determining multiple boundary selection boxes of the lesion region in the target ultrasound image based on the multiple border selection instructions; and a sixth determining module for determining boundary annotation information based on the multiple border selection instructions.

[0098] Optionally, the output unit includes: a first scaling module for scaling the target ultrasound image so that the size of the processed target ultrasound image is a preset segmentation size; and a first segmentation module for segmenting the lesion region and non-lesion region in the processed target ultrasound image using a preset classification model to obtain lesion annotation information. The preset classification model is trained on multiple sets of data using a preset loss function. The input data in the multiple sets of data includes at least: ultrasound image, boundary information map, and focus information map. The output data in the multiple sets of data includes: lesion delineation area and boundary identification information of the lesion region and non-lesion region.

[0099] Optionally, when training the preset classification model, the difference between the lesion segmentation result and the user-annotated result is determined, and then the segmentation weights are adjusted through back gradient propagation. The segmentation weights are used to adjust the boundary values ​​when segmenting lesion regions and non-lesion regions.

[0100] Optionally, the output unit further includes: a seventh determining module, used to determine the weights in each network channel of the regional feature information and global feature information; a first fusion module, used to perform weighted fusion of the weights in each network channel to obtain an image feature map; a first analysis module, used to analyze the regional difference information and regional association information between lesion areas and non-lesion areas based on the image feature map; and a first output module, used to output lesion annotation information based on the regional difference information and regional association information.

[0101] Optionally, the image annotation device further includes: a fitting unit, used to fit the lesion region after outputting lesion annotation information based on regional feature information and global feature information, to obtain the region's major axis and minor axis; and a calculation unit, used to calculate the ratio between the region's major axis and minor axis to obtain the region's aspect ratio.

[0102] The image annotation device described above may also include a processor and a memory. The acquisition unit 31, response unit 33, first determination unit 35, extraction unit 37, output unit 39, etc., are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0103] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, lesion annotation information can be output based on regional and global feature information. This lesion annotation information is used to distinguish between lesion and non-lesion regions in the target ultrasound image.

[0104] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0105] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image annotation method of any of the above by executing the executable instructions.

[0106] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the image annotation method of any of the above.

[0107] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: acquiring a target ultrasound image to be annotated; responding to an annotation operation on the boundary of the target ultrasound image and displaying boundary annotation information; determining the target image region based on the boundary annotation information; extracting regional feature information of the target image region and extracting global feature information of the target ultrasound image; and outputting lesion annotation information based on the regional feature information and the global feature information, wherein the lesion annotation information is used to distinguish lesion regions and non-lesion regions in the target ultrasound image.

[0108] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0109] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0114] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image annotation method, characterized in that, include: Acquire the target ultrasound image to be labeled; In response to the annotation operation on the boundary of the target ultrasound image, the boundary annotation information is displayed; Based on the boundary annotation information, the target image region is determined; Extract the regional feature information of the target image region and extract the global feature information of the target ultrasound image; Based on the regional feature information and global feature information, lesion annotation information is output, wherein the lesion annotation information is used to distinguish between lesion areas and non-lesion areas in the target ultrasound image.

2. The method according to claim 1, characterized in that, The step of responding to the boundary annotation operation of the target ultrasound image and displaying the boundary annotation information includes: Receive multiple point selection operation instructions, wherein the point selection operation instructions are used to indicate information for single-point annotation of the lesion boundary in the target ultrasound image; Based on the multiple point selection operation commands, multiple boundary marker points of the lesion region in the target ultrasound image are determined, wherein the multiple boundary marker points include: lesion boundary vertex, lesion boundary bottom point, lesion boundary left point, and lesion boundary right point; The boundary labeling information is determined based on the multiple boundary labeling points.

3. The method according to claim 2, characterized in that, The step of determining the target image region based on the boundary annotation information includes: Based on the multiple boundary markers, a boundary point information map and a focal point information map are determined. The boundary point information map is used to indicate the boundary of the initial lesion region in the target ultrasound image. The boundary point information map is calculated by applying a two-dimensional Gaussian function to all pixels in the entire image with each boundary point as the center, and each boundary point forms a corresponding Gaussian map. The focal point information map is used to indicate the coverage of the initial lesion region. Each pixel in the focal point information map represents the Euclidean distance from the pixel to the focal point. Based on the boundary point information map and the focus information map, the target image region is determined.

4. The method according to claim 3, characterized in that, The step of determining the boundary point information map and the focal point information map based on the multiple boundary annotation points includes: Multiple boundary information sub-graphs are constructed with each of the aforementioned boundary marker points as the center; Based on the multiple boundary information sub-graphs, the boundary point information graph is formed; Based on the vertex and bottom point of the lesion boundary, the first annotation line is determined; Based on the left point of the lesion boundary and the right point of the lesion boundary, determine the second annotation line; Based on the first and second labeled lines, the focal point of the lesion is determined; The focal information map is constructed with the focal point of the lesion as the center.

5. The method according to claim 4, characterized in that, The step of forming the boundary point information map based on the multiple boundary information sub-graphs includes: Obtain the pixel value of each boundary information sub-image at each pixel point; The boundary point information map is obtained by taking the maximum pixel value for each pixel at the same pixel position in multiple boundary information sub-maps.

6. The method according to claim 3, characterized in that, The step of determining the target image region based on the boundary point information map and the focus information map includes: Based on the boundary point information map, extract the abscissa and ordinate with the smallest coordinate value among all boundary marker points; Based on the extracted horizontal and vertical coordinates, a target image region is constructed, wherein the target image region includes: a lesion region.

7. The method according to claim 1, characterized in that, The step of responding to the boundary annotation operation of the target ultrasound image and displaying the boundary annotation information includes: Receive multiple border selection instructions, wherein the border selection instructions are used to indicate information for border selection of lesion regions in the target ultrasound image; Based on the multiple border selection instructions, multiple boundary selection boxes are determined for the lesion region in the target ultrasound image; The boundary annotation information is determined based on the multiple border selection instructions.

8. The method according to claim 1, characterized in that, The step of outputting lesion annotation information based on the regional feature information and global feature information includes: The target ultrasound image is scaled up so that the size of the processed target ultrasound image is the preset segmentation size. A preset classification model is used to segment the lesion area and non-lesion area in the processed target ultrasound image to obtain the lesion annotation information. The preset classification model is trained on multiple sets of data using a preset loss function. The input data of the multiple sets of data includes at least: ultrasound image, boundary information map and focal information map. The output data of the multiple sets of data includes: lesion delineation area and boundary identification information of lesion area and non-lesion area.

9. The method according to claim 8, characterized in that, When training the preset classification model, the difference between the lesion segmentation result and the user-annotated result is determined, and then the segmentation weights are adjusted through back gradient propagation. The segmentation weights are used to adjust the boundary values ​​when segmenting lesion regions and non-lesion regions.

10. The method according to claim 9, characterized in that, The step of outputting lesion annotation information based on the regional feature information and global feature information further includes: Determine the weights of each network channel in the regional feature information and the global feature information; The weights in each network channel are weighted and fused to obtain the image feature map; Based on image feature maps, analyze the regional differences and regional correlations between lesion areas and non-lesion areas; Based on the regional difference information and the regional association information, lesion annotation information is output.

11. The method according to claim 1, characterized in that, After outputting lesion annotation information based on the regional feature information and global feature information, the image annotation method further includes: The lesion region is fitted to obtain the region's long axis and short axis. The ratio between the major axis and minor axis of the region is calculated to obtain the aspect ratio of the region.

12. An image annotation device, characterized in that, include: The acquisition unit is used to acquire the target ultrasound image to be labeled. A response unit is used to respond to the annotation operation on the boundary of the target ultrasound image and display the boundary annotation information; The first determining unit is used to determine the target image region based on the boundary annotation information; The extraction unit is used to extract regional feature information of the target image region and global feature information of the target ultrasound image; The output unit is used to output lesion annotation information based on the regional feature information and global feature information, wherein the lesion annotation information is used to distinguish between lesion areas and non-lesion areas in the target ultrasound image.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the image annotation method of any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the image annotation method according to any one of claims 1 to 11.

Images