Annotation method and system for medical ultrasonic image

Abstract

The invention discloses a medical ultrasonic image annotation method and system, and the method comprises the steps: obtaining a medical ultrasonic image data set, and carrying out the preprocessing of an image, including image denoising and enhancement operation; for the preprocessed medical ultrasonic image, automatically detecting and positioning a text region in the image by adopting a text detection algorithm based on deep learning, and obtaining the position and size space information of a text; according to the detected text area, the OCR technology is used for text recognition, a method based on character segmentation and a method based on sequence learning are used, text content in the image is accurately extracted, and a preliminary text recognition result is obtained.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and system for annotating medical ultrasound images. Background Art

[0002] In the process of medical ultrasound image annotation, we are faced with the technical challenge of how to accurately identify and locate the text information in the image and match it with the image content. Due to the uneven quality of ultrasound images, the position, size, font, color and other features of the text information in the image vary greatly. Traditional text recognition methods are difficult to accurately extract and locate the text information in the image. At the same time, ultrasound images often contain a large number of professional terms and abbreviations. How to accurately understand and match these professional terms is also a problem that needs to be solved. In addition, the text information in ultrasound images is usually closely related to the image content. How to further analyze the semantic association between text and image on the basis of text recognition to achieve accurate image annotation is also a challenging topic. The solution to these problems requires the comprehensive use of technologies in multiple fields such as computer vision and natural language processing. How to build an efficient and robust text recognition and semantic understanding model in massive medical ultrasound image data to achieve automated and intelligent medical image annotation is a technical problem that needs to be overcome. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for annotating medical ultrasound images.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] A method for annotating a medical ultrasound image, comprising:

[0006] Step S1, obtaining a medical ultrasound image data set;

[0007] Step S2: automatically detecting and locating text areas in the image according to the medical ultrasound image dataset;

[0008] Step S3, obtaining a text recognition result according to the detected text area;

[0009] Step S4, obtaining a normalized text representation according to the text recognition result;

[0010] Step S5: Based on the normalized text representation, learn the joint representation of text and image, and mine the semantic correspondence between the two;

[0011] Step S6: Based on the results of text recognition and semantic understanding, a multimodal annotation model of medical ultrasound images is used to automatically generate text descriptions and keyword tags of the images, thereby achieving automated and intelligent annotation of image content.

[0012] Preferably, a text detection algorithm based on deep learning is used to automatically detect and locate the text area in the image, and obtain spatial information such as the position and size of the text.

[0013] Preferably, in step S3, text recognition is performed using OCR technology based on the detected text area, and a method based on character segmentation and a method based on sequence learning are used to extract text content in the image to obtain a text recognition result.

[0014] Preferably, in step S4, a professional terminology knowledge base and abbreviation dictionary in the medical field are constructed for the text recognized by OCR, and the recognized text is mapped to standard medical terms through dictionary matching and semantic analysis to obtain a standardized text representation.

[0015] The present invention can also provide a medical ultrasound image annotation system, comprising:

[0016] A first processing device, configured to acquire a medical ultrasound image data set;

[0017] a second processing device for automatically detecting and locating a text region in an image based on a medical ultrasound image dataset;

[0018] A third processing device, used to obtain a text recognition result based on the detected text area;

[0019] A fourth processing device, used to obtain a normalized text representation according to the text recognition result;

[0020] A fifth processing device, for learning a joint representation of text and image based on the normalized text representation, and mining a semantic correspondence between the two;

[0021] The sixth processing device is used to automatically generate text descriptions and keyword tags of the image based on the results of text recognition and semantic understanding through a multimodal annotation model of medical ultrasound images, thereby realizing automated and intelligent annotation of image content.

[0022] Preferably, the second processing device adopts a text detection algorithm based on deep learning to automatically detect and locate the text area in the image, and obtain spatial information such as the position and size of the text.

[0023] Preferably, the third processing device uses OCR technology to perform text recognition based on the detected text area, and uses a character segmentation-based method and a sequence learning-based method to extract the text content in the image to obtain a text recognition result.

[0024] Preferably, the fourth processing device constructs a professional terminology knowledge base and abbreviation dictionary in the medical field based on the text recognized by OCR, and maps the recognized text to standard medical terms through dictionary matching and semantic analysis to obtain a standardized text representation.

[0025] The beneficial effects of the present invention are as follows:

[0026] The present invention first preprocesses the ultrasound image to improve the quality, and then uses a deep learning algorithm to detect the text area and perform OCR recognition. Next, the identified text is normalized through a medical terminology knowledge base. Furthermore, the present invention uses multimodal learning technology to analyze the semantic association between text and image, constructs a multimodal annotation model, and automatically generates text descriptions and keyword tags for the image. Finally, the annotation results are continuously optimized through manual review and feedback mechanisms. The present invention solves the problem of automated and intelligent annotation of medical ultrasound images, realizes full-process intelligent processing from image preprocessing to text recognition, semantic understanding, and automatic annotation, significantly improves the efficiency and quality of medical ultrasound image annotation, and provides high-quality data support for subsequent medical image analysis and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention is a flowchart of a method for annotating medical ultrasound images. DETAILED DESCRIPTION

[0028] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0029] Embodiment 1:

[0030] like Figure 1 As shown, an embodiment of the present invention provides a method for annotating a medical ultrasound image, comprising:

[0031] Step S101, obtaining a medical ultrasound image dataset, and preprocessing the image, including operations such as image denoising and enhancement, to improve the image quality and provide high-quality input data for subsequent text recognition.

[0032] Obtain a medical ultrasound image dataset and preprocess the image, including image denoising, enhancement and other operations, to improve the image quality and provide high-quality input data for subsequent text recognition. Use a denoising convolution subnetwork to denoise the medical ultrasound image data to remove the noise generated by the network to ensure the accuracy and clarity of the prediction results. Extract residual mapping features related to the lesion area from the medical ultrasound image data through the feature extraction subnetwork. According to the extracted residual mapping features, use the structure extraction subnetwork to extract the residual mapping of the boundary structure to achieve the goal of lesion area interpretation. If the extracted residual mapping features are not obvious enough, the multi-layer output fusion technology is used to enhance the robustness and generalization of the classification interpretation of lesion areas of different sizes. If the residual mapping features are obvious, directly construct a convolutional neural network based on image segmentation, and use a large number of training samples and their labels to train the network to obtain a network model that can intelligently detect lesion areas in medical ultrasound image data. According to the trained network model, predict the medical ultrasound image data to determine whether there is a lesion area. If the prediction results show the presence of a lesion area, the network model is fine-tuned using the existing medical ultrasound image data set through transfer learning to improve the adaptability and generalization ability of the network model to new medical ultrasound image data. Based on the fine-tuned network model, the new medical ultrasound image data is predicted to obtain the final lesion area detection result, determine the location and size of the lesion area, and provide a reference for subsequent medical diagnosis.

[0033] Specifically, medical ultrasound images are often affected by noise, such as speckle noise, due to the characteristics of their imaging principles. This will lead to a decline in image quality and affect the accuracy of doctors' diagnosis, so the preprocessing step is crucial. Image denoising aims to remove noise from the image while retaining the edge and detail information of the image. Image enhancement is to improve the contrast and clarity of the image, so that the lesion area is easier to identify. For example, median filtering can be used to remove salt and pepper noise in the image, or adaptive histogram equalization can be used to enhance the contrast of the image. The advantage of this is that the image is clearer and the lesion area is more prominent, which is convenient for subsequent feature extraction and recognition. The denoising convolution subnetwork can effectively remove noise from medical ultrasound images. The principle is to extract a clean image from a noisy image by learning the pattern of noise. For example, a denoising convolution subnetwork based on U-Net can be constructed. The input of the network is a noisy ultrasound image, and the output is a denoised image. The network learns the characteristics of noise through the convolution layer and gradually restores the details of the image in the decoder part. The advantage of using a denoising convolutional subnetwork is that it can learn the characteristics of specific types of noise, thereby more effectively removing noise, such as speckle noise commonly seen in ultrasound images. In order to identify the lesion area, useful features need to be extracted from the image. The residual map feature can highlight the lesion area in the image, making it easier to identify. The feature extraction subnetwork can learn features related to the lesion area. For example, ResNet can be used as a feature extraction subnetwork, which can learn deep features of the image and effectively avoid the gradient vanishing problem. By extracting the residual map features, the difference between the lesion area and the surrounding tissue can be highlighted, which facilitates subsequent structure extraction and classification. The structure extraction subnetwork can further extract the boundary structure information of the lesion area. For example, a network containing multiple convolutional layers and pooling layers can be used to extract the boundary information of the lesion area. By extracting the residual map of the boundary structure, the lesion area can be more accurately located and information such as its shape and size can be obtained, which is crucial for subsequent diagnosis and treatment. Assuming that the extracted residual map features are not obvious, such as the lesion area is small or the contrast is low, a multi-layer output fusion technique can be used. For example, the outputs of different layers in the network can be fused to obtain feature information of different scales and improve the network's ability to recognize lesions of different sizes. This method can enhance the robustness and generalization ability of the network, and obtain better detection results even when the features of the lesion area are not obvious. If the extracted residual mapping features are obvious, such as the lesion area is large and the contrast is high, a convolutional neural network based on image segmentation, such as U-Net or SegNet, can be directly constructed. Suppose there is a data set containing 1,000 ultrasound images and their corresponding labels, where the labels mark the location and size of the lesion area. These data can be used to train the network so that it can automatically segment the lesion area.Through a large amount of training data, the network can learn the characteristics of the lesion area, thereby achieving accurate lesion area detection. After training, the network can be used to predict new medical ultrasound images. For example, there is a new ultrasound image. Through network prediction, it can be determined whether there is a lesion area in the image, and its location and size can be determined. If the prediction result shows that there is a lesion area, transfer learning can be further used to fine-tune the network model. Assuming that there is a larger medical ultrasound image dataset containing different types of lesions, this dataset can be used to fine-tune the previously trained network model. For example, the shallow parameters of the network can be frozen, and only the deep parameters can be fine-tuned, or a smaller learning rate can be used for fine-tuning. Doing so can improve the adaptability and generalization ability of the network model to new medical ultrasound image data, thereby obtaining more accurate lesion area detection results. The fine-tuned network model can more accurately detect the location and size of the lesion area, providing a more reliable reference for subsequent medical diagnosis. For example, the benign and malignant nature, size, shape and other information of the lesion can be judged based on the test results, helping doctors to formulate more effective treatment plans.

[0034] Step S102, for the preprocessed medical ultrasound image, a deep learning-based text detection algorithm, such as CTPN, EAST, etc., is used to automatically detect and locate the text area in the image, and obtain spatial information such as the position and size of the text.

[0035] The preprocessed medical ultrasound images are processed using deep learning-based text detection algorithms, such as CTPN and EAST, to automatically detect and locate the text area in the image. The residual mapping features related to the text area are extracted from the medical ultrasound image data through the feature extraction subnetwork. According to the extracted residual mapping features, the boundary structure of the text area is extracted by the structure extraction subnetwork to achieve accurate positioning of the text area. For the detected text area, the noise generated by the network is removed by the denoising convolution subnetwork to ensure the accuracy and clarity of the text area positioning results. The multi-layer output fusion technology is used to fuse feature information at different levels to enhance the robustness of detection and positioning of text areas of different sizes and shapes. Through transfer learning, the model pre-trained on other related tasks is used to improve the generalization ability of the text detection algorithm on medical ultrasound images. A convolutional neural network based on image segmentation is constructed, and a large number of medical ultrasound image training samples and their labels are used to train the network to obtain a network model that can intelligently detect and locate text areas in medical ultrasound images. In the prediction process, the trained network model automatically identifies and locates the text area in the medical ultrasound image, and obtains the spatial information such as the position and size of the text. The detected text regions are post-processed, such as merging overlapping text regions and filtering out text regions that are too small, to obtain the final text region detection and positioning results.

[0036] Specifically, medical ultrasound images often contain important text information, such as patient information, examination date, equipment parameters, etc. Extracting this text information is crucial for medical diagnosis, case management, and scientific research. Deep learning technology provides a powerful tool for automatically detecting and locating these text regions. Deep learning-based text detection algorithms such as CTPN and EAST can effectively identify text regions in medical ultrasound images. The CTPN algorithm is good at detecting horizontally arranged text, while the EAST algorithm is more suitable for detecting text in various directions. For example, in an abdominal ultrasound image, CTPN can accurately detect horizontal text information such as the patient's name and examination date located above the image, while EAST can detect annotation information in different positions and directions in the image, such as organ name, lesion size, etc. The feature extraction subnetwork extracts residual mapping features related to text regions from ultrasound images. These features contain the shape, texture, and edge information of the text region. For example, for the text annotation "liver", the residual mapping features highlight the strokes and edge information of the two characters "liver" and "viscera". The structure extraction subnetwork uses the residual mapping features to further extract the boundary structure of the text region for precise positioning. For example, it can accurately locate the start and end positions of the word "liver" to frame the entire text area. The denoising convolution subnetwork removes the noise accumulated during the deep network processing process to ensure the accuracy and clarity of the positioning results. For example, it can remove the influence of some artifacts and noise points in ultrasound images on the boundaries of text areas, making the boundaries of text areas clearer. The multi-layer output fusion technology fuses the feature information of different network layers to improve the detection robustness of text areas of different sizes and shapes. For example, the shallow network can capture the local detail features of the text, such as stroke information, while the deep network can capture the global structural features of the text, such as the shape of the text. Fusion of these features can improve the detection accuracy of texts of different sizes, such as "liver" and "gall bladder". Transfer learning uses the knowledge of pre-trained models to improve the generalization ability of text detection algorithms on medical ultrasound images. For example, an image classification model pre-trained on the ImageNet dataset can be used as the skeleton network of the text detection model, thereby accelerating the training of the model and improving its performance. In order to build a convolutional neural network for text detection in medical ultrasound images, a large amount of labeled data is required. This data needs to contain various types of text, such as text in different fonts, sizes, orientations, and locations. For example, it is necessary to collect ultrasound images with text annotations such as organ names, disease names, and measurement data. By training the network with these annotated data, a network model can be obtained that can accurately detect and locate text regions on new medical ultrasound images. In the prediction stage, the trained network model can automatically identify and locate text regions in ultrasound images and obtain information such as the location and size of the text.For example, it can frame the two words "liver" and give the coordinates of its upper left corner and lower right corner, as well as the width and height of the text box. The post-processing step can further optimize the text detection results. For example, it can merge the text boxes of the two words "liver" and "dirty" into a complete "liver" text box, and filter out some noise areas with too small an area, and finally obtain accurate text area detection and positioning results. This provides an important foundation for subsequent text recognition and information extraction.

[0037] Step S103, using OCR technology to perform text recognition based on the detected text area, comprehensively using a method based on character segmentation and a method based on sequence learning to accurately extract the text content in the image and obtain a preliminary text recognition result.

[0038] A convolutional neural network model based on deep learning is used to extract and encode the input image data to obtain the deep feature representation of the image. Based on the extracted deep features, a region proposal network (RPN) is used to generate candidate text regions, and a non-maximum suppression (NMS) algorithm is used to screen out high-confidence text regions. For each candidate text region, a character segmentation-based method is used to divide the text region into image blocks of single characters, and a convolutional neural network is used to extract and classify features of each character image block to identify the corresponding character. At the same time, a sequence learning-based method, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), is used to model and learn the character sequence in the text region to capture the contextual relationship and semantic information between characters. By integrating the results of character segmentation and sequence learning, the text content in the text region is comprehensively judged to obtain preliminary text recognition results. The preliminary text recognition results are post-processed and optimized, such as using language models and dictionary information to correct and complete the recognition results to improve the accuracy and robustness of recognition. By adopting the transfer learning method, language models pre-trained on large-scale text datasets, such as BERT or GPT, are used to further understand the semantics and context of the recognition results, and enhance the generalization ability of text recognition in complex scenarios. By building a neural network model based on the attention mechanism, the key characters and context information in the text area are focused and weighted, improving the recognition and understanding of important text content. According to the recognized text content, combined with the application scenario and business needs, the text is structured and semantically processed, such as entity recognition, relationship extraction and sentiment analysis, to provide support for subsequent information processing and application.

[0039] Specifically, deep learning plays a crucial role in medical ultrasound image text recognition. It can automatically extract features from images, recognize characters, and understand the text content, thereby improving the diagnostic efficiency and accuracy. The following will gradually analyze the application of deep learning in medical ultrasound image text recognition and illustrate it with specific examples. First of all, the convolutional neural network (CNN) is the core of deep learning, which can effectively extract the deep features of images. For example, a CNN for medical ultrasound image text recognition can contain multiple convolutional layers, pooling layers, and activation function layers. The convolutional layers use different convolutional kernels to extract different features of the image, such as edges, textures, etc. The pooling layers can reduce the dimension of the feature map and reduce the computational amount. The activation function layers introduce non-linearity into the network, enabling it to learn more complex patterns. Through the combination of these layers, the CNN can convert the original ultrasound image into a more representative feature vector. To locate the text region, the region proposal network (RPN) is widely used. The RPN generates multiple candidate boxes of different sizes and ratios on the feature map, and these candidate boxes may contain text. For example, on a feature map of size 512x512, the RPN can generate hundreds of candidate boxes, each of which is defined by its upper left coordinate, width, and height. To avoid a large number of overlapping boxes, the non-maximum suppression (NMS) algorithm is used to filter out the candidate box with the highest confidence. For example, if the overlapping area of two candidate boxes is large and the confidence of one box is significantly higher than the other, the NMS algorithm will retain the box with high confidence and suppress the box with low confidence. After accurately locating the text region, it is necessary to recognize the characters in it. The method based on character segmentation divides the text region into image blocks of individual characters. For example, the word "heart" will be divided into two character image blocks of "心" and "脏". Then, the CNN is used to extract features and classify each character image block to recognize the corresponding character. At the same time, sequence learning methods such as the recurrent neural network (RNN) can capture the context relationship between characters. For example, when recognizing the word "heart", the RNN will consider the front-back relationship between "心" and "脏" to more accurately recognize the whole word. To improve the recognition accuracy, it is necessary to post-process and optimize the recognition results. For example, a language model can be used to correct recognition errors. Suppose the recognition result is "liver enlargement", but the language model finds that "liver mass" is more in line with medical terms, then the recognition result will be corrected to "liver mass". In addition, dictionary information can also be used to complete missing characters. For example, if the recognition result is "肝_肿大" and the dictionary contains the word "liver enlargement", then it can be inferred that the missing character is "脏". Transfer learning can effectively utilize the language model pre-trained on a large-scale text dataset. For example, the BERT model has been pre-trained on a large-scale text data and learned rich language knowledge.Applying the pre-trained BERT model to medical ultrasound image text recognition can improve the generalization ability of the model, enabling it to better handle various medical terms and expressions. The attention mechanism can help the model focus on important text content. For example, when recognizing "left ventricular ejection fraction (LVEF)", the attention mechanism can focus more on the key characters "LVEF", thereby more accurately identifying the value of LVEF. Ultimately, the recognized text content needs to be structured and semantically processed. For example, the text "LVEF: 55%" can be entity recognized to identify "LVEF" as a medical indicator with a value of "55%". This will help doctors quickly obtain key information and make more accurate diagnoses. The application of deep learning has greatly improved the efficiency and accuracy of medical ultrasound image text recognition. Through the combination of feature extraction, region positioning, character recognition, sequence learning, post-processing optimization, transfer learning and attention mechanism, accurate recognition and understanding of complex text content in medical ultrasound images can be achieved, providing strong support for clinical diagnosis and treatment.

[0040] Step S104, for the text recognized by OCR, a professional terminology knowledge base and abbreviation dictionary in the medical field are constructed, and the recognized text is mapped to standard medical terms through dictionary matching, semantic analysis and other technologies, eliminating the influence of synonyms and abbreviations to obtain a standardized text representation.

[0041] According to the medical text recognized by OCR, the text is divided into several independent words and phrases through word segmentation and part-of-speech tagging technology, and the part of speech of each word is marked. The named entity recognition method based on rules and statistics is used to identify medical professional terms, abbreviations and other named entities in the text to obtain a preliminary list of terms. A professional terminology knowledge base and abbreviation dictionary in the medical field are constructed, and the recognized terms are matched with the standard terms in the knowledge base to eliminate the influence of synonyms and abbreviations. If there are unmatched terms, the semantic similarity between the unmatched terms and the terms in the knowledge base is calculated through semantic analysis technology to find the most similar standard terms. The recognized medical terms are mapped to the standard terms to obtain a standardized text representation and form structured medical text data. The standardized medical text is feature extracted, and the semantic features and context information of the text are learned through a deep learning model to obtain a low-dimensional dense vector representation of the text. A text classification model based on a convolutional neural network is used, and the extracted text feature vector is used as input to train the classifier to classify the medical text and determine the medical specialty or disease type to which the text belongs. On the basis of classification, the sequence annotation model is used to further analyze the semantic structure of medical texts, identify key information such as symptoms, signs, and examination results, and obtain structured medical information representation. The results of classification and semantic analysis are stored in a structured database to build a medical knowledge graph and realize intelligent retrieval, reasoning, and decision support of medical texts.

[0042] Specifically, word segmentation and part-of-speech tagging are basic steps in natural language processing. For example, for the sentence "The patient complained of headache, accompanied by nausea and vomiting.", the word segmentation result is "The patient / n complained of / v headache / n, / w accompanied by / v nausea / a vomiting / v. / w", where n represents noun, v represents verb, a represents adjective, and w represents punctuation. Part-of-speech tagging helps with subsequent named entity recognition and semantic analysis. Named entity recognition is to identify entities with specific meanings from text, such as diseases, symptoms, and drugs. In this example, "headache", "nausea", and "vomiting" are all medical-related named entities. Named entity recognition can use rule-based methods, such as maintaining a medical terminology dictionary and matching the words in the text with the dictionary; it can also use statistical machine learning-based methods, such as using a conditional random field (CRF) model to perform sequence annotation on the text and identify named entities. It is crucial to build a professional terminology knowledge base and abbreviation dictionary in the medical field. For example, "CAD" can refer to "coronary artery disease" or "computer-aided design". In medical texts, "CAD" usually refers to "coronary artery disease". By building a dictionary of medical abbreviations, "CAD" can be correctly mapped to "coronary artery disease". Semantic analysis techniques can help understand the meaning of text. For example, if an unmatched term "intermittent claudication" is identified, it can be mapped to the standard term "vascular claudication" by calculating its semantic similarity with the terms in the knowledge base and finding that it is very close to the semantics of "vascular claudication". Through the above steps, the original medical text can be converted into standardized structured data. For example, "The patient complained of headache, accompanied by nausea and vomiting." is converted to {"Main complaint": "Headache", "Accompanying symptoms": ["Nausea", "Vomiting"]}. Feature extraction is the process of converting text into a numerical representation that can be understood by the computer. For example, word embedding technology can be used to convert each word into a vector, and then the vectors of all words are combined to represent the entire text. Deep learning models can learn deep semantic features of text. For example, they can learn that "headache" and "nausea" often appear together, so that they are represented closer in the vector space. Text classification models can classify medical text into different categories. For example, a classifier can be trained to classify medical text into categories such as "cardiovascular disease", "respiratory disease", and "digestive disease". In this example, "The patient complained of headache, accompanied by nausea and vomiting." may be classified into the category of "digestive disease" or "neurological disease". The sequence labeling model can identify key information in the text. For example, for the sentence "The patient was admitted to the hospital due to continuous high fever for three days and was diagnosed with pneumonia." The sequence labeling model can identify that "continuous high fever for three days" is a "symptom" and "pneumonia" is a "disease". Storing the results of classification and semantic analysis in a structured database can be easily retrieved and queried.For example, you can query all cases diagnosed with "pneumonia" or all cases with the symptom of "headache". Building a medical knowledge graph can achieve more advanced applications. For example, you can infer possible diseases based on the patient's symptoms and test results, and provide corresponding treatment plans. For example, based on symptoms such as "headache", "nausea", and "vomiting", you can infer that the possible disease is "migraine" or "gastroenteritis", and provide corresponding treatment recommendations.

[0043] Step S105, based on text recognition, further analyze the semantic association between text and image, adopt multimodal learning methods, such as attention mechanism, image-text matching and other technologies, learn the joint representation of text and image, and explore the semantic correspondence between the two.

[0044] Convolutional neural networks are used to extract features from text and images to obtain deep semantic representations of text and images. The correlation between text and image features is calculated through the attention mechanism to obtain the relevance between text and image. According to the relevance between text and image, a text-image matching model is constructed to learn the joint representation of text and image. A multimodal learning method is used to fuse the joint representation of text and image to mine the semantic correspondence between the two. If the semantic correspondence between text and image meets the preset threshold, it is judged that there is a semantic association between the text and image; otherwise, it is judged that there is no semantic association between the text and image. For text and images with semantic association, a convolutional neural network based on image segmentation is used to segment and identify the target area in the image. Through transfer learning, the pre-trained network model is used to extract high-level semantic features of the target area in the image. The extracted semantic features of the target area are fused with the text features to construct a multimodal semantic understanding model to deeply understand the semantic content of text and image. An end-to-end training method is used to optimize the multimodal semantic understanding model, improve the accuracy and robustness of text and image semantic association analysis, and obtain high-quality text-image semantic association results.

[0045] Specifically, in the field of modern multimodal learning, convolutional neural networks (CNNs) are widely used for image feature extraction. They can effectively identify basic visual elements such as edges and textures from images and abstract higher-level semantic information layer by layer. For example, in medical image analysis, CNNs can identify the shape and size of tumors, which are crucial for disease diagnosis. At the same time, text feature extraction usually uses natural language processing techniques, such as word embedding, to convert text into vector form, which enables key information in the text, such as symptom descriptions, to be understood and processed by machines. The attention mechanism plays a bridging role in this process. It strengthens the model's attention to key information by calculating the correlation between text and image features. For example, when processing medical reports and corresponding X-ray images, the attention mechanism can help the model focus on analyzing the specific lesion area mentioned in the report and the corresponding position in the image, thereby improving the accuracy of diagnosis. When building a picture-text matching model, by learning the joint representation of text and image, the model can better understand the semantic relationship between the two. On e-commerce platforms, this technology can be applied to the matching of product images and descriptions to ensure the consistency of descriptions and image content and enhance the user's shopping experience. Multimodal learning methods further integrate text and image information to explore the deep semantic correspondence between the two. In autonomous driving systems, vehicles need to process image information of road signs and text information of traffic rules. Multimodal learning enables the system to integrate this information and make more accurate driving decisions. For image segmentation, convolutional neural networks can identify individual objects in images. For example, in images of farmland taken by drones, image segmentation technology can accurately identify the types of crops and their growth conditions, which is of great significance to precision agriculture. Using transfer learning, models pre-trained on large-scale datasets can be applied to specific small datasets, which not only improves the training efficiency of the model, but also maintains a high recognition accuracy. Finally, through end-to-end training, the optimization of multimodal semantic understanding models can be automatically adjusted throughout the data stream, improving the accuracy and robustness of the model for semantic association analysis of text and images. In social media analysis, this technology can be used to automatically identify and annotate pictures and texts posted by users, helping platforms better understand the emotional tendencies and topic trends of content, thereby providing more personalized content recommendations.

[0046] Step S106, integrating the results of text recognition and semantic understanding, constructing a multimodal annotation model for medical ultrasound images, and automatically generating text descriptions and keyword tags for the images through deep learning algorithms such as CNN and LSTM, thereby realizing automated and intelligent annotation of image content.

[0047] Convolutional neural networks are used to extract features from medical ultrasound images to obtain residual mapping features of the images. Based on the residual mapping features, the boundary structure information in the image is extracted through the structure extraction subnetwork. The noise generated by the network is removed by the denoising convolution subnetwork to obtain clear and accurate prediction results. The model's classification and interpretation robustness and generalization ability for medical ultrasound images of different scales are enhanced through multi-layer output fusion technology and transfer learning. Text recognition is performed on medical ultrasound images to obtain text information in the image. Natural language processing technology is used to perform semantic understanding on the recognized text information and extract keywords and semantic information. A multimodal annotation model for medical ultrasound images is constructed by integrating the results of image feature extraction and text semantic understanding. The multimodal annotation model is trained using deep learning algorithms such as CNN and LSTM to automatically generate text descriptions and keyword tags for the image. If the model prediction results meet the accuracy requirements, the automatically generated image text descriptions and keyword tags are output to achieve automatic and intelligent annotation of the content of medical ultrasound images; otherwise, return to step 1 to continue optimizing the model.

[0048] Specifically, in medical ultrasound image processing, convolutional neural network (CNN) is first used to extract features from the image. This step is mainly to obtain the residual mapping features of the image. For example, when processing cardiac ultrasound images, CNN can effectively identify the edges of the ventricles and the texture differences of the tissues. These features are crucial for subsequent disease diagnosis. Next, the boundary structure information in the image is further extracted through the structure extraction subnetwork. In practical applications, such as in liver ultrasound image analysis, the structure extraction subnetwork can accurately mark the boundaries of the liver and the internal vascular structure, which provides important structural information for subsequent liver disease analysis and diagnosis. In order to improve the image quality, the denoising convolution subnetwork is then used to remove the noise accumulated by the network. When processing fetal ultrasound images, the denoising network can effectively reduce the random noise and background interference in the image, making the fetal morphology and the images of important organs such as the heart and brain clearer, which is convenient for doctors to diagnose. Through multi-layer output fusion technology and transfer learning, the model's classification interpretation robustness and generalization of medical ultrasound images of different scales are enhanced. For example, when processing ultrasound images from different devices, transfer learning can help the model quickly adapt to the image characteristics of the new device, while multi-layer output fusion ensures that the features extracted from different levels can be used comprehensively, improving the diagnostic accuracy of the model. While processing the image, text recognition is performed on the medical ultrasound image to obtain text information in the image, such as patient information, device parameters, etc. Natural language processing technology is used to semantically understand the recognized text information and extract keywords and semantic information. This step is particularly important when processing ultrasound report images, which can automatically extract key diagnostic information in the report and assist doctors in quickly understanding the patient's condition. Combining the results of image feature extraction and text semantic understanding, a multimodal annotation model for medical ultrasound images is constructed. Using deep learning algorithms such as CNN and long short-term memory network (LSTM), the multimodal annotation model is trained to automatically generate text descriptions and keyword tags for the image. In practical applications, this can help doctors quickly obtain key information and possible diagnostic suggestions for the image when viewing ultrasound images, thereby improving diagnostic efficiency. If the model prediction results meet the accuracy requirements, the automatically generated image text description and keyword tags are output to achieve automated and intelligent annotation of medical ultrasound image content. This process greatly reduces the workload of doctors in daily medical practice and improves work efficiency and diagnostic accuracy.

[0049] Step S107, evaluate and optimize the automatically generated image annotation results, and continuously iterate and update the annotation model through manual review and feedback mechanism to improve the accuracy and completeness of the annotations, and finally form a high-quality medical ultrasound image annotation dataset to support subsequent medical image analysis and application.

[0050] The feature extraction subnetwork is used to extract residual mapping features related to the target structure from the medical ultrasound image data to obtain a preliminary image feature representation. Based on the extracted residual mapping features, the structure extraction subnetwork is used to further extract the boundary structure information in the medical ultrasound image and obtain the refined interpretation results of the target area. The interpretation results generated by the structure extraction subnetwork are processed by the denoising convolution subnetwork to remove the noise accumulated by the network and determine the clear and accurate segmentation results of the target area. Through the multi-layer output fusion technology, the feature extraction and structure interpretation results at different levels are fused to enhance the segmentation robustness and generalization ability of target structures of different scales. The transfer learning method is used to initialize the current ultrasound image annotation model using the network model pre-trained on other medical image segmentation tasks to accelerate the training convergence process of the model. A convolutional neural network based on image segmentation is constructed, and a large number of ultrasound image training samples and their manual annotation results are used to train the network to obtain a network model that can intelligently annotate the target structure in the medical ultrasound image. The trained ultrasound image annotation model is applied to new ultrasound image data, and automatic image annotation results are generated through the forward reasoning process. The automatically generated ultrasound image annotation results are evaluated through manual review to identify errors or incompleteness in the annotation results and form feedback information for the annotation model. Based on the feedback information from manual review, the annotation model is iteratively optimized and updated, and the accuracy and completeness of the annotation model are improved through retraining or fine-tuning, ultimately obtaining a high-quality medical ultrasound image annotation dataset to provide support for subsequent medical image analysis tasks.

[0051] Specifically, the goal of residual mapping feature extraction is to extract features that are sensitive to the target structure from the ultrasound image while suppressing the interference of irrelevant information. For example, suppose you want to extract the features of the fetal heart from the fetal ultrasound image. Due to the interference of noise and surrounding tissues in the ultrasound image, it is difficult to accurately identify the heart using the original image features directly. The residual mapping feature extraction network learns the difference between the heart area and the surrounding tissues, thereby highlighting the features of the heart and suppressing other irrelevant information. The feature representation obtained in this way can better reflect the characteristics of the target structure, such as the shape, size, and edges of the heart. The role of the structure extraction subnetwork is to further extract the boundary information of the target structure in the image based on the residual mapping features. Continuing with the fetal heart ultrasound image as an example, the structure extraction subnetwork can accurately locate the boundary of the heart and distinguish the heart from surrounding tissues, such as amniotic fluid and placenta. This is crucial for subsequent segmentation and annotation. The structure extraction subnetwork can use techniques such as edge detection and contour extraction to identify the boundary of the heart based on the difference information in the residual mapping features. The denoising convolution subnetwork is used to remove the noise generated by the network cumulatively to improve the clarity and accuracy of the prediction results. In the example of fetal heart ultrasound images, due to the characteristics of ultrasound imaging itself, there may be some speckle noise or artifacts in the image. The denoising convolutional subnetwork can learn the pattern of noise and remove this noise from the output of the structure extraction subnetwork to obtain a clearer heart boundary. This can improve the accuracy of subsequent segmentation and annotation and avoid misidentifying noise as the target structure. Multi-layer output fusion technology can combine feature information at different levels to improve the robustness and generalization ability of the model. For fetal heart ultrasound images, the features extracted by the shallow network may contain some texture detail information, while the features extracted by the deep network are more abstract and semantic. Fusion of these different levels of features can more comprehensively describe the target structure and improve the model's adaptability to heart images of different scales and shapes. Transfer learning can use network models pre-trained on other medical image segmentation tasks to speed up model training and improve performance. For example, a model pre-trained on other organ segmentation tasks (such as liver and kidney segmentation) can be used, its weights can be used as initialization parameters, and then fine-tuned on a fetal heart ultrasound image dataset. In this way, the medical image features that have been learned by the pre-trained model can be used to accelerate the convergence of the model and improve the performance on the fetal heart segmentation task. Construct a convolutional neural network based on image segmentation, and use a large number of ultrasound image training samples and their manual annotation results to train the network. Taking fetal heart ultrasound images as an example, a large number of fetal heart ultrasound images need to be collected and manually annotated by professional doctors to mark the heart area. Then use these annotated image data to train the convolutional neural network so that the network can learn the characteristics of the heart and automatically segment the heart area. Applying the trained model to new ultrasound image data can automatically generate image annotation results.For example, by applying a trained fetal heart segmentation model to a new fetal ultrasound image, the model can automatically identify and segment the fetal heart and annotate it on the image, such as outlining the heart with a red line or marking different parts of the heart with different colors. Manual review can evaluate the automatically generated annotation results and provide feedback information for model optimization. For example, a doctor can check the fetal heart segmentation results automatically generated by the model. If segmentation errors or omissions are found, they can be corrected and the corrected results can be used as new training data for retraining or fine-tuning the model. This can continuously improve the accuracy and completeness of the model. Through continuous iterative optimization, a high-quality medical ultrasound image annotation dataset can be obtained. This dataset can be used to assist doctors in diagnosis or to train more complex medical image analysis models. For example, a model can be trained using annotated fetal heart images to predict the health of the fetal heart.

[0052] The description of the above embodiments is only used to help understand the technical solutions and core ideas of the present application. Ordinary technical personnel in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0053] Embodiment 2:

[0054] The embodiment of the present invention further provides a medical ultrasound image annotation device, comprising:

[0055] A first processing device, configured to acquire a medical ultrasound image data set;

[0056] a second processing device for automatically detecting and locating a text region in an image based on a medical ultrasound image dataset;

[0057] A third processing device, used to obtain a text recognition result based on the detected text area;

[0058] A fourth processing device, used to obtain a normalized text representation according to the text recognition result;

[0059] A fifth processing device, for learning a joint representation of text and image based on the normalized text representation, and mining a semantic correspondence between the two;

[0060] The sixth processing device is used to automatically generate text descriptions and keyword tags of the image based on the results of text recognition and semantic understanding through a multimodal annotation model of medical ultrasound images, thereby realizing automated and intelligent annotation of image content.

[0061] As an implementation mode of an embodiment of the present invention, the second processing device adopts a text detection algorithm based on deep learning to automatically detect and locate the text area in the image and obtain spatial information such as the position and size of the text.

[0062] As an implementation mode of an embodiment of the present invention, the third processing device uses OCR technology to perform text recognition based on the detected text area, and uses a character segmentation-based method and a sequence learning-based method to extract text content in the image to obtain a text recognition result.

[0063] As an implementation mode of an embodiment of the present invention, the fourth processing device constructs a professional terminology knowledge base and abbreviation dictionary in the medical field based on the text recognized by OCR, and maps the recognized text to standard medical terms through dictionary matching and semantic analysis to obtain a standardized text representation.

[0064] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for annotating medical ultrasound images, characterized in that: include: Step S1, obtaining a medical ultrasound image data set; Step S2: automatically detecting and locating text areas in the image according to the medical ultrasound image dataset; Step S3, obtaining a text recognition result according to the detected text area; Step S4, obtaining a normalized text representation according to the text recognition result; Step S5: Based on the normalized text representation, learn the joint representation of text and image, and mine the semantic correspondence between the two; Step S6: Based on the results of text recognition and semantic understanding, a multimodal annotation model of medical ultrasound images is used to automatically generate text descriptions and keyword tags of the images, thereby achieving automated and intelligent annotation of image content.

2. The medical ultrasound image annotation method according to claim 1, characterized in that: In step S2, a text detection algorithm based on deep learning is used to automatically detect and locate the text area in the image, and obtain spatial information such as the position and size of the text.

3. The medical ultrasound image annotation method according to claim 2, characterized in that: In step S3, text recognition is performed using OCR technology based on the detected text area, and the text content in the image is extracted using a character segmentation-based method and a sequence learning-based method to obtain a text recognition result.

4. The method for annotating medical ultrasound images according to claim 3, characterized in that: In step S4, a professional terminology knowledge base and abbreviation dictionary in the medical field are constructed for the text recognized by OCR, and the recognized text is mapped to standard medical terms through dictionary matching and semantic analysis to obtain a standardized text representation.

5. A medical ultrasound image annotation system, characterized in that: include: A first processing device, configured to acquire a medical ultrasound image data set; a second processing device for automatically detecting and locating a text region in an image based on a medical ultrasound image dataset; A third processing device, used to obtain a text recognition result based on the detected text area; A fourth processing device, used to obtain a normalized text representation according to the text recognition result; A fifth processing device, for learning a joint representation of text and image based on the normalized text representation, and mining a semantic correspondence between the two; The sixth processing device is used to automatically generate text descriptions and keyword tags of the image based on the results of text recognition and semantic understanding through a multimodal annotation model of medical ultrasound images, thereby realizing automated and intelligent annotation of image content.

6. The medical ultrasound image annotation system according to claim 5, characterized in that: The second processing device adopts a text detection algorithm based on deep learning to automatically detect and locate the text area in the image and obtain spatial information such as the position and size of the text.

7. The medical ultrasound image annotation system according to claim 6, characterized in that: The third processing device uses the OCR technology to perform text recognition based on the detected text area, and uses a method based on character segmentation and a method based on sequence learning to extract the text content in the image to obtain a text recognition result.

8. The medical ultrasound image annotation system according to claim 7, characterized in that: The fourth processing device constructs a professional terminology knowledge base and abbreviation dictionary in the medical field based on the text recognized by OCR, and maps the recognized text to standard medical terms through dictionary matching and semantic analysis to obtain a standardized text representation.

Images