Automatic Recognition Method, Device, Terminal and Storage Medium for Thyroid Ultrasound Images

Abstract

The present invention discloses an automatic recognition method, device, terminal and storage medium for thyroid ultrasound images. By simultaneously extracting the feature data of the transverse view and the longitudinal view, and using a neural network regression model to determine the respective weights of the transverse view and the longitudinal view according to the feature data, the recognition results of the transverse view and the longitudinal view are weighted and fused to obtain the final recognition result. Compared with the prior art, the feature information extracted by the present invention is richer, and it can comprehensively focus on the ultrasound image recognition results of the two views, with a lower dependence on the clarity of the ultrasound image, and can automatically recognize the lesion area in the ultrasound image.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an automatic recognition method, device, terminal and storage medium for thyroid ultrasound images. Background Art

[0002] At present, the interpretation of thyroid ultrasound images mainly relies on the knowledge and experience of ultrasound doctors, which has great subjectivity. Moreover, with the sharp increase in the number of patients, the labor intensity of doctors will inevitably increase significantly.

[0003] In recent years, with the development of artificial intelligence and the improvement of computer technology, deep learning has developed rapidly and continuously made significant progress in the field of medical image processing, which can effectively reduce the misjudgment caused by subjective factors. However, due to the large differences in the size, shape and edge appearance of thyroid nodules in different thyroid ultrasound images, the existing network models only identify and process the transverse view in the ultrasound image, and the extracted information is relatively limited, and the lesion area in the image cannot be accurately segmented and recognized.

[0004] Therefore, the existing technology needs to be improved. Summary of the Invention

[0005] The main purpose of the present invention is to provide an automatic recognition method, device, intelligent terminal and storage medium for thyroid ultrasound images, which can extract various feature information and accurately segment and recognize the lesion area in the ultrasound image.

[0006] To achieve the above object, in the first aspect of the present invention, an automatic recognition method for thyroid ultrasound images is provided, and the method includes:

[0007] Obtain an ultrasound image, where the ultrasound image includes a transverse view and a longitudinal view;

[0008] Synchronously preprocess the transverse view and the longitudinal view;

[0009] Input the preprocessed transverse view and longitudinal view into a double-branch recognition network to obtain the feature data extracted by each branch and the recognition data output;

[0010] Input the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasound image;

[0011] Based on the weight data, perform weighted fusion on the recognition data to obtain a recognition result.

[0012] Optionally, the inputting the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasound image includes:

[0013] Input the feature data extracted by each branch in the dual-branch recognition network into the neural network regression model;

[0014] The neural network regression model is provided with multiple hidden layers. Through multiple fully connected operations of the hidden layers, the weight data corresponding to each ultrasonic image is obtained.

[0015] Optionally, each branch in the dual-branch recognition network includes a Swin-Transformer model and a convolutional neural network model. The Swin-Transformer models of the two branches share network weight parameters, and the convolutional neural network models of each branch use independent network weight parameters.

[0016] Optionally, it further includes pre-training the dual-branch recognition network, and the training steps include:

[0017] Obtain a training data set, which includes multiple pairs of transverse views and longitudinal views and corresponding label information;

[0018] Perform enhancement processing on the training data set to expand the sample quantity of the training data set;

[0019] Iteratively train the dual-branch recognition network according to the transverse views, longitudinal views and label information in the training data set.

[0020] Optionally, when performing iterative training on the dual-branch recognition network, the gradient update algorithm further includes a similarity constraint. The steps for calculating the similarity constraint include:

[0021] Obtain the feature data extracted by the dual-branch recognition network and input the feature data into the neural network regression model to obtain weight data;

[0022] Based on the comparison result of the weight data of the transverse view and the longitudinal view, divide the training data set into two groups;

[0023] Calculate the similarity constraint of each group of training data sets.

[0024] The second aspect of the present invention provides an automatic recognition device for thyroid ultrasound images. Among them, the above device includes:

[0025] An ultrasonic image acquisition module, configured to acquire ultrasonic images, where the ultrasonic images include transverse views and longitudinal views;

[0026] A preprocessing module, configured to perform synchronous preprocessing on the transverse view and the longitudinal view;

[0027] An automatic recognition module, configured to input the preprocessed transverse view and longitudinal view into the dual-branch recognition network to obtain the feature data extracted by each branch and the output recognition data;

[0028] A view weight module, configured to input the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasonic image;

[0029] A fusion module, configured to perform weighted fusion on the recognition data based on the weight data to obtain a recognition result.

[0030] Optionally, it further includes a training module, configured to obtain a training data set, where the training data set includes multiple pairs of transverse views and longitudinal views and corresponding label information, perform enhancement processing on the training data set to expand the sample quantity of the training data set, and iteratively train the dual-branch recognition network according to the transverse views, longitudinal views, and label information in the training data set.

[0031] Optionally, the training module further includes a similarity constraint unit for calculating the similarity constraint in the gradient update algorithm. The similarity constraint unit is configured to obtain the feature data extracted by the dual-branch recognition network and input the feature data into the neural network regression model to obtain weight data, divide the training data set into two groups based on the comparison result of the weight data of the transverse view and the longitudinal view, and calculate the similarity constraint of each group of training data sets.

[0032] A third aspect of the present invention provides an intelligent terminal. The intelligent terminal includes a memory, a processor, and a thyroid ultrasonic image automatic recognition program stored on the memory and executable on the processor. When the thyroid ultrasonic image automatic recognition program is executed by the processor, it implements the steps of any one of the above thyroid ultrasonic image automatic recognition methods.

[0033] A fourth aspect of the present invention provides a computer-readable storage medium. A thyroid ultrasonic image automatic recognition program is stored on the computer-readable storage medium. When the thyroid ultrasonic image automatic recognition program is executed by a processor, it implements the steps of any one of the above thyroid ultrasonic image automatic recognition methods.

[0034] As can be seen from the above, the present invention simultaneously extracts the feature data of the transverse view and the longitudinal view, and uses a neural network regression model to determine the respective weights of the transverse view and the longitudinal view according to the feature data, so as to perform weighted fusion on the recognition results of the transverse view and the longitudinal view to obtain a final recognition result. Compared with the prior art, the feature information extracted by the present invention is richer, and it can comprehensively focus on the ultrasonic image recognition results of the two views, has a lower dependence on the clarity of the ultrasonic image, and can automatically recognize the lesion area in the ultrasonic image. Description of the Drawings

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0036] Figure 1 is a schematic flowchart of the specific process of the automatic thyroid ultrasound image recognition method provided by the embodiment of the present invention;

[0037] Figure 2 is Figure 1 the functional block diagram of the embodiment;

[0038] Figure 3 is a schematic flowchart of the specific process of the embodiment for training the dual-branch recognition network;

[0039] Figure 4 is Figure 3 the schematic flowchart of the specific process of step A300 in the embodiment;

[0040] Figure 5 is a schematic structural diagram of the automatic thyroid ultrasound image recognition device provided by the embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the internal structural principle of an intelligent terminal provided by the embodiment of the present invention. Specific Embodiments

[0042] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0043] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0044] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0045] It should also be further understood that the term "and / or" as used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0046] As used in this specification and the appended claims, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to recognizing". Similarly, the phrase "if determined" or "if [the described condition or event] is recognized" may be construed, depending on the context, as meaning "once determined", "in response to determining", "once [the described condition or event] is recognized", or "in response to recognizing [the described condition or event]".

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0048] Many specific details are set forth in the following description in order to provide a thorough understanding of the present invention. However, the present invention may be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0049] Thyroid cancer is the most common malignant tumor in endocrine system tumors, and its incidence shows an increasing trend year by year. Thyroid ultrasound examination is the preferred method for identifying thyroid nodules. However, the differences between different types of thyroid nodules are small, and the differences within the same type are large. It is difficult to distinguish the boundaries between thyroid nodules and the surrounding thyroid tissues. The determination results of ultrasound images rely on the clinical experience and cognitive ability of doctors, and have great subjectivity.

[0050] Existing network models only perform recognition processing on transverse views, and the extracted feature information is limited, and the accuracy of lesion area recognition is not high. How to use computer technology to process thyroid ultrasound images and replace doctors to complete the tracking and recognition of diseased tissues is an urgent problem to be solved at present.

[0051] The present invention constructs a dual-branch recognition network to simultaneously extract the feature information in the transverse view and longitudinal view obtained by ultrasound scanning, so that the extracted feature information is richer, and can comprehensively focus on the ultrasound image recognition results of the two views, and can automatically recognize the lesion area in the ultrasound image.

[0052] Exemplary method

[0053] The automatic recognition method for thyroid ultrasound images in this embodiment is deployed on an electronic terminal, such as a doctor's computer terminal, to obtain ultrasound image data for processing and display information such as recognition results and the importance of each ultrasound image on the computer screen. As Figure 1 shown, the automatic recognition method specifically includes the following steps:

[0054] Step S100: Obtain an ultrasound image, where the ultrasound image includes a transverse view and a longitudinal view;

[0055] Specifically, during an ultrasound examination, simultaneous transverse and longitudinal ultrasound scans can be performed to obtain a transverse view and a longitudinal view. In this embodiment, the electronic terminal accesses an ultrasound image database through a network to obtain the ultrasound image after ultrasound scanning.

[0056] Step S200: Synchronously preprocess the transverse view and the longitudinal view;

[0057] Specifically, redundant information such as the model information and patient information in the ultrasound image is cropped out, and only the actual content obtained by scanning in the ultrasound image is retained to prevent redundant information from interfering with the prediction model; then, normalization operation is performed on the ultrasound image. Through the ultrasound image preprocessing step, the applicability of the recognition method of the present invention in different ultrasound data styles can be enhanced, not limited to a single ultrasound device, and the dependence on the imaging quality of the ultrasound device can be reduced.

[0058] Optionally, denoising and enhancement processing can also be performed on the ultrasound image from two aspects of time domain and frequency domain. For example: The image denoising and enhancement methods in the time domain are all based on the gray value transformation of the pixel points in the image, including filtering the local pixel neighborhood of the image using various specially designed small sliding windows, and achieving histogram equalization of the ultrasound image by changing the distribution of pixel points at different gray levels; Most of the image enhancement methods in the frequency domain are to convert the time domain image to the corresponding frequency domain through Fourier transform, Z transform, etc., then specifically adjust and transform a certain frequency or certain frequencies, and finally inverse transform back to the time domain.

[0059] To facilitate the processing of the feature information of the transverse view and the longitudinal view, it is necessary to synchronously process the transverse view and the longitudinal view, that is, the processing steps performed on one ultrasound image are also used to process the other ultrasound image.

[0060] Step S300: Input the preprocessed transverse view and longitudinal view into a dual-branch recognition network to obtain the feature data extracted by each branch and the recognition data output;

[0061] Specifically, in this embodiment, an identification model is constructed based on the Swin-Transformer model and the convolutional neural network model to form a dual-branch identification network. The dual-branch identification network includes two branches, and the architectures of each branch are the same. One branch processes the preprocessed cross-sectional view, and the other branch processes the preprocessed longitudinal view. The front end of each branch is a Swin transformer model, and the back end is a convolutional neural network model. The Swin transformer network, as the backbone network for ultrasonic image feature extraction, can improve the ability to extract effective features of the image compared to other network models. Moreover, the same set of network weight parameters is shared between the Swin transformer models at the front end of the branches, and the convolutional neural network models at the back end of the branches each use independent network weight parameters. The shared network weight parameters enable the two branches to learn the feature information of the other view, realizing the interactive fusion of feature information between different views. Then, by using independent network weight parameters for each convolutional neural network, the results output by each branch are more accurate.

[0062] As Figure 2 shown, after inputting the preprocessed cross-sectional view and longitudinal view into the dual-branch identification network, the output result of the Swin transformer model at the front end of the dual-branch identification network is the extracted feature data, and the output result of the convolutional neural network model at the back end is the identification data.

[0063] Step S400: Input the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasonic image;

[0064] Specifically, the present invention does not treat the information of different ultrasonic images indifferently. Because this method ignores the fact that the importance of the information contained in different views is different in diagnosis; moreover, for different patients, there are significant differences in the size, shape, and appearance of the nodule edges of thyroid nodules. Using the same identification strategy for the ultrasonic images of each patient, the identification accuracy is limited. The present invention can quantify the importance of different ultrasonic images according to the specific situation of each patient, form a personalized identification scheme, and improve the identification accuracy.

[0065] First, the feature data corresponding to each ultrasonic image, that is, the feature data corresponding to the cross-sectional view and the feature data corresponding to the longitudinal view, are both input into the neural network regression model. The neural network regression model has multiple hidden layers. After multiple full connection operations of the hidden layers, the weight data corresponding to each ultrasonic image is output. It should be noted that the specific architecture of the neural network regression model is not limited. It can be a separate architecture or a part of the dual-branch identification network as a view weight model.

[0066] Step S500: Perform weighted fusion on the recognition data based on the weight data, obtain the recognition result and output it.

[0067] Specifically, according to the obtained weight data corresponding to each view, perform weighted fusion on the recognition data obtained from each branch to obtain the final recognition result, that is, segment the lesion area to form a distinction from the surrounding thyroid tissue, so as to facilitate the recognition of nodule images.

[0068] As described above, the present invention comprehensively utilizes the information of the transverse view and the longitudinal view, and quantifies the importance of each view of each patient. Compared with the indiscriminate fusion of the information of different views and the neglect of the patient's specificity and the difference between views, the present invention can automatically assign weights to different views according to the specific situation of the multi-view ultrasound images of each patient, form a personalized plan, and finally obtain the recognition result of personalized weighted decision-making. Therefore, it better meets the personalized needs, can provide auxiliary opinions for clinical diagnosis, reduce the errors caused by subjective factors, improve the diagnosis efficiency and accuracy, and at the same time reduce the workload of doctors.

[0069] Furthermore, the nodules can be further classified and judged according to the recognition result to provide a basis for classifying the benign and malignant nature of the nodules, etc.

[0070] In one embodiment, as Figure 3 shown, the specific steps for pre-training the dual-branch recognition network include:

[0071] Step A100: Obtain a training data set, where the training data set includes multiple pairs of transverse views and longitudinal views and corresponding label information;

[0072] Specifically, in this embodiment, each pair of data sets (transverse view and longitudinal view) after clinical ultrasound scanning is obtained from a medical image database, and then pathological examination is performed on each view to obtain label information.

[0073] Step A200: Perform enhancement processing on the training data set to expand the sample quantity of the training data set;

[0074] Specifically, since annotating and labeling ultrasound images is a very time-consuming task. Therefore, when training the dual-branch recognition network, in order to make up for the possible shortage of image data volume in the training data set, the present invention first performs image enhancement processing on the ultrasound images in the training data set. The specific method of image enhancement is not limited, such as translating, rotating, mirroring, and scaling the image, etc. Through the enhancement processing, the diversity of the training data set can be expanded, the sample quantity of the training data can be increased, the model accuracy and generalization ability can be improved, and overfitting can be avoided.

[0075] In this embodiment, a combination of contrast adjustment, rotation, translation, and scaling with random quantities and random transformation degrees is adopted to achieve a better model training effect.

[0076] It should be noted that since the dual-branch recognition network processes the transverse view and the longitudinal view simultaneously, the above two types of ultrasonic images need to be processed synchronously. That is, after one ultrasonic image is subjected to image enhancement, the other ultrasonic image also needs to be subjected to the same image enhancement operation.

[0077] Step A300: Iteratively train the dual-branch recognition network according to the transverse view, longitudinal view, and label information in the training dataset.

[0078] Specifically, each pair of transverse view and longitudinal view in the training dataset and the corresponding label information are input into the dual-branch recognition network. The dual-branch recognition network can automatically learn the image features of the thyroid nodules in the transverse view and longitudinal view according to the guidance of the label information, obtain the recognition result, and use gradient backpropagation according to the comparison result between the label information and the recognition result to guide the update of the network parameters of the dual-branch recognition network.

[0079] In this embodiment, the loss function used by the dual-branch recognition network is the cross-entropy loss function, which can obtain the global minimum value.

[0080] Furthermore, the neural network regression model is also trained simultaneously to improve the accuracy of determining the importance of the transverse view and the longitudinal view.

[0081] As described above, this embodiment adopts the enhancement processing of the training dataset, which can expand the diversity of the training dataset, increase the sample quantity of the training data, improve the model accuracy and generalization ability, and avoid overfitting. And according to the label information, gradient backpropagation is used to guide the update of the network parameters of the dual-branch recognition network and the neural network regression model, so as to generate more accurate recognition results and view weights.

[0082] Furthermore, when the dual-branch recognition network is iteratively trained in the present invention, the gradient update algorithm also considers the similarity constraint. Among them, as Figure 4 shown, the steps of calculating the similarity constraint include:

[0083] Step A310: Obtain the feature data extracted by the dual-branch recognition network and input the feature data into the neural network regression model to obtain the weight data;

[0084] Step A320: Divide the training dataset into two groups based on the comparison result of the weight data of the transverse view and the longitudinal view;

[0085] Step A330: Calculate the similarity constraint of each group of training datasets.

[0086] Specifically, first, obtain the feature data output by the Swin transformer model at the front end of the dual-branch recognition network, and input the feature data into the neural network regression model to obtain the weight data. Then, divide the training data set into two corresponding groups according to the weight of the cross-sectional view >= the weight of the longitudinal view and the weight of the cross-sectional view < the weight of the longitudinal view. Then, calculate the within-group similarity constraint based on the feature data, thereby constraining the update of the network parameters. The similarity constraint is to calculate the absolute error distance between every two data sets within the group, sum them up and take the average, and then perform backpropagation by calculating the gradient to make the absolute error distance smaller and smaller.

[0087] Among them, the method for calculating the similarity constraint is: calculate the first distance between the feature data of every two cross-sectional views in each group; calculate the second distance between the feature data of every two longitudinal views in each group; then calculate the average value of all the first distances, the average value of all the second distances, and obtain the similarity constraint based on the direction in which all the average values decrease. The specific expression is:

[0088]

[0089] Ls represents the calculation result of the similarity (i.e., MAE Loss), Among them, f represents the feature data, t and l represent the cross-sectional view and the longitudinal view respectively, g1 and g2 represent the corresponding two groups of data, g1_num and g2_num represent the number of each group, and i and j are two different training data within each group. C is the combination formula. ∈ is a very small positive number to prevent the denominator from being 0.

[0090] As described above, in this embodiment, the within-group contrast learning strategy is used to enable the dual-branch recognition network to obtain more accurate recognition results.

[0091] The following are the experimental results obtained in the internal thyroid data set in this embodiment, as well as the results of relevant comparative experiments. All experimental results come from the same training and test data. The evaluation metrics used are accuracy (Accuracy, ACC) and F1-score. And, the network used in the single view is consistent with the network backbone of the classification module in the present invention to ensure the fairness of the comparison. It can be seen from the following table that the nodule prediction results of the present invention are better than those of the single-view method. The results show that the performance of the recognition method of the present invention is significantly better than the other two methods.

[0092]

[0093] Exemplary device

[0094] Such as Figure 5As shown, corresponding to the above-mentioned automatic recognition method for thyroid ultrasound images, an embodiment of the present invention further provides an automatic recognition device for thyroid ultrasound images, and the above-mentioned automatic recognition device for thyroid ultrasound images includes:

[0095] An ultrasonic image acquisition module 600, configured to acquire an ultrasonic image, where the ultrasonic image includes a transverse view and a longitudinal view;

[0096] A preprocessing module 610, configured to perform synchronous preprocessing on the transverse view and the longitudinal view;

[0097] An automatic recognition module 620, configured to input the preprocessed transverse view and longitudinal view into a dual-branch recognition network, and obtain feature data extracted by each branch and recognition data output;

[0098] A view weight module 630, configured to input the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasonic image;

[0099] A fusion module 640, configured to perform weighted fusion on the recognition data based on the weight data to obtain a recognition result.

[0100] Optionally, it further includes a training module, configured to obtain a training data set, where the training data set includes multiple pairs of transverse views and longitudinal views and corresponding label information; perform enhancement processing on the training data set to expand the sample quantity of the training data set; perform iterative training on the dual-branch recognition network according to the transverse views, longitudinal views, and label information in the training data set.

[0101] Optionally, the training module further includes a similarity constraint unit for calculating the similarity constraint in the gradient update algorithm. The similarity constraint unit is configured to obtain the feature data extracted by the dual-branch recognition network and input the feature data into a neural network regression model to obtain weight data, and divide the training data set into two groups based on the comparison result of the weight data of the transverse view and the longitudinal view, and calculate the similarity constraint of each group of training data sets.

[0102] Specifically, in this embodiment, the specific functions of the above-mentioned modules of the automatic recognition device for thyroid ultrasound images can refer to the corresponding descriptions in the above-mentioned automatic recognition method for thyroid ultrasound images, and will not be elaborated here.

[0103] Based on the above embodiment, the present invention further provides an intelligent terminal, and its principle block diagram can be as Figure 6As shown in the figure. The above intelligent terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and an automatic thyroid ultrasound image recognition program. The internal memory provides an environment for the operation of the operating system and the automatic thyroid ultrasound image recognition program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the automatic thyroid ultrasound image recognition program is executed by the processor, it implements the steps of any one of the above automatic thyroid ultrasound image recognition methods. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0104] Those skilled in the art can understand that Figure 6 the block diagram of the principle shown in the figure is only the block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the intelligent terminal to which the solution of the present invention is applied. The specific intelligent terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0105] In one embodiment, an intelligent terminal is provided. The above intelligent terminal includes a memory, a processor, and an automatic thyroid ultrasound image recognition program stored on the above memory and executable on the above processor. When the automatic thyroid ultrasound image recognition program is executed by the above processor, the following operation instructions are performed:

[0106] Obtain an ultrasound image, where the ultrasound image includes a transverse view and a longitudinal view;

[0107] Synchronously preprocess the transverse view and the longitudinal view;

[0108] Input the preprocessed transverse view and longitudinal view into a dual-branch recognition network to obtain the feature data extracted by each branch and the recognition data output;

[0109] Input the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasound image;

[0110] Based on the weight data, perform weighted fusion on the recognition data to obtain a recognition result.

[0111] Optionally, the step of inputting the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasound image includes:

[0112] Input the feature data extracted by each branch in the dual-branch recognition network into the neural network regression model;

[0113] The neural network regression model has multiple hidden layers. Through multiple fully connected operations of the hidden layers, the weight data corresponding to each ultrasonic image is obtained.

[0114] Optionally, each branch in the dual-branch recognition network includes a Swin-Transformer model and a convolutional neural network model. The Swin-Transformer models of the two branches share network weight parameters, and the convolutional neural network models of each branch use independent network weight parameters.

[0115] Optionally, it further includes pre-training the dual-branch recognition network. The training steps include:

[0116] Obtain a training data set, where the training data set includes multiple pairs of transverse views and longitudinal views and corresponding label information;

[0117] Perform enhancement processing on the training data set to expand the sample quantity of the training data set;

[0118] Iteratively train the dual-branch recognition network according to the transverse views, longitudinal views and label information in the training data set.

[0119] Optionally, when iteratively training the dual-branch recognition network, the gradient update algorithm further includes a similarity constraint. The steps for calculating the similarity constraint include:

[0120] Obtain the feature data extracted by the dual-branch recognition network and input the feature data into the neural network regression model to obtain weight data;

[0121] Based on the comparison result of the weight data of the transverse view and the longitudinal view, divide the training data set into two groups;

[0122] Calculate the similarity constraint of each group of training data sets.

[0123] Optionally, the calculation of the similarity constraint of each group of training data sets includes:

[0124] Calculate the first distance between the feature data of every two transverse views in each group;

[0125] Calculate the second distance between the feature data of every two longitudinal views in each group;

[0126] Calculate the average value of all the first distances, the average value of all the second distances and obtain the similarity constraint based on the direction in which all the average values decrease.

[0127] An embodiment of the present invention further provides a computer-readable storage medium, on which a thyroid ultrasound image automatic recognition program is stored. When the thyroid ultrasound image automatic recognition program is executed by a processor, the steps of any one of the thyroid ultrasound image automatic recognition methods provided by the embodiments of the present invention are implemented.

[0128] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and do not limit the protection scope of the present invention. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments and will not be elaborated here.

[0130] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0132] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are only illustrative. For example, the above-mentioned division of modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0133] If the above integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present invention, it can also be completed by a computer program instructing relevant hardware. The above computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the above computer program includes computer program code, and the above computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The above computer-readable medium can include: any entity or device capable of carrying the above computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the above computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0134] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the present invention for the corresponding technical solutions, and should all be included in the protection scope of the present invention.

Claims

1. An automatic recognition method for thyroid ultrasound images, characterized in that The method includes: Obtaining an ultrasonic image, which includes a transverse view and a longitudinal view; Performing synchronous preprocessing on the transverse view and the longitudinal view; Inputting the preprocessed transverse view and longitudinal view into a dual-branch recognition network to obtain the feature data extracted by each branch and the output recognition data; Inputting the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasonic image; Performing weighted fusion on the recognition data based on the weight data to obtain and output a recognition result; Each branch in the dual-branch recognition network includes a Swin-Transformer model and a convolutional neural network model. The Swin-Transformer models of the two branches share network weight parameters, and the convolutional neural network models of each branch use independent network weight parameters; It also includes pre-training the dual-branch recognition network, and the training steps include: Obtaining a training data set, which includes multiple pairs of transverse views and longitudinal views and corresponding label information; Performing enhancement processing on the training data set to expand the sample quantity of the training data set; Iteratively training the dual-branch recognition network according to the transverse views, longitudinal views, and label information in the training data set; When iteratively training the dual-branch recognition network, a similarity constraint is also included in the gradient update algorithm. The steps for calculating the similarity constraint include: Obtaining the feature data extracted by the dual-branch recognition network and inputting the feature data into a neural network regression model to obtain weight data; Based on the comparison result of the weight data of the transverse view and the longitudinal view, dividing the training data set into two groups; Calculating the similarity constraint of each group of training data sets; Based on the comparison result of the weight data of the transverse view and the longitudinal view, dividing the training data set into two groups, including: Dividing the training data set into corresponding two groups according to whether the weight of the transverse view is greater than or equal to the weight of the longitudinal view or the weight of the transverse view is less than the weight of the longitudinal view.

2. The automatic thyroid ultrasound image recognition method according to claim 1, wherein The step of inputting the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasonic image includes: Inputting the feature data extracted by each branch in the dual-branch recognition network into the neural network regression model; The neural network regression model is provided with multiple hidden layers. Through multiple full connection operations of the hidden layers, the weight data corresponding to each ultrasonic image is obtained.

3. An automatic recognition device for thyroid ultrasound images, characterized in that, The device includes: An ultrasonic image acquisition module for obtaining an ultrasonic image, which includes a transverse view and a longitudinal view; A preprocessing module for performing synchronous preprocessing on the transverse view and the longitudinal view; An automatic recognition module for inputting the preprocessed transverse view and longitudinal view into a dual-branch recognition network to obtain the feature data extracted by each branch and the output recognition data; A view weight module for inputting the feature data into a neural network regression model to obtain weight data for reflecting the importance of the ultrasonic image; A fusion module for performing weighted fusion on the recognition data based on the weight data to obtain a recognition result; Each branch in the dual-branch recognition network includes a Swin-Transformer model and a convolutional neural network model. The Swin-Transformer models of the two branches share network weight parameters, and the convolutional neural network models of each branch use independent network weight parameters; It further includes a training module for obtaining a training data set. The training data set includes multiple pairs of cross-sectional views and longitudinal-sectional views and corresponding label information, enhancing the training data set to expand the number of samples in the training data set, and iteratively training the dual-branch recognition network according to the cross-sectional views, longitudinal-sectional views, and label information in the training data set; The training module further includes a similarity constraint unit for calculating the similarity constraint in the gradient update algorithm. The similarity constraint unit is used to obtain the feature data extracted by the dual-branch recognition network and input the feature data into a neural network regression model to obtain weight data. Based on the comparison result of the weight data of the cross-sectional view and the longitudinal-sectional view, the training data set is divided into two groups, and the similarity constraint of each group of training data sets is calculated; Based on the comparison result of the weight data of the cross-sectional view and the longitudinal-sectional view, the training data set is divided into two groups, including: dividing the training data set into corresponding two groups according to the cross-sectional view weight being greater than or equal to the longitudinal-sectional view weight and the cross-sectional view weight being less than the longitudinal-sectional view weight.

4. Smart terminal, characterized in that, The intelligent terminal includes a memory, a processor, and a thyroid ultrasound image automatic recognition program stored on the memory and executable on the processor. When the thyroid ultrasound image automatic recognition program is executed by the processor, it implements the steps of the thyroid ultrasound image automatic recognition method according to any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, A thyroid ultrasound image automatic recognition program is stored on the computer-readable storage medium. When the thyroid ultrasound image automatic recognition program is executed by the processor, it implements the steps of the thyroid ultrasound image automatic recognition method according to any one of claims 1-2.

Images