Image recognition device, electronic device, medium, and program

Abstract

The present disclosure provides an image recognition device, an electronic device, a computer readable storage medium and a computer program. The image recognition device comprises a fusion perception unit, a feature recognition unit and a collaborative inference unit. The fusion perception unit is configured to obtain a fusion feature expression of medical images and metadata; the feature recognition unit is configured to determine a plurality of candidate feature groups from a candidate feature set based on the medical images or the metadata, the plurality of candidate feature groups corresponding to a plurality of professional dimensions one by one; and for each of the plurality of professional dimensions, the fusion feature expression is classified to obtain a prediction result description of each of the plurality of candidate feature groups; and the collaborative inference unit is configured to diagnose and grade the medical images based on the prediction result description to obtain a diagnosis conclusion. The image recognition device can identify medical images according to clinical features, make the diagnosis process transparent and consistent with the doctor's thinking, improve the processing performance of medical images with fuzzy boundaries, and reduce the generation of illusions.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, and in particular to an image recognition device, electronic device, computer-readable storage medium, and computer program. Background Technology

[0002] With the rapid iteration of modern clinical medicine and imaging technology, multimodal medical imaging such as X-ray, CT, MRI, ultrasound, and pathological sections has become the core basis for disease screening, clinical diagnosis, preoperative evaluation, and postoperative follow-up, and is widely used in the entire process of diagnosis and treatment of various diseases such as tumors, cardiovascular and cerebrovascular diseases, respiratory diseases, and nervous diseases.

[0003] Currently, mainstream AI-based ultrasound imaging diagnostic solutions are mainly divided into two categories: classification based on convolutional neural networks (CNNs) and localization based on object detection / segmentation. CNN-based classification typically uses mature backbone networks such as ResNet and EfficientNet. The process involves first preprocessing the data (e.g., cropping, denoising, and normalizing the ultrasound image), then extracting features (e.g., automatically extracting deep features through multi-layer convolutional operations), and finally outputting a probability score indicating whether the ultrasound image is "benign" or "malignant" through a fully connected layer. Object detection / segmentation-based localization typically uses frameworks such as YOLO, U-Net, or Mask R-CNN. The process involves first manually annotating the image (e.g., doctors manually delineating lesion boundaries or drawing bounding rectangles), and then predicting the region, i.e., the model predicts pixel-level segmentation masks or bounding boxes to assist doctors in locating lesions. Summary of the Invention

[0004] To address the problems in the related technologies, embodiments of this disclosure provide an image recognition device, an electronic device, a computer-readable storage medium, and a computer program.

[0005] One aspect of this disclosure provides an image recognition apparatus, comprising: a fusion perception unit, a feature recognition unit, and a collaborative reasoning unit, wherein the fusion perception unit is configured to acquire a fusion feature representation of a medical image and metadata, wherein the metadata is clinical information related to the medical image; the feature recognition unit is configured to determine multiple candidate feature groups from a candidate feature set based on the medical image or the metadata, wherein the multiple candidate feature groups correspond one-to-one with multiple professional dimensions; and to perform classification processing on the fusion feature representation for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups; the collaborative reasoning unit is configured to perform diagnostic grading on the medical image based on the prediction result description. The collaborative reasoning unit comprises: a matching subunit configured to divide the medical image into multiple first local regions, and match the local features of the multiple first local regions with the prediction result description to obtain the matching degree between the multiple first local regions and the prediction result description; an analysis subunit configured to determine the region of interest and uncertainty score in the medical image based on the matching degree; a detail extraction subunit configured to extract details from the region of interest in response to the uncertainty score meeting a preset condition; and a diagnostic grading subunit configured to perform diagnostic grading on the medical image based on the medical image, the details, and the multiple professional dimensions to obtain a diagnostic conclusion.

[0006] According to an embodiment of this disclosure, the analysis subunit includes: a heatmap generation subunit configured to generate a heatmap based on the matching degree and according to the original physical locations of the plurality of first local regions in the medical image; a region of interest determination subunit configured to perform threshold segmentation and connected component analysis based on the spatial response value distribution of the heatmap to determine the region of interest, wherein the spatial response values ​​are suppressed based on a first threshold and the spatial response values ​​are connected and aggregated based on a second threshold; and an uncertainty scoring subunit configured to determine the uncertainty score based on the spatial statistical features of the heatmap, wherein the spatial statistical features are calculated based on the normalized spatial response value distribution of the heatmap; the spatial statistical features include one or more of peak intensity, spatial entropy, and spatial variance; and the uncertainty score is obtained by weighted combination of the spatial statistical features.

[0007] According to an embodiment of this disclosure, the image recognition device further includes: a logic closed-loop verification unit, configured to compare the medical diagnostic criteria corresponding to the diagnostic conclusion with the prediction result description to determine whether the prediction result description and the diagnostic conclusion have a logical conflict; and to perform feedback redirection in response to the logical conflict between the prediction result description and the diagnostic grading result, wherein the feedback redirection includes: reducing the sampling step size and the collaborative inference unit re-grading the medical image based on the prediction result description based on the reduced sampling step size to obtain a diagnostic conclusion, wherein the sampling step size is the step size used by the collaborative inference unit when sampling the local features of the multiple local regions respectively.

[0008] According to an embodiment of this disclosure, the analysis subunit is further configured to: label the medical image in response to the uncertainty score being higher than a third threshold, and perform the feedback redirection.

[0009] According to an embodiment of this disclosure, the collaborative reasoning unit further includes an output subunit configured to output preliminary structured text based on the prediction result description, and to output the diagnostic conclusion.

[0010] According to an embodiment of this disclosure, the collaborative inference unit further includes an adaptive subunit. The parameters of the adaptive subunit include trainable adaptive parameters, which are loaded into the attention mechanism-related network layer of the image recognition device and configured to adaptively adjust the image recognition device while keeping the backbone network parameters unchanged, so as to adapt to the differences in image style produced by different imaging devices.

[0011] According to an embodiment of this disclosure, the detail extraction subunit is configured to: in response to the uncertainty score satisfying the preset condition, use a high-resolution feature operator to extract features from the region of interest to obtain the detail features, wherein the high-resolution feature operator refers to dividing the region of interest into multiple second local regions, the size of the second local regions being smaller than the first local regions, or reducing the sampling step size of the collaborative inference unit in sampling the multiple first local regions to obtain the local features.

[0012] According to an embodiment of this disclosure, the diagnostic grading subunit is configured to: acquire global image contour features of the medical image based on the medical image; and perform diagnostic grading on the medical image based on the global image contour features, the detail features, and the multiple professional dimensions to obtain a diagnostic conclusion.

[0013] According to embodiments of this disclosure, multiple professional dimensions include at least one of the following: echo level, edge morphology, aspect ratio / length-to-width ratio, calcification features / microcalcification, growth pattern, and internal structure. The candidate feature group corresponding to the echo level includes at least two of the following: extremely low echo, low echo, isoechoic, anechoic, homogeneous internal echo, and heterogeneous echo. The candidate feature group corresponding to the edge morphology includes at least two of the following: unclear edge, clear boundary, irregular edge, lobed, and burr-like. The candidate feature group corresponding to the aspect ratio / length-to-width ratio includes: aspect ratio > 1 and aspect ratio < 1. The candidate feature group corresponding to the calcification features / microcalcification includes at least two of the following: presence of microcalcification, coarse calcification, annular calcification, and absence of calcification. The candidate feature group corresponding to the growth pattern includes: vertical growth and parallel growth. The candidate feature group corresponding to the internal structure includes at least two of the following: solid, cystic-solid, cystic, and spongy.

[0014] According to an embodiment of this disclosure, the fusion perception unit, feature recognition unit, and collaborative reasoning unit include a deep learning model with a self-attention mechanism. The parameters of the deep learning model include backbone network parameters and adaptive network parameters. The adaptive network parameters include the parameters of the adaptive subunit. The image recognition device further includes a post-training unit configured to update the adaptive network parameters while keeping the backbone network parameters unchanged.

[0015] According to an embodiment of this disclosure, the fusion perception unit includes: an image encoder configured to convert the medical image into a high-dimensional spatial vector; a text encoder configured to convert the metadata into a semantic vector; and a fusion subunit configured to map the high-dimensional spatial vector and the semantic vector into feature vectors of the same dimension via a visual projection matrix and a text projection matrix, respectively, and then perform feature concatenation and fusion to obtain the fused feature expression. The backbone network parameters include parameters from the image encoder and the text encoder, and the adaptive network parameters further include parameters from the visual projection matrix and the text projection matrix.

[0016] According to an embodiment of this disclosure, the feature recognition unit includes: a structured feature prediction classification layer configured to perform a linear mapping operation on the fused feature expression to obtain a full candidate feature score matrix; a candidate feature group extraction subunit configured to extract the scores of each channel in the plurality of candidate feature groups from the full candidate feature score matrix; and a prediction result description generation subunit configured to obtain the prediction result description based on the scores of each channel in the plurality of candidate feature groups. The adaptive network parameters also include the parameters of the structured feature prediction classification layer.

[0017] According to embodiments of this disclosure, the parameters of the adaptive subunit include the low-rank adaptation (LoRA) matrix pair in the self-attention and cross-attention mechanism bypass of the decoder in the deep learning model.

[0018] According to an embodiment of this disclosure, the post-training unit is configured to: filter target data in unlabeled data whose diagnostic conclusion confidence is greater than a preset threshold; calculate the prediction result of the target data, and calculate the loss function of the prediction result and the diagnostic conclusion; and use the backpropagation algorithm to update the adaptive network parameters so that the loss function converges.

[0019] Another aspect of this disclosure provides an electronic device, including an image recognition device provided in any embodiment of this disclosure.

[0020] Another aspect of this disclosure provides a computer-readable storage medium configured to store non-transitory computer-readable instructions that, when executed by a computer, perform the following: acquiring a fusion feature representation of a medical image and metadata, wherein the metadata is clinical information related to the medical image; determining multiple candidate feature groups from a candidate feature set based on the medical image or the metadata, wherein the multiple candidate feature groups correspond one-to-one with multiple professional dimensions; classifying the fusion feature representation for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups; and performing a diagnostic grading of the medical image based on the prediction result description to obtain a diagnostic conclusion.

[0021] Another aspect of this disclosure provides a computer program product, including a computer program carried on a non-transitory computer-readable medium, the computer program comprising: acquiring a fusion feature representation of a medical image and metadata, wherein the metadata is clinical information related to the medical image; determining a plurality of candidate feature groups from a candidate feature set based on the medical image or the metadata, the plurality of candidate feature groups corresponding one-to-one with a plurality of professional dimensions; classifying the fusion feature representation for each of the plurality of professional dimensions to obtain a prediction result description for each of the plurality of candidate feature groups; and performing a diagnostic grading of the medical image based on the prediction result description to obtain a diagnostic conclusion. Attached Figure Description

[0022] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0023] Figure 1 A block diagram schematically illustrates an image recognition apparatus provided according to at least one embodiment of the present disclosure; Figure 2 This diagram illustrates the overall architecture of an image recognition device provided in at least one embodiment of the present disclosure; Figure 3 A block diagram of an analysis subunit provided in at least one embodiment of the present disclosure is shown; Figure 4 A schematic block diagram of an electronic device provided for some embodiments of this disclosure; Figure 5 A schematic block diagram of another electronic device provided for some embodiments of this disclosure; and Figure 6 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. Detailed Implementation

[0024] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to implement them more readily. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings. In this disclosure, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, operations, behaviors, components, portions, or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, numbers, operations, behaviors, components, portions, or combinations thereof. It should also be noted that, without conflict, embodiments and features in the embodiments of this disclosure can be combined with each other. The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Using classification paths based on convolutional neural networks for medical image recognition is a typical "end-to-end black box" model. The system cannot explain why it is judged as malignant, and doctors cannot verify the specific anatomical basis through AI feedback. Localization based on object detection / segmentation relies excessively on high-quality pixel-level annotations and handles the "boundary ambiguity" of medical images very poorly. Due to the presence of acoustic shadowing and artifacts in medical images, traditional "hard segmentation" is prone to false alarms (hallucinations) under noise interference and lacks clinical logical support, which may lead to a logical gap between the localization results and the final diagnostic classification (such as TI-RADS, Thyroid Imaging Reporting and Data System). Therefore, current medical image recognition suffers from the following problems: 1. Existing models cannot inform doctors of the specific anatomical features upon which they determine benign or malignant conditions; 2. Ultrasound image quality is highly susceptible to operator technique and equipment gain, and existing models are prone to misjudgment or "illusions" when faced with noise or blurred sections, exhibiting poor robustness; 3. Current segmentation tasks rely on a large amount of precise pixel-level annotations, but obtaining such annotations in clinical practice is time-consuming, labor-intensive, and inconsistent in standards, requiring different models to be trained for different diseases or organs. Furthermore, current medical image recognition is prone to logical conflicts, where the model-generated descriptive text contradicts the final risk classification, leading to low clinical reliability.

[0026] Therefore, at least one embodiment of this disclosure provides an image recognition device, which includes: a fusion perception unit, a feature recognition unit, and a collaborative reasoning unit. The fusion perception unit is configured to acquire a fusion feature representation of medical images and metadata, where the metadata is clinical information related to the medical images. The feature recognition unit is configured to determine multiple candidate feature groups from a candidate feature set based on the medical images or metadata, with each candidate feature group corresponding to multiple professional dimensions; and to classify the fusion feature representation for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups. The collaborative reasoning unit is configured to perform diagnostic grading on the medical images based on the prediction result description to obtain a diagnostic conclusion. The collaborative reasoning unit includes a matching subunit, an analysis subunit, a detail extraction subunit, and a diagnostic grading subunit. The matching subunit is configured to divide the medical image into multiple first local regions, and match the local features of each of the multiple first local regions with the predicted result description to obtain the matching degree between the multiple first local regions and the predicted result description; the analysis subunit is configured to determine the region of interest and uncertainty score in the medical image based on the matching degree; the detail extraction subunit is configured to extract details from the region of interest in response to the uncertainty score meeting preset conditions; and the diagnostic grading subunit is configured to perform diagnostic grading on the medical image based on the medical image, details, and multiple professional dimensions to obtain a diagnostic conclusion. This image recognition device can recognize medical images based on clinical characteristics, making the diagnostic process transparent and consistent with doctors' thinking, and improving the processing performance of medical images with blurred boundaries, reducing the occurrence of hallucinations. In some other embodiments of this disclosure, the image recognition device can also alleviate the problem of logical conflicts, at least partially avoiding contradictions between descriptive text and risk levels, and improving clinical credibility.

[0027] Figure 1 A block diagram of an image recognition device 100 provided according to at least one embodiment of the present disclosure is shown schematically.

[0028] like Figure 1 As shown, the image recognition device 100 includes a fusion perception unit 101, a feature recognition unit 102, and a collaborative reasoning unit 103.

[0029] The fusion sensing unit 101 is configured to acquire a fusion feature representation of medical images and metadata, wherein the metadata is clinical information related to the medical images.

[0030] The feature recognition unit 102 is configured to determine multiple candidate feature groups from the candidate feature set based on medical images or metadata, with each candidate feature group corresponding to a specific professional dimension; and to classify the fused feature expression for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups.

[0031] The collaborative reasoning unit 103 is configured to perform diagnostic grading on medical images based on the prediction results to obtain a diagnostic conclusion.

[0032] The collaborative reasoning unit 103 includes a matching subunit 113, an analysis subunit 123, a detail extraction subunit 133, and a diagnostic grading subunit 143.

[0033] The matching subunit 113 is configured to divide the medical image into multiple first local regions, and match the local features of the multiple first local regions with the prediction result description to obtain the matching degree between the multiple first local regions and the prediction result description.

[0034] Analysis subunit 123 is configured to determine the region of interest and uncertainty score in medical images based on the degree of matching.

[0035] The detail extraction subunit 133 is configured to extract detailed features from the region of interest in response to the uncertainty score meeting a preset condition.

[0036] The diagnostic grading subunit 143 is configured to perform diagnostic grading on medical images based on medical images, detailed features, and multiple professional dimensions to obtain diagnostic conclusions.

[0037] This image recognition device can identify medical images based on clinical characteristics (i.e., professional dimensions), making the diagnostic process transparent and in line with doctors' thinking. It also improves the processing performance of medical images with blurred boundaries through uncertainty scoring, reducing the occurrence of hallucinations.

[0038] In some embodiments of this disclosure, medical images may include, but are not limited to, static ultrasound images, X-ray images, computed tomography (CT) scans, magnetic resonance imaging (MRI) images, nuclear medicine images, endoscopic images, and pathological microscopic images. Metadata may include, for example, the clinical information of the patient to whom the medical image pertains. Clinical information may be textual information or structured data, carrying information including, but not limited to, age, family history of nodules, laboratory thyroid function indicators, chief symptoms, radiological examination reports, pathology reports, and historical medical records.

[0039] The following is combined with Figure 2 The fusion perception unit, feature recognition unit, and collaborative reasoning unit provided in some embodiments of this disclosure are described in detail. Figure 2 An overall architecture diagram of an image recognition device 100 provided in at least one embodiment of the present disclosure is shown.

[0040] For example, the fusion sensing unit 101 receives medical images (e.g., ultrasound images) and metadata (e.g., age / medical history, etc.), uses an image encoder (such as VisionTransformer, ViT) to transform the image into a high-dimensional spatial vector, uses a text encoder to transform the clinical background into a semantic vector, and then performs feature fusion through feature concatenation to form a fused feature representation with clinical context awareness. The image encoder can also be other types of encoders, not limited to ViT, such as ResNet, EfficientNet, MLP-Mixer, ConvMixer, etc.

[0041] like Figure 2 As shown, when the metadata is text information, the fusion sensing unit 101 may include an image encoder 111, a text encoder 121, and a fusion subunit 131.

[0042] Image encoder 111 is configured to convert medical images into high-dimensional spatial vectors; text encoder 121 is configured to convert metadata into semantic vectors; fusion subunit 131 is configured to map the high-dimensional spatial vectors and semantic vectors into feature vectors of the same dimension through visual projection matrix and text projection matrix respectively, and then perform feature splicing and fusion to obtain fused feature expression.

[0043] For example, the fusion sensing unit 101 inputs medical images into the image encoder 111 (e.g., VisionTransformer, ViT), and the medical images are encoded by the image encoder (ViT) to obtain visual features (i.e., high-dimensional spatial vectors). The fusion sensing unit 101 inputs metadata into the text encoder 121, and the text information is encoded by the text encoder 121 to obtain text features.

[0044] The fusion subunit 131 includes, for example, a visual projection matrix and a text projection matrix. Visual features and text features are mapped to feature vectors via their respective visual and text projection matrices, and then fused feature representations are formed through feature fusion or feature concatenation. The dimensions of the visual and text feature vectors obtained from the mapping between the visual and text projection matrices can be similar or dissimilar. When their dimensions are dissimilar, the visual or text feature vectors can be processed by filling in null values ​​to make their dimensions fusionable.

[0045] In some embodiments of this disclosure, the feature recognition unit 102 first determines the anatomical location of the medical image, and after determining the anatomical location currently being processed, determines multiple candidate feature groups from the candidate feature set.

[0046] For example, anatomical locations in medical images can be determined through metadata. For instance, by parsing metadata and obtaining entity labels for anatomical locations output by a large model, the system can confirm that the currently scanned site is the thyroid gland. Because the fused feature representation contains a deep semantic vector about "thyroid" from the clinical metadata, the large model's attention mechanism can confirm that the currently processed anatomical location is the thyroid gland when receiving this fused feature representation. When processing fused features, the large model can be guided, for example, by using prompts to output entity labels for anatomical locations according to specified format constraints (schema).

[0047] In other embodiments of this disclosure, the anatomical location to be processed can also be determined based on medical images. For example, the anatomical location to be processed can be determined by identifying the shape, size, etc., of the medical images.

[0048] In some embodiments of this disclosure, the candidate feature set includes, for example, all candidate feature options supported by the image recognition device. All candidate feature options include, for example, candidate feature groups corresponding to multiple clinical organs (e.g., thyroid, breast, abdomen, carotid artery, heart, etc.) supported by the image recognition device. For example, each clinical organ may have one or more candidate feature groups. A candidate feature group represents the characteristics of a clinical organ in a medical professional dimension (e.g., edge morphology, echo level, focal calcification, etc.). For example, the candidate feature group corresponding to the thyroid gland may include an edge morphology feature group, an echo level feature group, and a focal calcification feature group. The edge morphology feature group may include, for example, three classification features: clear edge, unclear edge, and angular edge; the echo level feature group may include, for example, four classification features: hyperechoic, isoechoic, hypoechoic, and very hypoechoic; the focal calcification feature group may include, for example, two classification features: with focal calcification and without focal calcification.

[0049] In some embodiments of this disclosure, for example, an image recognition device can pre-construct a discrete unified candidate feature grouping index table, whose spatial dimension is denoted as T (the unified candidate feature grouping index table constitutes a system-level shared medical sign semantic space, and different organ tasks complete conditional probability partitioning and structured prediction within the same feature space through the dynamic candidate feature grouping index table). This unified candidate feature grouping index table is used to uniformly store all candidate feature options supported by the system (e.g., including thyroid, breast, abdomen, carotid artery, heart, etc.). All candidate feature options serve as a set of clinical local feature candidates and are arranged sequentially according to a discrete index method. The value of T can be dynamically expanded according to actual task requirements (e.g., when T = 60, the table is: [0, 1, 2, ..., 58, 59]). Therefore, when the system adds a new organ type, disease type, or new clinical sign definition, it only needs to add the corresponding discrete index interval in the unified candidate feature grouping index space and update the corresponding candidate feature grouping index table, without modifying the backbone network structure or rebuilding the independent prediction model. For example, the thyroid candidate feature groups corresponding to the current input include: the edge morphology feature group, corresponding to channels 0-2 in the unified candidate feature grouping index table, where channels 0-2 represent clear edges, unclear edges, and angular edges, respectively; the echo level feature group, corresponding to channels 3-6 in the unified candidate feature grouping index table, where channels 3-6 represent hyperechoic, isoechoic, hypoechoic, and very hypoechoic, respectively; and the focal calcification feature group, corresponding to channels 7-8 in the unified candidate feature grouping index table, where channels 7-8 represent focal calcification and no focal calcification, respectively.

[0050] For example, multiple professional dimensions may include at least one of the following: echo level, edge morphology, aspect ratio, calcification features / microcalcification, growth pattern, and internal structure. The candidate feature group corresponding to echo level may include at least two of the following: extremely low echo, low echo, isoechoic, anechoic, homogeneous internal echo, and heterogeneous echo; the candidate feature group corresponding to edge morphology may include at least two of the following: indistinct edge, clear boundary, irregular edge, lobed, and spiculated; the candidate feature group corresponding to aspect ratio / length-width ratio may include: aspect ratio > 1 and aspect ratio < 1; the candidate feature group corresponding to calcification features / microcalcification may include at least two of the following: presence of microcalcification, coarse calcification, annular calcification, and absence of calcification; the candidate feature group corresponding to growth pattern may include: vertical growth and parallel growth; the candidate feature group corresponding to internal structure may include at least two of the following: solid, cystic-solid, cystic, and spongy.

[0051] For example, for edge morphology in the professional dimension, the fused features are classified to obtain the prediction results for the edge morphology feature group. For instance, if the classification result of the fused features for the edge morphology feature group is [Clarity: 0.12, Unclear: 0.85, Angularity: 0.03], then the prediction result is described as: Unclear (pre-score 0.85). [Clarity: 0.12, Unclear: 0.85, Angularity: 0.03] means that the edge morphology score is 0.12 for clarity, 0.85 for uncleanliness, and 0.03 for angularity. The prediction result is described as the channel with the highest score in the candidate feature group and the score of that channel. Similarly, for the echo level feature group, if the classification result of the fused features is [High Echo: 0.05, Iso-echo: 0.05, Low Echo: 0.2, Very Low Echo: 0.7], then the prediction result is described as: Very Low Echo (pre-score 0.7).

[0052] like Figure 2 As shown, in some embodiments of this disclosure, the feature recognition unit 102 includes a structured feature prediction classification layer 112, a candidate feature group extraction subunit 122, and a prediction result description generation subunit 132.

[0053] The structured feature prediction classification layer 112 is configured to perform a linear mapping operation on the fused feature representation to obtain the full candidate feature score matrix.

[0054] The candidate feature group extraction subunit 122 is configured to extract the scores of each channel in multiple candidate feature groups from the full candidate feature score matrix.

[0055] The prediction result description generation subunit 132 is configured to obtain the prediction result description based on the scores of each member in multiple candidate feature groups.

[0056] This embodiment strengthens the judgment logic of medical image recognition through clinical features, making the diagnostic process transparent and in line with doctors' thinking, improving the interpretability of the output conclusions and increasing the credibility of the conclusions.

[0057] For example, the structured feature prediction classification layer 112 is an independent classification layer that shares backbone features with the output head of the large model's native vocabulary. Its internal linear mapping matrix has dimensions set to [D,T], where D is the dimension of the hidden layer features and T is the number of unified candidate feature grouping indices, i.e., the spatial dimension of the aforementioned unified candidate feature grouping index table.

[0058] Obtain the fused feature representation (e.g., denoted as the feature tensor Z). thyroid ), characteristic tensor Z thyroid ∈[B,D], where B is the batch size and D is the hidden layer feature dimension. The feature tensor Z... thyroidA linear mapping operation is performed with the structured feature prediction classification layer 112 to obtain the full candidate feature score matrix CLS_Logits∈[B,T]([B,D]x[D,T]), which contains the prediction scores of the current input corresponding to the grouping index of all candidate features.

[0059] The candidate feature group extraction subunit 122 extracts the predicted scores of candidate features at corresponding index positions in the full candidate feature score matrix CLS_Logits based on multiple candidate feature groups. Candidate feature groups are denoted as Group_1_List, Group_2_List, Group_3_List, ..., Group_Index_List, where Index is a positive integer. For example, for the input of the current scene: Group_1_Logits = CLS_Logits[:, [0,1,2]]; Group_2_Logits = CLS_Logits[:, [3,4,5,6]]; Group_3_Logits = CLS_Logits[:, [7,8]].

[0060] For example, the prediction result description generation subunit 132 independently performs Softmax probability normalization calculation on the prediction scores of each candidate feature group to obtain the conditional probability distribution of each clinical professional dimension. Furthermore, it performs maximum index extraction (Argmax) and word segmentation text decoding on the probability matrix after Softmax activation, concurrently outputting the prediction result description of the candidate feature group related to the current thyroid gland for each feature. For example, edge morphology: [Clear: 0.12, Unclear: 0.85, Angularity: 0.03], output: Unclear (predicted score 0.85); echo level: [Hyperecho: 0.05, Isoecho: 0.05, Hypoecho: 0.2, Very Hypoecho: 0.7], output: Very Hypoecho (predicted score 0.7); focal calcification: [Present: 0.2, Absent: 0.8], output: Absent (predicted score 0.8).

[0061] The collaborative reasoning unit 103 is configured to diagnose and classify medical images based on the prediction results to arrive at a diagnostic conclusion. For example, for medical images of the thyroid gland, the diagnostic conclusion could be TI-RADS 5. RADS (Thyroid Imaging Reporting and Data System) is a standard for grading the malignancy risk of thyroid nodules using ultrasound. Its core function is to score and classify nodules using a unified standard to guide whether a biopsy or surgery is necessary.

[0062] For medical images of other organs, a diagnostic classification can be derived by referring to the risk grading standards for those organs, such as BI. RADS (breast cancer), LI RADS (Liver Health), PI RADS (prostate), etc.

[0063] like Figure 1 As shown, the collaborative reasoning unit 103 includes a matching subunit 113, an analysis subunit 123, a detail extraction subunit 133, and a diagnostic grading subunit 143.

[0064] like Figure 2 As shown, the matching subunit 113 obtains the degree of matching between multiple first local regions of the medical image and the prediction results, thereby obtaining the region of interest in the medical image through the analysis subunit 123, and further obtaining an uncertainty score; then, the detail extraction subunit 133 and the diagnostic grading subunit 143 perform global-local collaborative reasoning. For example, the detail extraction subunit 133 extracts features from the region of interest to obtain detailed features, and then the diagnostic grading subunit 143 performs diagnostic grading on the medical image based on the medical image, detailed features, and multiple professional dimensions to obtain a diagnostic conclusion.

[0065] For example, matching subunit 113 divides the medical image into multiple small regions (patch), each of which is called a first local region. Then, for each small region, local features are sampled according to the sampling step size. The text feature vector of the prediction result description (such as "microcalcification", "blurred edges", etc.) is compared with the local features of the small regions to obtain the matching degree between multiple first local regions and the prediction result description.

[0066] Text feature vectors are high-dimensional semantic vectors (e.g., 512-dimensional or 768-dimensional) obtained by encoding medical terms (such as "microcalcification") using a text encoder. For example, the degree of matching is determined through dot product or cosine similarity calculations. For instance, the system performs dot product operations or calculates cosine similarity between the text feature vector of "microcalcification" and the visual feature vector generated from image encoding, one by one.

[0067] The patch size is a customizable parameter; for example, it can be 16 x 16 or 14 x 14, and can be designed according to different clinical medical tasks, such as setting it to 32 x 32 for larger and clearer images. This disclosure does not limit the patch size.

[0068] like Figure 2As shown, the collaborative reasoning unit 103 also includes an output subunit 153, which is configured to output preliminary structured text based on the prediction result description, and output the diagnostic conclusion. For example, the collaborative reasoning unit 103 outputs a preliminary structured text containing descriptions of each dimension (such as "irregular edges, aspect ratio > 1", etc.) based on the prediction result description.

[0069] For example, the selected text feature vector, such as the encoded vector of "spiculated edges," is placed in the same vector space as the image patch feature vector extracted by the visual encoder, and the cosine similarity between each patch vector and the text vector is calculated. Each patch will obtain a similarity score (i.e., the degree of matching). The higher the score, the more the visual features of that tiny region match the medical description of "spiculated edges."

[0070] Figure 3 A block diagram of an analysis subunit 123 provided in at least one embodiment of the present disclosure is shown.

[0071] like Figure 3 As shown, the analysis subunit 123 includes a heatmap generation subunit 1231, a region of interest determination subunit 1232, and an uncertainty scoring subunit 1233.

[0072] The heatmap generation subunit 1231 is configured to generate a heatmap based on the degree of matching and according to the original physical location of multiple first local regions in the medical image.

[0073] For example, the heatmap generation subunit 1231 rearranges these similarity scores scattered in space according to the original physical location of the patch, forming a two-dimensional similarity matrix, i.e., a heatmap. The weights of the entire image are normalized using the Softmax function, transforming the originally discrete scores into a probability distribution between 0 and 1. On the heatmap, areas with high similarity are presented as "highlighted" or "hot spots," and these areas are the evidence regions identified by the image recognition device that support the medical sign.

[0074] The region of interest determination subunit 1232 is configured to perform threshold segmentation and connected component analysis based on the spatial response value distribution of the heatmap to determine the region of interest, suppress the spatial response value based on the first threshold, and perform connected component aggregation based on the spatial response value based on the second threshold.

[0075] In some embodiments of this disclosure, the region of interest is dynamically delineated and generated, unlike traditional image recognition where a bounding box is directly drawn. For example, the region of interest is delineated through threshold filtering and connected component analysis.

[0076] The spatial response value distribution can be exemplified by the matching degree distribution in medical images. The spatial response value represents the matching degree, such as cosine similarity. By setting a first threshold (e.g., 0.6), irrelevant background regions (i.e., regions with low matching degrees) are eliminated. Then, regions with high matching degrees (i.e., matching degrees higher than a second threshold) are connected and aggregated to automatically form one or more irregular shapes (e.g., these shapes precisely cover the parts of the lesion with "marginal spiculation" features). The second threshold can be the same as or different from the first threshold. For example, the second threshold could be 0.6, 0.8, etc. The resulting region of interest is a set of these high-weight pixels that not only defines the location of the lesion but also clarifies "which specific boundary segment" within the lesion leads to the relevant diagnostic conclusion.

[0077] In some embodiments of this disclosure, the region of interest (ROI) is initially obtained based on threshold segmentation and connected component analysis. The initially obtained ROI can then be modulated to obtain a ROI that conforms to the medical context. For example, ultrasound nodules often have extremely blurred boundaries, and the ROI generated solely by threshold filtering may cut off some edge details. In practice, after initially obtaining the ROI, a morphological dilation operator is used to adaptively expand the region (e.g., expand it by N pixels) to include the lesion's infiltrative edges, thereby obtaining a set of ROI pixels with an irregular shape that accurately covers the heterogeneous signs of the lesion.

[0078] Uncertainty scoring subunit 1233 is configured to determine the uncertainty score based on the spatial statistical characteristics of the heatmap. The spatial statistical characteristics are calculated based on the normalized spatial response value distribution of the heatmap. The spatial statistical characteristics include one or more of peak intensity, spatial entropy, and spatial variance. The uncertainty score is obtained by weighted combination of the spatial statistical characteristics.

[0079] Uncertainty scoring subunit 1233 provides an uncertainty score by analyzing the distribution of the region of interest. If the weights are highly concentrated in a specific region, it indicates that the image recognition device has clearly found the feature, and the uncertainty score is reduced. If the region of interest is very diffusely distributed (i.e., it looks like something everywhere, but nothing is certain), a higher uncertainty score is output, indicating insufficient information in that region of interest. In some embodiments of this disclosure, if the maximum value of the matching degree is extremely low, a higher uncertainty score is output, indicating insufficient information in that region of interest.

[0080] In some embodiments of this disclosure, the uncertainty score is primarily achieved by analyzing the spatial statistical characteristics of the heatmap. This judgment process does not rely on manually set fixed thresholds, but rather on the discriminative ability learned by the model during the training phase through large-scale medical image data. The system dynamically provides the uncertainty score for the diagnosis by calculating the deviation between the "pixel-level matching degree distribution" (i.e., the local feature vector of a local region in the medical image) and the "standard clinical sign distribution" (i.e., the text feature vector). The introduction of uncertainty, and the handling of blurred edges in medical images through probability distribution, better suits the physical imaging characteristics of ultrasound than traditional "hard segmentation."

[0081] For example, in practice, the uncertainty score is calculated using the spatial statistical features of the heatmap generated by cross-modal cosine similarity, which include one or more of the heatmap's peak intensity, spatial entropy, and spatial variance.

[0082] Specifically, the heatmap is used as input, and the aforementioned spatial statistical features are extracted and then weighted and summed to generate an uncertainty score. The calculation process is as follows: , This represents the response value of the i-th local region in the heatmap. Text feature vectors.

[0083] Peak intensity: ;entropy: ; It is a heat map The spatial probability distribution obtained after normalization (softmax). Variance: ; ; .

[0084] The normalized uncertainty score. , and The weighting coefficients can be set to fixed values ​​or learned during the training phase based on labeled medical image data to adapt to the uncertainty assessment needs of data in different domains. In some embodiments of this disclosure, the analysis subunit 123 is also configured to label the medical image and perform feedback redirection in response to an uncertainty score higher than a third threshold.

[0085] In some embodiments of this disclosure, the third threshold is determined through comparative testing. For example, during the development phase of an image recognition device, the device processes a large number of ultrasound images with known diagnostic results. The device records confidence scores when recognition is accurate, and sets an appropriate threshold by statistically analyzing the distribution of these scores. The third threshold may be, for example, 0.7, but is not limited thereto. Labeling medical images, for example, refers to labeling medical images as regions of interest when information is insufficient.

[0086] Please refer to the following description for feedback redirection.

[0087] In some embodiments of this disclosure, the detail extraction subunit 133 is configured to: in response to the uncertainty score meeting a preset condition, use a high-resolution feature operator to extract features from the region of interest to obtain detailed features. The high-resolution feature operator refers to dividing the region of interest into multiple second local regions, the size of which is smaller than that of the first local region, or reducing the sampling step size of the collaborative inference unit to sample the local features from the multiple first local regions respectively.

[0088] The uncertainty score meets a preset condition, such as being less than or equal to a third threshold. If the uncertainty score meets the preset condition, a higher resolution feature operator is used to extract detailed features from the region of interest.

[0089] For example, key lesion areas (i.e., regions of interest) can be dynamically located and cropped from the original medical images. Then, a higher-resolution feature operator can be used, employing smaller patches or a denser sampling strategy. For instance, the first local region (patch) can be adjusted from 16x16 to 4x4, creating more second local regions. This allows for the precise extraction of subtle pathological features such as microcalcifications and edge spiculations, simulating the progressive diagnostic process of clinicians, moving "from a full scan to precise local examination." Alternatively, without changing the patch size, the sampling step size can be adjusted, for example, changing from sampling every 16 pixels to sampling every 4 pixels, thereby extracting richer features.

[0090] In some embodiments of this disclosure, the diagnostic grading subunit 143 is configured to: acquire global image contour features of medical images based on medical images; and perform diagnostic grading on medical images based on global image contour features, detail features, and multiple professional dimensions to obtain a diagnostic conclusion.

[0091] Global image contours focus on macroscopic perspectives, such as the overall geometry of the nodule (e.g., circular, elliptical), growth direction (aspect ratio), anatomical spatial relationship with surrounding capsules or tissues, and background sonograms. Detail features focus on microscopic details, such as fine burrs or lobes at the edges, microcalcifications with a diameter less than 1 mm inside, local echo attenuation or enhancement, and fine textural structures within the solid portion.

[0092] For example, global image contour features and detail features are input into the diagnostic grading subunit 143. The diagnostic grading subunit 143 is based on an attention mechanism and performs a comprehensive score according to the weights of multiple professional dimensions (such as "vertical growth" having a higher weight than "extremely low echo").

[0093] The comprehensive scoring is achieved through an attention mechanism. Different weight coefficients are assigned to different professional dimensions according to clinical diagnostic criteria (such as TI-RADS). The scores of the identified signs in each dimension are weighted and summed with the corresponding clinical weights. The final output is a diagnostic conclusion that includes a structured description of the signs and the corresponding confidence level (0-1) (e.g., TI-RADS level 3: 0.85).

[0094] In some embodiments of this disclosure, such as Figure 2 As shown, the collaborative inference unit also includes an adaptive subunit 163. The parameters of the adaptive subunit 163 include trainable adaptive parameters, which are loaded into the attention mechanism-related network layer of the image recognition device. These adaptive parameters are configured to adaptively adjust the image recognition device while keeping the backbone network parameters unchanged, adapting to the differences in image style produced by different imaging devices. For example, there are systematic differences in image style between different hospitals and different models of ultrasound machines, such as subtle differences in contrast and brightness. By setting an adaptive layer, the systematic differences caused by the devices can be eliminated. This adaptive subunit 163 can be updated based on the backpropagation algorithm during post-training, without relying on manual annotation. Please refer to the description below for post-training information. In some embodiments of this disclosure, the trainable adaptive parameters can be implemented using a low-rank adaptive (LoRA) structure, which efficiently fine-tunes the parameters of the pre-trained model by introducing low-rank incremental parameters. In other embodiments of this disclosure, the adaptive parameters can also be implemented using weight decomposition low-rank fine-tuning (DoRA), adaptive rank allocation (AdaLoRA), adapters, or other parameter-efficient fine-tuning structures.

[0095] like Figure 2 As shown, in some embodiments of this disclosure, the image recognition device 100 may further include a logic closed-loop verification unit 104.

[0096] The logic closed-loop verification unit 104 is configured to compare the medical diagnostic criteria corresponding to the diagnostic conclusion with the predicted result description to determine whether there is a logical conflict between the predicted result description and the diagnostic conclusion; and to perform feedback redirection in response to a logical conflict between the predicted result description and the diagnostic grading result. Feedback redirection includes: reducing the sampling step size, and then, based on the reduced sampling step size, the collaborative inference unit re-grades the medical image according to the predicted result description to obtain a diagnostic conclusion. The sampling step size is the period during which the collaborative inference unit samples multiple local regions to obtain local features.

[0097] By introducing a logical closed-loop verification unit, strong constraints are imposed on the logic, effectively eliminating the common low-level error of "contradictory description and conclusion" in generative AI.

[0098] For example, the logic closed-loop verification unit 104 constructs a set of constraint rules based on medical diagnostic standards, and performs consistency verification between the prediction result description and the diagnostic conclusion based on the set of constraint rules to determine whether there is a logical conflict. Reducing the sampling step size is, for example, to adjust the granularity of local region segmentation or the feature extraction resolution.

[0099] For example, the final output TI-RADS rating (e.g., TI-RADS level 5) and the initial structured text are input into the logic loop closure verification unit. If the initial structured text states "extremely clear boundaries, uniform internal echoes," but the rating output is "TI-RADS level 5 (high risk)," the logic loop closure verification unit will report a significant contradiction. For example, if... Figure 2 As shown, if the logic closed-loop verification unit has no conflict, a structured diagnostic report is output. The structured diagnostic report may include a description of the prediction results and diagnostic conclusions.

[0100] Once a contradiction or uncertainty score is detected to exceed a preset threshold, the system will automatically increase the sampling weight (i.e., decrease the sampling step size), redirect the feedback, and force the system to rethink the diagnostic logic.

[0101] Feedback redirection refers to reducing the sampling step size. Based on the reduced sampling step size, the collaborative inference unit reuses the matching subunit, analysis subunit, detail extraction subunit, and diagnostic grading subunit to re-describe the medical image according to the prediction results and obtain a diagnostic conclusion.

[0102] In some embodiments of this disclosure, the fusion perception unit, feature recognition unit, and collaborative reasoning unit include a deep learning model with a self-attention mechanism. The parameters of the deep learning model include backbone network parameters and adaptive network parameters, and the adaptive network parameters include parameters of adaptive subunits. The image recognition device also includes a post-training unit configured to update the adaptive network parameters while keeping the backbone network parameters unchanged.

[0103] For example, the fusion of the perception unit, feature recognition unit, and collaborative reasoning unit each includes a deep learning model with a self-attention mechanism, which performs recognition and prediction through the deep learning model. A deep learning model with a self-attention mechanism typically includes an encoder and a decoder. The encoder is used to understand the input and extract global contextual features; the decoder is used to generate the output and produce the prediction result.

[0104] The image recognition apparatus of some embodiments of this disclosure employs a collaborative training strategy of "backbone network parameter freezing, projection layer alignment, and adaptive low-rank (LoRA) fine-tuning". In some embodiments of this disclosure, the post-training unit updates and freezes the parameters of each module during the post-training phase.

[0105] During post-training, the image recognition device strictly divides the parameters of the deep learning model into two parts: "explicit freezing" and "dynamic updating," in order to achieve task adaptation with low computational consumption while preserving general medical knowledge.

[0106] Explicitly frozen parameter structure refers to parameters that do not participate in post-training. During the entire training process, the system explicitly freezes all weights of the pre-trained backbone network, and its gradients are not backpropagated.

[0107] a) Image encoder backbone parameters: corresponding to the image encoder (ViT) backbone weights in the fusion sensing unit, maintaining its original image feature extraction capability.

[0108] b) Text encoder backbone parameters: corresponding to the text encoder backbone weights in the fusion sensing unit.

[0109] c) Main parameters of the large language model: corresponding to the generation of all candidate feature options in the feature recognition unit and the native pre-trained weights of the large model decoder (Transformer Decoder) in the collaborative reasoning unit, used to lock in general medical common sense, as the physical basis for triggering general medical semantic reasoning ability when facing unknown organs.

[0110] The adaptive network parameters that are dynamically updated and fine-tuned refer to the set of parameters that participate in the training. The specific structure and logic of parameter updates are as follows.

[0111] According to embodiments of this disclosure, the adaptive network parameters include the parameters of the adaptive subunit. Please refer to the description above for the parameters of the adaptive subunit. Since the parameter updates of this adaptive subunit radiate and affect the parameter distribution of the entire model, its core function is to adapt to the system differences of different medical instruments, such as differences in brightness and contrast, ensuring that the model maintains stable feature extraction and generation capabilities even when facing diverse hardware acquisition conditions and underlying data distributions. The parameters of the adaptive subunit can be updated through fine-tuning, using a small learning rate to make small, localized corrections to the parameters, keeping the overall weight changes gradual.

[0112] According to embodiments of this disclosure, the adaptive network parameters may further include parameters for the structured sign prediction classification layer and the feature projection layer. The structured sign prediction classification layer (i.e., a linear classification head with a weight matrix dimension of [D, T]) undergoes full gradient updates during training to ensure the model can fully learn the accurate mapping relationship from the image semantic space to specific clinical sign categories, compensating for insufficient coverage in specific classification tasks during pre-training. The feature projection layer parameters (fully trained) correspond to all weights and biases of the visual projection matrix (e.g., composed of a 2-layer perceptron MLP) and the text projection matrix (e.g., composed of a 1-layer linear mapping layer), which are connected to the back ends of the two encoders in the fusion perception unit and used to perform "feature fusion." Updating this part aims to optimize the fusion expression effect of multimodal features in specific inference scenarios. These two parts can be fully updated, i.e., the parameters of the structured sign prediction classification layer and the feature projection layer are significantly calculated and iterated. Therefore, in cases where the model's pre-training phase does not cover medical images or does not fully cover the actual clinical inference scenarios in specific medical fields, the difference between pre-training and real inference scenarios can be effectively compensated.

[0113] In some embodiments, the complete update process can begin by updating the parameters of the structured sign prediction classification layer and the feature projection layer, enabling the model to quickly adapt to the semantic mapping and multimodal feature fusion requirements of specific inference scenarios. Subsequently, the parameters of the adaptive sub-unit are updated to further eliminate underlying differences caused by different instrument systems. This update sequence, proceeding from the surface to the core and prioritizing scenario adaptation before system alignment, can maximize the improvement of the model's global performance and clinical robustness.

[0114] Combination Figure 2 The unit cascade relationship is used, and the overall training process is achieved through the following loss calculation and gradient backpropagation.

[0115] For multimodal alignment, forward propagation is performed first, followed by loss calculation and update. During forward propagation, the original features output from the image encoder and text encoder are input into the corresponding visual projection matrix and text projection matrix, respectively, and mapped to the same hidden layer dimension for feature concatenation and fusion. During loss calculation and update, the multimodal contrast loss (InfoNCE Loss, formula below) is calculated at the output of the fusion perceptual unit. The backpropagation algorithm is used to completely update only the parameters of the visual projection matrix and text projection matrix, achieving deep alignment of cross-modal features in the joint embedding semantic space.

[0116] B represents the batch size, which is the total number of samples in the current training batch; v i t represents the image feature vector of the i-th sample after being mapped by the visual projection matrix; i t j sim(v) represents the text feature vectors of the i-th and j-th samples after being mapped by the text projection matrix; i , t i ) represents the similarity calculation operator, which can be cosine similarity, mathematically expressed as the dot product of two vectors; τ (Temperature) represents a constant used to adjust and control the degree of concentration of the overall probability distribution.

[0117] For Low Rank Parameter Fine-Tuning (LoRA), during the forward propagation phase, the large model decoder receives the fused features after multimodal spatial alignment and performs two-stage collaborative inference.

[0118] The first stage completes the classification of the anatomical location corresponding to the current input medical image and outputs the corresponding organ type identifier, which is used to determine the medical scanning task type to which the current image belongs.

[0119] The second stage, based on the identified organ type, dynamically loads the corresponding candidate feature group index table (Group_Index_List) from the system's predefined unified candidate feature group index table, and outputs the full candidate feature score matrix through the structured feature prediction classification layer: CLS_Logits∈[B,T], where B represents the batch size and T represents the total number of unified candidate feature group indices in the system.

[0120] Based on the Group_Index_List of the current task, the predicted scores of features at corresponding index positions in CLS_Logits are dynamically grouped and extracted. Then, Softmax probability normalization is independently performed on each candidate feature group to obtain the conditional probability distribution within each clinical feature dimension, and the corresponding candidate sign prediction results are output. Different organ tasks share a unified candidate feature grouping index space and prediction head parameters. The conditional probability space partitioning between different organ tasks is achieved only by dynamically switching the candidate feature grouping index table, thereby improving the model's cross-organ transferability and parameter reuse efficiency.

[0121] In the joint loss calculation stage, a hierarchical joint loss function was designed: , This represents the overall joint optimization loss of the system; This indicates the loss in organ type classification during the first stage; Let K represent the conditional probability prediction loss corresponding to the k-th candidate feature group; K represents the number of candidate feature groups corresponding to the current organ task. and This is the loss weight hyperparameter.

[0122] Organ classification loss Cross-entropy loss is used to supervise the system in identifying anatomical sites from the current input image. Loss is calculated for each candidate feature group. The cross-entropy loss is calculated using the within-group conditional probability method. , This represents the number of candidate categories in the k-th candidate feature group; This represents the actual label for the corresponding category; This represents the conditional probability output of the current candidate feature group after Softmax normalization.

[0123] Then, gradient updates are performed using the backpropagation algorithm and the joint loss function. The calculated gradients are only fine-tuned for the parameters of the parallel low-rank adaptive matrix (LoRA) in the large model, while the parameters of the remaining pre-trained backbone network remain frozen. Simultaneously, the parameters of the structured feature prediction classification layer and the multimodal feature projection layer participate in the complete gradient update, achieving synergistic optimization of cross-modal semantic alignment capabilities and medical structured feature prediction capabilities.

[0124] During post-training, a test-time self-evolution mechanism can also be implemented to update the adaptive sub-units in the collaborative reasoning unit.

[0125] like Figure 2As shown, the image recognition device may also include a post-training unit 105, which can perform a self-evolution mechanism during testing.

[0126] The self-evolution mechanism can include: during clinical testing, multiple forward propagation runs are performed continuously on unlabeled input data; the logic closed-loop verification unit performs consistency checks on the initial structured output report, autonomously selecting high-deterministic reasoning cases with completely consistent logic verification (no conflicts) and uncertainty scores below a preset threshold, and using these as pseudo-labels. These pseudo-labels are then substituted into the loss function, and backpropagation updates only the gradients of the adaptation layer (i.e., LoRA matrix parameters) in the adaptive subunit in real time. This allows the model to autonomously adapt to image style differences (such as subtle differences in contrast and brightness) between different hospitals and ultrasound machines without relying on manual annotation. Without increasing the cost of manual annotation, the autonomous update technology during testing solves the problems of equipment compatibility and departmental style differences.

[0127] The functions of the fusion sensing unit and the feature recognition unit in the image recognition device provided in this disclosure will be further explained below with reference to embodiments.

[0128] For example, in a medical imaging study, a thyroid ultrasound image is used. The ultrasound cross-sectional image of the thyroid nodule and clinical metadata (e.g., female patient, 45 years old, thyroid mass discovered during a physical examination) are input into a fusion sensing unit. The image is encoded by an image encoder to obtain raw visual features, and the text is encoded by a text encoder to obtain raw text features. Both outputs are mapped to feature vectors of the same dimension via their respective visual and text projection matrices, and then concatenated to form a fused feature representation Z. thyroid At this point, because the fused feature representation contains a deep semantic vector about "thyroid" from the clinical metadata, the attention mechanism of the large model implicitly senses and locks the current anatomical target being processed as the thyroid task when it receives the fused feature.

[0129] The feature recognition unit parses clinical metadata to obtain anatomical site entity labels generated by the large model prefix, and the system confirms that the current scanning site is the thyroid gland. By querying the mapping table in the backend, it activates the thyroid-specific N... thyroid There are several candidate features. For example, three professional dimensions were activated: marginal morphology, echo level, and focal calcification (this is just an example; in practice, there may be more or fewer than three, and these medical professional dimensions can be configured).

[0130] As described above, the image recognition device can pre-construct a discrete unified candidate feature group index table, whose spatial dimension is denoted as T. Examples of thyroid candidate feature groups corresponding to the current input are: i. Edge morphology: Channels 0 to 2: [clear edge, unclear edge, angular edge]; ii. Echo level: Channels 3 to 6: [hyperecho, isoecho, hypoecho, very hypoecho]; iii. Focal calcification: Channels 7 to 8: [focal calcification present, no focal calcification].

[0131] Based on multiple candidate feature groups, the predicted scores of candidate features at corresponding index positions in the full candidate feature score matrix CLS_Logits are extracted in groups. Candidate feature groups are denoted as Group_1_List, Group_2_List, Group_3_List, ..., Group_Index_List, where Index is a positive integer. For example, for the input of the current scene: Group_1_Logits = CLS_Logits[:, [0,1,2]]; Group_2_Logits = CLS_Logits[:, [3,4,5,6]]; Group_3_Logits = CLS_Logits[:, [7,8]].

[0132] Next, a structured feature prediction classification layer is introduced in parallel at the output of the last layer of the multimodal large model decoder. This structured feature prediction classification layer, as an independent classification layer, shares backbone features with the original vocabulary output of the large model, and its internal linear mapping matrix is ​​set to dimensions [D,T]. The hidden layer feature vector Z of the last layer output of the large model is then obtained. raw Its dimensions are [B, L, D] (where L is the length of the output token sequence). First, the sequence dimension is compressed to Z using mean pooling. thyroid ∈[B,D], where B is the batch size and D is the hidden layer feature dimension. Then the feature tensor Z is... thyroidA linear mapping operation is performed with the structured feature prediction classification layer to obtain the full candidate feature score matrix CLS_Logits∈[B,T]([B,D]x[D,T]), which contains the prediction scores of the current input corresponding to the grouping index of all candidate features. Then, according to Group_Index_List, the prediction scores of the features at the corresponding index positions in CLS_Logits are extracted in groups. For example, for the input of the current scene: Group_1_Logits = CLS_Logits[:, [0,1,2]]; Group_2_Logits = CLS_Logits[:, [3,4,5,6]]; Group_3_Logits = CLS_Logits[:, [7,8]].

[0133] The predicted scores of each group of candidate features are independently calculated using Softmax probability normalization to obtain the conditional probability distribution within each clinical feature dimension. After Softmax activation, the probability matrix undergoes Argmax extraction and text decoding by a word segmenter. Pre-scoring results for candidate features related to the current thyroid condition are concurrently output for each feature. For example: Marginal morphology: [Clear: 0.12, Unclear: 0.85, Angular: 0.03], Output: Unclear (Pre-scoring 0.85); Echo level: [Hyperecho: 0.05, Isoecho: 0.05, Hypoecho: 0.2, Very Hypoecho: 0.7], Output: Very Hypoecho (Pre-scoring 0.7); Focal calcification: [Present: 0.2, Absent: 0.8], Output: Absent (Pre-scoring 0.8).

[0134] For example, when the next patient enters the examination process, or when the same patient undergoes an additional breast ultrasound examination, there is no need to modify the backbone network structure of the large model. Adaptive responses to different organ tasks can be achieved simply by dynamically switching the configuration of multiple candidate feature groups. The execution flow is as follows: Step S1: The fusion sensing unit performs multimodal input encoding and context fusion. That is, the complete input data includes: a) medical images: breast ultrasound scan images; b) clinical metadata (e.g., patient female, 38 years old, breast BI-RADS screening...), followed by feature extraction and feature concatenation to generate a new fusion feature representation Z. breast .

[0135] Step S2: Description of prediction results for each of the multiple candidate feature groups guided by clinical features 1. Organ type recognition: The system analyzes the current input and confirms that the scanned organ has been switched to the breast.

[0136] 2. Load multiple candidate feature groups: Query the unified candidate feature grouping index table in the background and dynamically activate multiple candidate feature groups corresponding to the breast. For the current input as follows: mass morphology: [oval, round, irregular]; posterior acoustic features: [no change, enhancement, attenuation, lateral acoustic shadowing]; microcalcifications: [present, absent] (only examples, not representing the actual complete number).

[0137] 3. Constructing the candidate feature group index table: The system dynamically generates the corresponding candidate feature group index table Group_Index_List based on the current organ type Breast. Its structure is shown in the following example: [[12,13,14], [15,16,17,18], [19,20]], where the above index positions correspond to the discrete feature channel numbers in the system's unified overall feature channel table.

[0138] 4. Parallel computation of grouped conditional probabilities: a) Large model decoder receives fused feature Z breas The full candidate feature score matrix CLS_Logits∈[B,T] is output by the structured feature prediction classification layer, where B is the batch size and T is the total number of candidate feature group indices in the system.

[0139] b) The system dynamically extracts the predicted scores of the features at the corresponding positions in CLS_Logits based on Group_Index_List: Group_1_Logits = CLS_Logits[:, [12,13,14]]; Group_2_Logits = CLS_Logits[:, [15,16,17,18]]; Group_3_Logits = CLS_Logits[:, [19,20]].

[0140] c) The system independently performs Softmax probability normalization calculations on each candidate feature group to obtain the conditional probability distribution within each clinical feature dimension.

[0141] 5. Output probability prediction results: a) Mass morphology [elliptical: 0.05, round: 0.0, irregular: 0.95], output the prediction with the highest probability: irregular (predicted score 0.95); b) Rear acoustics [No change: 0.10, Enhancement: 0.00, Attenuation: 0.90, Side shadow: 0.00], Output: Attenuation (pre-score 0.90); c) Microcalcifications [Yes: 0.24, No: 0.76], Output: No (Pre-score 0.76).

[0142] For example, when the input medical image has poor image quality or is a special image that the system cannot determine which organ it belongs to, it will be marked, for example, as "General / Unknown". In this case, the image recognition device dynamically activates a set of "general medical description candidate features" and loads a general high-frequency medical vocabulary for ultrasound image diagnosis (e.g., basic physical feature descriptions applicable to all organs, such as [nodule / mass, cystic, solid, clear borders, poorly clear borders, rich blood flow, no obvious abnormalities]). Then, the probability of these most basic physical signs is calculated in parallel by a large model to obtain multiple candidate feature groups. Although there is no precise candidate feature prediction result for a specific organ, it avoids the image processing device getting stuck when encountering such input and can provide the necessary information for subsequent coordinated inference units to locate the image.

[0143] Furthermore, if the organ type is unknown, and the image recognition device cannot perform targeted hierarchical logic verification (e.g., it cannot verify whether it meets the TI-RADS 5 criteria), then the prediction result description can be directly output: "Unknown area image detected. The system has initiated a general physical sign assessment. Please combine clinical findings for the final diagnosis." The case will then be marked as a "high uncertainty case," prompting a human doctor to intervene for review, or it can be included in a self-evolving dataset for future automatic learning.

[0144] The embodiments of this disclosure design an adaptive structured prediction mechanism and a general fallback strategy based on dynamic candidate feature grouping index in an image recognition device. When encountering unknown image locations with low confidence, the system can dynamically degenerate into a general medical physical sign description mode. By releasing the zero-sample generalization capability of the large model, the initialization and localization of basic sonographic features are completed. This avoids abnormal system blockage caused by rigid rules and ensures robustness to rare cases and complex multi-organ combined images. Furthermore, compared to the traditional approach of building independent classifiers for different organs, the embodiments of this disclosure achieve parameter sharing and semantic transfer between cross-organ tasks through a unified candidate feature grouping index space and a dynamic grouping index mechanism, reducing the system parameter scale and training complexity.

[0145] For example, the fusion sensing unit 101, the feature recognition unit 102, and the collaborative reasoning unit 103 can be hardware, software, firmware, or any feasible combination thereof. For example, the fusion sensing unit 101, the feature recognition unit 102, and the collaborative reasoning unit 103 can be dedicated or general-purpose circuits, chips, or devices, or they can be a combination of a processor and memory. The embodiments of this disclosure do not limit the specific implementation of the above-mentioned units.

[0146] The components and structures of the image recognition device 100 shown in the embodiments of this disclosure are merely exemplary and not limiting. The image recognition device 100 may also include other components and structures as needed.

[0147] At least one embodiment of this disclosure also provides an electronic device including the image recognition apparatus described in any one of the present disclosures. This electronic device is capable of recognizing medical images based on clinical features, making the diagnostic process transparent and consistent with the doctor's thinking, and improving the processing performance of medical images with blurred boundaries, thereby reducing the occurrence of hallucinations.

[0148] Figure 4 This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. For example... Figure 4 As shown, the electronic device 1200 includes a processor 1210 and a memory 1220. The memory 1220 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 1210 is used to execute the non-transitory computer-readable instructions, which, when executed by the processor 1210, can perform one or more functions corresponding to the image recognition device described above. The memory 1220 and the processor 1210 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0149] For example, processor 1210 may be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing and / or program execution capabilities. For example, the central processing unit (CPU) may be an x86 or ARM architecture. Processor 1210 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 1200 to perform desired functions.

[0150] For example, memory 1220 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 1210 may run one or more computer program modules to implement various functions of electronic device 1200. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0151] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 1200 can be referred to the description of the construction method or output prediction method above, and will not be repeated here.

[0152] Figure 5 This is a schematic block diagram of another electronic device provided in some embodiments of the present disclosure. The electronic device 1300 is, for example, adapted to perform the functions of the image recognition apparatus provided in the embodiments of the present disclosure, namely, acquiring a fused feature representation of medical images and metadata, wherein the metadata is clinical information related to the medical images; determining multiple candidate feature groups from a candidate feature set based on the medical images or the metadata, wherein the multiple candidate feature groups correspond one-to-one with multiple professional dimensions; classifying the fused feature representation for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups; and performing a diagnostic grading of the medical images based on the prediction result description to obtain a diagnostic conclusion. It should be noted that... Figure 5 The illustrated electronic device 1300 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0153] like Figure 5 As shown, electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1310, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1320 or a program loaded from storage device 1380 into random access memory (RAM) 1330. RAM 1330 also stores various programs and data required for the operation of electronic device 1300. Processing device 1310, ROM 1320, and RAM 1330 are interconnected via bus 1340. Input / output (I / O) interface 1350 is also connected to bus 1340.

[0154] Typically, the following devices can be connected to I / O interface 1350: input devices 1360 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1370 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1380 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1390. Communication device 1390 allows electronic device 1300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 5 An electronic device 1300 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 1300 may alternatively implement or have more or fewer devices.

[0155] For example, according to embodiments of this disclosure, the image recognition device described above can be implemented as a computer software program. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the functions performed by the image recognition device described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1390, or installed from a storage device 1380, or installed from a ROM 1320. When the computer program is executed by the processing device 1310, it can implement the functions defined in the construction method or output prediction method provided in embodiments of this disclosure.

[0156] At least one embodiment of this disclosure also provides a computer-readable storage medium for storing non-transitory computer-readable instructions. When executed by a computer, these instructions can: acquire a fused feature representation of a medical image and metadata, wherein the metadata is clinical information related to the medical image; determine multiple candidate feature groups from a candidate feature set based on the medical image or the metadata, wherein each candidate feature group corresponds one-to-one with multiple professional dimensions; classify the fused feature representation for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups; and perform diagnostic grading on the medical image based on the prediction result description to obtain a diagnostic conclusion. Using this computer-readable storage medium to identify medical images based on clinical features makes the diagnostic process transparent and consistent with physician thinking, improves the processing performance of medical images with blurred boundaries, and reduces the occurrence of hallucinations.

[0157] Figure 6 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example... Figure 6 As shown, storage medium 1400 is used to store non-transitory computer-readable instructions 1410. For example, when non-transitory computer-readable instructions 1410 are executed by a computer, they can perform the following: acquiring a fusion feature representation of a medical image and metadata, wherein the metadata is clinical information related to the medical image; determining multiple candidate feature groups from a candidate feature set based on the medical image or the metadata, wherein the multiple candidate feature groups correspond one-to-one with multiple professional dimensions; and classifying the fusion feature representation for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups; and performing a diagnostic grading on the medical image based on the prediction result description to obtain a diagnostic conclusion.

[0158] For example, the storage medium 1400 can be used in the aforementioned electronic device 1200. For example, the storage medium 1400 can be... Figure 4The memory 1220 in the illustrated electronic device 1200. For example, a description of the storage medium 1400 can be found here. Figure 4 The corresponding description of the memory 1220 in the illustrated electronic device 1200 will not be repeated here.

[0159] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An image recognition device, comprising: The system integrates a perception unit, a feature recognition unit, and a collaborative reasoning unit, among which... The fusion sensing unit is configured to acquire a fusion feature representation of medical images and metadata, wherein the metadata is clinical information related to the medical images; The feature recognition unit is configured to determine multiple candidate feature groups from a candidate feature set based on the medical image or the metadata, wherein each candidate feature group corresponds one-to-one with multiple professional dimensions; and to perform classification processing on the fused feature expression for each of the multiple professional dimensions to obtain a prediction result description for each of the multiple candidate feature groups. The collaborative reasoning unit is configured to perform diagnostic grading on the medical image based on the prediction results to obtain a diagnostic conclusion. The collaborative reasoning unit includes: The matching subunit is configured to divide the medical image into multiple first local regions, and match the local features of the multiple first local regions with the prediction result description respectively to obtain the matching degree between the multiple first local regions and the prediction result description; The analysis subunit is configured to determine the region of interest and uncertainty score in the medical image based on the matching degree. The detail extraction subunit is configured to extract detail features from the region of interest in response to the uncertainty score meeting a preset condition; and The diagnostic grading subunit is configured to perform diagnostic grading on the medical image based on the medical image, the detailed features, and the multiple professional dimensions to obtain a diagnostic conclusion.

2. The image recognition device according to claim 1, wherein, The analysis subunit includes: A heatmap generation subunit is configured to generate a heatmap based on the matching degree and according to the original physical locations of the plurality of first local regions in the medical image; The region of interest (ROI) determination subunit is configured to perform threshold segmentation and connected component analysis based on the spatial response value distribution of the heatmap to determine the ROI, wherein the spatial response values ​​are suppressed based on a first threshold, and connected component aggregation is performed on the spatial response values ​​based on a second threshold; and The uncertainty scoring subunit is configured to determine the uncertainty score based on the spatial statistical characteristics of the heatmap. The spatial statistical features are calculated based on the normalized spatial response value distribution of the heatmap; the spatial statistical features include one or more of peak intensity, spatial entropy, and spatial variance; the uncertainty score is obtained by weighted combination of the spatial statistical features.

3. The image recognition device according to claim 1, further comprising: The logical closed-loop verification unit is configured to compare the medical diagnostic criteria corresponding to the diagnostic conclusion with the prediction result description to determine whether the prediction result description and the diagnostic conclusion are logically conflicting. In response to a logical conflict between the prediction result description and the diagnostic grading result, feedback redirection is performed. The feedback redirection includes: The sampling step size is reduced, and the collaborative inference unit, based on the reduced sampling step size, re-describes the prediction results to obtain a diagnostic conclusion by grading the medical image. The sampling step size is the step size used by the collaborative inference unit when sampling the local features of the multiple local regions respectively.

4. The image recognition device according to claim 3, wherein, The analysis subunit is also configured to: In response to the uncertainty score being higher than a third threshold, the medical image is labeled, and the feedback redirection is performed.

5. The image recognition device according to claim 1, wherein, The collaborative reasoning unit also includes: The output subunit is configured to output preliminary structured text based on the prediction results and to output the diagnostic conclusion.

6. The image recognition device according to claim 1, wherein, The collaborative reasoning unit also includes an adaptive subunit. The parameters of the adaptive subunit include trainable adaptive parameters, which are loaded into the attention mechanism-related network layer of the image recognition device and configured to adaptively adjust the image recognition device while keeping the backbone network parameters unchanged, so as to adapt to the differences in image style produced by different imaging devices.

7. The image recognition device according to claim 1, wherein, The detail extraction subunit is configured as follows: In response to the uncertainty score satisfying the preset condition, a high-resolution feature operator is used to extract features from the region of interest to obtain the detailed features. The high-resolution feature operator refers to dividing the region of interest into multiple second local regions, the size of which is smaller than that of the first local region, or reducing the sampling step size of the collaborative inference unit in sampling the multiple first local regions to obtain the local features.

8. The image recognition device according to claim 1, wherein, The diagnostic grading subunit is configured as follows: Based on the medical image, obtain the global image contour features of the medical image; Based on the global image contour features, the detailed features, and the multiple professional dimensions, the medical image is diagnosed and graded to obtain a diagnostic conclusion.

9. The image recognition device according to claim 1, wherein, The aforementioned professional dimensions include at least one of the following: echo level, edge morphology, aspect ratio, calcification features / microcalcifications, growth pattern, and internal structure. The candidate feature group corresponding to the echo level includes at least two of the following: extremely low echo, low echo, isoecho, no echo, uniform internal echo, and non-uniform echo. The candidate feature group corresponding to the edge morphology includes at least two of the following: unclear edge, clear boundary, irregular edge, lobed, and burr; The candidate feature groups corresponding to the aspect ratio / length-width ratio include: aspect ratio > 1 and aspect ratio < 1; The candidate feature group corresponding to the calcification features / microcalcifications includes at least two of the following: presence of microcalcifications, coarse calcifications, ring calcifications, and absence of calcifications; The candidate feature groups corresponding to the growth methods include: vertical position growth and parallel position growth; The candidate feature groups corresponding to the internal structure include at least two of the following: solid, cystic-solid, cystic, and spongy.

10. The image recognition device according to claim 6, wherein, The fusion perception unit, feature recognition unit, and collaborative reasoning unit include a deep learning model with a self-attention mechanism. The parameters of the deep learning model include backbone network parameters and adaptive network parameters. The adaptive network parameters include the parameters of the adaptive subunits. The image recognition device further includes a post-training unit configured to update the adaptive network parameters while keeping the backbone network parameters unchanged.

11. The image recognition device according to claim 10, wherein, The fusion sensing unit includes: An image encoder configured to convert the medical image into a high-dimensional spatial vector; A text encoder configured to convert the metadata into a semantic vector; and The fusion subunit is configured to map the high-dimensional spatial vector and the semantic vector to feature vectors of the same dimension via visual projection matrix and text projection matrix, respectively, and then perform feature concatenation and fusion to obtain the fused feature representation. The backbone network parameters include the parameters in the image encoder and the parameters in the text encoder, and the adaptive network parameters also include the parameters of the visual projection matrix and the parameters of the text projection matrix.

12. The image recognition device according to claim 10, wherein, The feature recognition unit includes: The structured feature prediction classification layer is configured to perform a linear mapping operation on the fused feature representation to obtain a full candidate feature score matrix; The candidate feature group extraction subunit is configured to extract the scores of each channel in the multiple candidate feature groups from the full candidate feature score matrix; The prediction result description generation subunit is configured to obtain the prediction result description based on the scores of each channel in the plurality of candidate feature groups. The adaptive network parameters also include the parameters of the structured feature prediction classification layer.

13. The image recognition device according to claim 10, wherein, The post-training unit is configured as follows: Filter out target data from unlabeled data whose diagnostic confidence is greater than a preset threshold; Calculate the prediction result of the target data, and calculate the loss function between the prediction result and the diagnostic conclusion; as well as The parameters of the adaptive network are updated using the backpropagation algorithm to make the loss function converge.

14. An electronic device comprising: The image recognition device according to any one of claims 1-13.

15. A computer-readable storage medium configured to store non-transitory computer-readable instructions that, when executed by a computer, implement: Obtain the fusion feature representation of medical images and metadata, where, The metadata is clinical information related to the medical images; Based on the medical image or the metadata, multiple candidate feature groups are determined from the candidate feature set, and the multiple candidate feature groups correspond one-to-one with multiple professional dimensions. For each of the multiple professional dimensions, the fused feature expression is classified to obtain the prediction result description of each of the multiple candidate feature groups; as well as Based on the prediction results, the medical images are classified and a diagnostic conclusion is obtained.

16. A computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program being used to execute: Obtain the fusion feature representation of medical images and metadata, where, The metadata is clinical information related to the medical images; Based on the medical image or the metadata, multiple candidate feature groups are determined from the candidate feature set, and the multiple candidate feature groups correspond one-to-one with multiple professional dimensions. For each of the multiple professional dimensions, the fused feature expression is classified to obtain the prediction result description of each of the multiple candidate feature groups; as well as Based on the prediction results, the medical images are classified and a diagnostic conclusion is obtained.

Images