Method for training a recognition model for recognition of image regions and related products

Abstract

The application discloses a method for training an identification model for image part identification and related products. The method comprises: obtaining multi-modal image data labeled with image parts; using a feature extraction module to extract features and obtain an initial feature vector; using an auxiliary model containing double encoding of image modalities and image parts to perform auxiliary encoding based on the multi-modal image data and obtain adaptive incremental parameters; adding the adaptive incremental parameters to shared invariant parameters of a feature encoding module, using the feature encoding module to perform feature encoding on the initial feature vector, and obtaining an intermediate feature vector; inputting the intermediate feature vector into a feature decision module for feature decision, and based on image part label and identification label contrast learning, the identification model outputs the image part to train the identification model for image part identification. The scheme of the application can realize automatic and high-precision consistency determination of image parts and labels.

Description

Technical Field

[0001] This application generally relates to the field of image processing technology. More specifically, this application relates to a method for training a recognition model for image part recognition, an apparatus for image part recognition, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the construction of imaging cloud platforms in various regions, massive amounts of radiological image data such as CT, MR, and DR are collected, stored, and shared for use, providing data support for cross-institutional diagnosis and treatment.

[0003] Current radiological image quality control mainly relies on manual sampling and visual review. However, the massive amount of image data makes manual methods insufficient to meet the requirements for real-time performance and accuracy. Furthermore, due to differences in equipment among medical institutions, varying operator skill levels, and inconsistent scanning protocols, radiological images commonly suffer from quality issues such as mismatch between image location and label, and insufficient scanning range. The accuracy of image location directly impacts clinicians' diagnostic and treatment decisions and determines whether images can be effectively shared and reused across different institutions. Current technologies lack intelligent solutions that can automatically identify the consistency between image location and label, exhibiting weak adaptability across modalities, devices, and scanning protocols, and insufficient recognition accuracy and generalization ability.

[0004] In view of this, there is an urgent need to provide a scheme for training an image part recognition model. Through the collaborative design of dual-encoding auxiliary model, adaptive parameter superposition, and comparative supervised learning, the system can achieve automated and high-precision consistency determination of image parts and labels, improve quality control efficiency and cross-scenario adaptability, and provide reliable technical support for medical image mutual recognition. Summary of the Invention

[0005] In order to at least solve one or more of the technical problems mentioned above, this application proposes a scheme for training an image part recognition model in several aspects.

[0006] In a first aspect, this application provides a method for training a recognition model for image part recognition, wherein the recognition model includes a feature extraction module, a feature encoding module, and a feature decision module, and the method includes: acquiring multimodal image data labeled with image parts; performing feature extraction using the feature extraction module based on the multimodal image data to obtain an initial feature vector; performing auxiliary encoding using an auxiliary model that includes dual encoding of image modality and image part based on the multimodal image data to obtain adaptive incremental parameters; superimposing the adaptive incremental parameters onto the shared invariant parameters of the feature encoding module, and using the feature encoding module to perform feature encoding on the initial feature vector to obtain an intermediate feature vector; inputting the intermediate feature vector into the feature decision module for feature decision, and performing comparative learning based on image part labels and recognition labels, so that the recognition model outputs image parts, thereby training the recognition model for image part recognition.

[0007] In some embodiments, prior to feature extraction using the feature extraction module based on the multimodal image data, the method further includes: performing a preprocessing operation on the multimodal image data, wherein the preprocessing operation includes at least one or more of normalization, resampling, or cropping.

[0008] In some embodiments, the multimodal image data includes at least two or more of CT images, MR images, and DR images.

[0009] In some embodiments, the feature encoding module includes at least one or more attention layers and feedforward network layers, and the shared invariant parameters include a first parameter of the attention layer and a second parameter of the feedforward network layer.

[0010] In some embodiments, the adaptive incremental parameter includes a third parameter corresponding to the attention layer and a fourth parameter corresponding to the feedforward network layer. The adaptive incremental parameter is superimposed on the shared invariant parameter of the feature encoding module by the following operation: superimposing the third parameter with the first parameter, and superimposing the fourth parameter with the second parameter.

[0011] In some embodiments, the auxiliary model includes a perceptron, an image part encoder, and an image modality encoder. Based on the multimodal image data, auxiliary encoding is performed using an auxiliary model that includes both image modality and image part encoding to obtain adaptive incremental parameters. This includes: performing corresponding encoding using the perceptron, the image part encoder, and the image modality encoder respectively based on the multimodal image data; and obtaining the adaptive incremental parameters based on the weighted average result of each encoding.

[0012] In some embodiments, the recognition model includes the ViT model.

[0013] In a second aspect, this application provides an apparatus for image part recognition, comprising: a processor; and a memory storing computer instructions for image part recognition, wherein when the computer instructions are executed by the processor, the following operations are performed: acquiring image data to be recognized; inputting the image data into a recognition model trained according to the method described in the first aspect above for image part recognition, thereby obtaining the recognition result of the image part.

[0014] In a third aspect, this application provides an electronic device, including: a processor; and a memory having computer instructions stored thereon for training a recognition model for image part recognition, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed; or having computer instructions stored thereon for image part recognition, wherein when the computer instructions are executed by the processor, the operation performed by the device described in the second aspect is performed.

[0015] In a fourth aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions for training a recognition model for image part recognition, which, when executed by one or more processors, cause the method described in the first aspect to be implemented; or having stored thereon computer program instructions for image part recognition, which, when executed by one or more processors, cause the operation performed by the apparatus described in the second aspect to be implemented.

[0016] The training scheme for the image part recognition model provided above, in this embodiment, involves acquiring multimodal image data labeled with image parts. After obtaining initial feature vectors through a feature extraction module, an auxiliary model with dual encoding of image modality and image part generates adaptive incremental parameters. These parameters are then superimposed with shared invariant parameters from the feature encoding module to form dynamic adaptation weights. Model training is completed through feature encoding, feature decision-making, and comparative learning between image part labels and recognition labels. The shared invariant parameters ensure the general feature capture capability across multiple modalities and parts, the adaptive incremental parameters accurately adapt to specific scene differences, and the comparative learning enhances the recognizability of part features. This significantly improves the real-time performance, reliability, and cross-scene adaptability of radiological image quality control, providing core technical support for image mutual recognition. Attached Figure Description

[0017] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0018] Figure 1 This is an exemplary flowchart illustrating a method 100 for training a recognition model for image part recognition according to an embodiment of this application;

[0019] Figure 2 This is an exemplary flowchart illustrating the overall training of a recognition model for image part recognition according to an embodiment of this application;

[0020] Figure 3 This is an exemplary schematic diagram illustrating the superposition of parameters according to an embodiment of this application;

[0021] Figure 4 This is an exemplary structural block diagram illustrating an apparatus 400 for image part recognition according to an embodiment of this application;

[0022] Figure 5 An exemplary structural block diagram of an electronic device 500 according to an embodiment of this application is shown. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be understood that the terms "comprising" and "including" used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] Figure 1 This is an exemplary flowchart illustrating a method 100 for training a recognition model for image region recognition according to an embodiment of this application. In some embodiments, the recognition model may include a feature extraction module, a feature encoding module, and a feature decision module. In some embodiments, the recognition model may be, for example, a ViT model.

[0029] like Figure 1 As shown, the method 100 may include: Step S101: acquiring multimodal image data with labeled image parts; Step S102: extracting features using a feature extraction module based on the multimodal image data to obtain an initial feature vector; Step S103: performing auxiliary encoding using an auxiliary model that includes dual encoding of image modality and image part based on the multimodal image data to obtain adaptive incremental parameters; Step S104: superimposing the adaptive incremental parameters into the shared invariant parameters of the feature encoding module, and using the feature encoding module to perform feature encoding on the initial feature vector to obtain an intermediate feature vector; Step S105: inputting the intermediate feature vector into a feature decision module for feature decision-making, and learning based on the image part label and the recognition label, so that the recognition model outputs the image part, thereby training the recognition model for image part recognition.

[0030] First, in step S101, multimodal image data of the labeled image regions is acquired. In some embodiments, the multimodal image data may include at least two or more of CT images, MR images, and DR images. The aforementioned multimodal image data can cover common clinical radiological imaging types, such as head CT, chest MR, and lung DR, ensuring that the model can learn the features of the regions under different imaging principles, laying a data foundation for cross-modal adaptation. In the implementation scenario, each image data contains complete annotation information, i.e., the ground truth value of the image region, such as brain, chest, and abdomen. To further improve the model's accuracy, information such as the patient's examination type (e.g., plain scan, enhanced scan) and scanning machine type (e.g., 1.5TMR, 64-slice CT) can be used as auxiliary information and input into the model along with the image data for training, helping the model learn the feature offset rules related to the equipment.

[0031] Next, in step S102, feature extraction is performed using the feature extraction module based on the multimodal image data to obtain an initial feature vector. In some embodiments, before performing feature extraction using the feature extraction module based on the multimodal image data, the method may further include: preprocessing the multimodal image data, wherein the preprocessing operation includes at least one or more of normalization, resampling, or cropping.

[0032] In some implementation scenarios, to address the differences in grayscale value ranges between different modalities, z-score normalization is used to map the grayscale values ​​of all images to a unified range with a mean close to 0 and a variance close to 1, eliminating the impact of magnitude differences between different modalities on model training. Since the pixel spacing and slice thickness of images differ under different devices and scanning protocols, trilinear interpolation can be used to resample all images to a uniform spatial resolution, ensuring consistent spatial scale for the same anatomical region and avoiding feature misjudgment due to differences in physical dimensions.

[0033] Additionally, the image can be segmented into image patches. For example, the preprocessed image can be uniformly divided into image patches of 16×16 pixels, which preserves local anatomical details while balancing computational efficiency. After segmentation, each image patch serves as an independent input unit, which will then be converted into a feature vector through a feature extraction module.

[0034] In some implementation scenarios, the feature extraction module is a linear projection layer. It flattens the image data (or image patches) into one-dimensional vectors, then performs a linear transformation on these vectors using matrix multiplication, mapping the low-dimensional pixel vectors to high-dimensional feature vectors to obtain the initial feature vectors. In some implementation scenarios, because the linearly projected feature vectors only contain content information of the image patches and lack spatial location information, the model cannot distinguish the relative positions of different image patches in the original image. Therefore, a corresponding positional encoding can be added to each feature vector. By learning the encoding differences at different positions, the model can capture the spatial relationships between image patches.

[0035] In step S103, based on the multimodal image data, an auxiliary model incorporating both image modality and image region encoding is used for auxiliary encoding to obtain adaptive incremental parameters. In some embodiments, the auxiliary model may include a perceptron, an image region encoder, and an image modality encoder. Specifically, based on the multimodal image data, corresponding encodings are performed using a perceptron, an image region encoder, and an image modality encoder, respectively, and then adaptive incremental parameters are obtained based on the weighted average of the encoding results.

[0036] In some implementation scenarios, the perceptron can be, for example, a 2-layer MLP structure, including an input layer, a hidden layer (e.g., an average pooling layer) and an output layer, to extract low-level features from preprocessed multimodal image data and capture common features (such as grayscale gradients and edge contours) shared by different modalities and different parts.

[0037] Image site encoders and image modality encoders can be, for example, convolutional models. Image site encoders focus on learning features specific to different anatomical sites, such as the bronchial tree texture features of the lungs, the ventricular outline features of the brain, and the morphological features of the liver / kidneys in the abdomen. Image modality encoders focus on learning features specific to different imaging modalities, such as density difference features in CT images (distinguishing between the density of bone and soft tissue), T1 / T2 signal difference features in MR images (distinguishing between the signal of gray matter and white matter), and transmittance features in DR images (transmittance difference between organs and air).

[0038] In the implementation scenario, preprocessed multimodal image data is input into a perceptron, an image modality encoder, and an image part encoder, respectively. These three modules perform encoding operations in parallel, outputting perceptron-encoded features, modal feature vectors, and part feature vectors, respectively. By introducing learnable modality and part dictionaries, matrix multiplication is performed between the modality feature vectors and the modality dictionary to obtain continuous modality embeddings; matrix multiplication is performed between the part feature vectors and the part dictionary to obtain continuous part embeddings. This dictionary mapping enhances the continuity and discriminative power of the features.

[0039] Furthermore, a weighted average is applied to the perceptron-encoded features, continuous modality embeddings, and continuous part embeddings. The weight parameters are learnable vectors, and the contribution ratio of each encoding result is dynamically adjusted through training to ultimately generate low-rank adaptive incremental parameters. Through the combined design of dual encoders and perceptrons, the adaptive incremental parameters can accurately capture the interaction between modal differences and part features, avoiding a one-size-fits-all approach to all scenarios with a single parameter, and providing a foundation for the dynamic adjustment of subsequent feature encodings.

[0040] In step S104, the adaptive incremental parameter is superimposed on the shared invariant parameter of the feature encoding module, and the feature encoding module is used to encode the initial feature vector to obtain the intermediate feature vector.

[0041] In some implementations, the feature encoding module can be, for example, a Transformer model. In some embodiments, the feature encoding module may include at least one or more attention layers and feedforward network layers. The shared invariant parameters include the first parameter of the attention layer and the second parameter of the feedforward network layer. It can be understood that the shared invariant parameters are fundamental weight parameters in the feature encoding module that are common to all modalities and anatomical sites, used to capture common structural features of radiographic images.

[0042] In some embodiments, the adaptive incremental parameters include a third parameter corresponding to the attention layer and a fourth parameter corresponding to the feedforward network layer. The adaptive incremental parameters are superimposed onto the shared invariant parameters of the feature encoding module through the following operations: superimposing the third parameter with the first parameter, and superimposing the fourth parameter with the second parameter. The adaptive incremental parameters are low-rank weight parameters generated by the auxiliary model, specifically adapted to combinations of specific modalities and specific parts, achieving dynamic adjustment. The superposition method can be, for example, weighted addition.

[0043] As an example, assuming the first and second parameters are denoted as Att_Param1 and FFN_Param2 respectively, and the third and fourth parameters are denoted as Att_Param3 and FFN_Param4 respectively, after superimposing the adaptive incremental parameters onto the shared invariant parameters of the feature encoding module, the attention layer parameter Att_Params = Att_Param1 + α×Att_Param3 is used to balance the general feature capture capability of the shared invariant parameters and the scene adaptation capability of the adaptive incremental parameters, avoiding excessive interference of the incremental parameters with the general feature learning; the feedforward network layer parameter FFN_Params = FFN_Param2 + β×FFN_Param4, where α and β are both adaptation coefficients (e.g., 0.5), ensuring that the adaptive adjustment does not deviate from the overall feature learning law of the model.

[0044] In some implementation scenarios, if the dimensions of the adaptive incremental parameters do not match due to modal / part differences, the incremental parameters can be mapped to the same dimension as the shared invariant parameters through low-rank decomposition (such as SVD decomposition) to ensure the feasibility of the superposition operation.

[0045] In other embodiments, the feature encoding module may further include, for example, a normalization layer positioned before each attention layer and feedforward network layer. In the implementation scenario, the initial feature vector sequence with added positional encoding is input into the feature encoding module. First, the features are normalized by the normalization layer, adjusting the mean and variance of the feature vectors to a fixed range to avoid training instability caused by feature value fluctuations. Next, a multi-head attention mechanism is constructed using the stacked attention layer parameters (Att_Params). Each attention head computes feature dependencies at different scales in parallel, such as the correlation between local image patches and the long-range correlation between global image patches. The attention-weighted feature vector is output through weighted summation to capture the spatial correlation information of anatomical structures.

[0046] The output of the attention layer is residually connected to the input feature vector to preserve the original feature information. This is then normalized a second time through a normalization layer, providing a stable feature distribution as input to the feedforward network layer. The stacked feedforward network layers (FFN_Params) perform a non-linear mapping on the normalized feature vector, deepening the high-dimensional features and uncovering hidden, subtle features, such as differences in grayscale distribution across different modalities of organs. The feedforward network layer is then residually connected to the input feature vector, completing multiple rounds of deep encoding, ultimately outputting a sequence of intermediate feature vectors.

[0047] Furthermore, in step S105, the intermediate feature vector is input to the feature decision module for feature decision-making, and learning is performed based on the comparison between the image part label and the recognition label, so that the recognition model outputs the image part, thereby training the recognition model for image part recognition.

[0048] In some implementation scenarios, the feature decision module can be, for example, a multilayer perceptron (MLP). In this scenario, the feature decision module can perform element-wise averaging of the intermediate feature vectors of all image patches to generate a global feature vector for the entire image. This global feature vector is then mapped to a high-dimensional vector for deep feature mining. The high-dimensional vector is mapped to a preset anatomical site category dimension. Finally, a softmax activation function can be used to transform the category-dimensional feature vector into matching probabilities for each site. The site category with the highest probability becomes the model's recognition label. Simultaneously, the cross-entropy loss value between this recognition label and the true label of the image site is output to supervise model parameter optimization and complete the training of the recognition model.

[0049] In other implementation scenarios, for contrastive learning training, contrastive learning sample pairs can be constructed based on multimodal image data. Positive sample pairs consist of different modal image samples of the same anatomical site, such as brain CT images and brain MR images. Negative sample pairs consist of image samples of different anatomical sites, such as brain CT images and chest DR images. By extracting feature embedding vectors for positive and negative sample pairs from the intermediate feature vector sequence, it is ensured that the embedding vectors represent the core features of the images. The contrastive loss can be calculated using a function such as InfoNCE, and then superimposed (e.g., weighted) with the aforementioned cross-entropy loss. The AdamW optimizer is then used for backpropagation to update all learnable parameters of the model.

[0050] As described above, the embodiments of this application, through a dual-parameter collaborative mechanism of shared invariant parameters and adaptive incremental parameters, not only ensure the model's basic recognition capabilities and avoid over-reliance on specific scenarios, but also accurately adapt to combinations of specific modalities and specific body parts. By superimposing both, the weights of the feature encoding module retain the recognition capability of general features while dynamically adjusting the encoding logic through incremental parameters to offset feature shifts caused by different device resolutions and scanning parameters, thereby achieving stable cross-scenario adaptation.

[0051] Furthermore, a cross-modal consistency self-supervised contrastive learning mechanism plays a crucial role. The contrastive loss function forces the model to maintain high similarity in feature embedding vectors of the same body part across different modalities, while ensuring significant differentiation between feature embedding vectors of different body parts. This constraint makes the learned features more universal and discriminative, effectively filtering out the interference of modal differences and image noise on the recognition results, and significantly improving the accuracy of body part identification. Through the synergistic effect of parameter adaptive adaptation, precise feature encoding, and self-supervised constraint optimization, the real-time performance, reliability, and cross-scenario adaptability of radiological image quality control are significantly improved, providing core technical support for image mutual recognition among medical institutions and promoting the intelligent upgrade of medical image quality control.

[0052] Figure 2 This is an exemplary flowchart illustrating the overall process of training a recognition model for image part recognition according to an embodiment of this application. Figure 2 As shown, firstly, multimodal image data (e.g., CT images, MR images, DR images) 201 are acquired, with each multimodal image data 201 labeled with an image region. Next, preprocessing operations are performed on the multimodal image data 201. These preprocessing operations include, for example, normalization, resampling, cropping, or image patch segmentation, as shown in the figure, where image patch 202 is obtained through image patch segmentation. This image patch 202 serves as input to the recognition model 203 for subsequent model training.

[0053] As shown in the figure, the recognition model 203 may include a feature extraction module 203-1, a feature encoding module 203-2, and a feature decision module 203-3. As mentioned earlier, the feature extraction module 203-1 is a linear projection layer, the feature encoding module 203-2 may be a Transformer model, and the feature decision module 203-3 may be a multilayer perceptron. Specifically, based on the image patch 202, features are first extracted by the feature extraction module 203-1 to obtain an initial feature vector 204. In addition, a corresponding positional encoding 205 can be added to each feature vector.

[0054] Next, feature encoding is performed via feature encoding module 203-2 to obtain intermediate feature vectors. The parameters in feature encoding module 203-2 integrate shared invariant parameters (first and second parameters) and adaptive incremental parameters (third and fourth parameters) from auxiliary model 206, which will be discussed later. Figure 3 Detailed Explanation. Further, the intermediate feature vector undergoes feature decision-making via feature decision module 203-3, outputting the recognition result of the image region. Additionally, the recognition model is optimized through comparative learning between the recognition label 207 and the image region label 208 to obtain the trained recognition model.

[0055] Figure 3 This is an exemplary schematic diagram illustrating the superposition of parameters according to an embodiment of this application. For example... Figure 3 The left side of the image shows a partial structure of the feature encoding module 203-2. As mentioned earlier, the feature encoding module 203-2 may include an attention layer 301 and a feedforward network layer 302. A normalization layer 303 may also be included before each attention layer 301 and feedforward network layer 302. Shared invariant parameters include the first parameter Att_Param1 of the attention layer and the second parameter FFN_Param2 of the feedforward network layer.

[0056] Figure 3 The right side of the image shows an exemplary structure of the auxiliary model 206, which may include a perceptron 304, an image part encoder 305, and an image modality encoder 306. The perceptron 304 may include, for example, two MLP layers 307 and an average pooling layer 308. The image part encoder 305 and the image modality encoder 306 may be, for example, convolutional models. As mentioned earlier, the three modules perform encoding operations in parallel, outputting perceptron-encoded features, modality feature vectors, and part feature vectors, respectively. Then, feature embeddings are introduced and weighted averages are performed 309 to generate adaptive incremental parameters, namely the third parameter Att_Param3 and the fourth parameter FFN_Param4.

[0057] In the implementation scenario, the third parameter Att_Param3 and the fourth parameter FFN_Param4 are superimposed on the first parameter Att_Param1 and the second parameter FFN_Param2. Feature encoding module 203-2 outputs an intermediate feature vector based on the superimposed parameters. For more details regarding each module, please refer to the preceding text. Figure 1 The descriptions made will not be repeated here.

[0058] Based on the recognition model trained above, it can be directly used for image part recognition to obtain image part recognition results, providing reliability and cross-scene adaptability, and providing core technical support for image mutual recognition.

[0059] In some embodiments, this application provides an apparatus for image part recognition, for example... Figure 4 As shown, Figure 4 This is an exemplary structural block diagram illustrating an apparatus 400 for image part recognition according to an embodiment of this application.

[0060] like Figure 4 As shown, the device 400 may include a processor 401 and a memory 402. The processor 401 may include, for example, a general-purpose processor (“CPU”) or a dedicated graphics processor (“GPU”), and the memory 402 stores program instructions executable on the processor. In some embodiments, the memory 402 may include, but is not limited to, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), and enhanced dynamic random access memory (EDRAM).

[0061] Furthermore, the memory 402 may store program instructions for image part recognition. When the program instructions are executed by the processor, the device 400 performs the following operations: acquiring image data to be recognized, inputting the image data into the trained recognition model for image part recognition, and obtaining the recognition result of the image part.

[0062] Figure 5 An exemplary structural block diagram of an electronic device 500 according to an embodiment of this application is shown. It will be understood that the device implementing the solution of this application may be a single device (e.g., a computing device) or a multifunctional device including various peripheral devices.

[0063] like Figure 5As shown, the electronic device of this application may include a central processing unit (“CPU”) 511, which may be a general-purpose CPU, a dedicated CPU, or other information processing and program execution unit. Furthermore, the electronic device 500 may also include a mass storage device 512 and a read-only memory (“ROM”) 513. The mass storage device 512 may be configured to store various types of data, including various multimodal image data, feature vectors, adaptive incremental parameters, shared invariant parameters, algorithm data, intermediate results, and various programs required to run the electronic device 500. The ROM 513 may be configured to store data and instructions required for the power-on self-test of the electronic device 500, the initialization of various functional modules in the system, the drivers for the system's basic input / output, and the booting of the operating system.

[0064] Optionally, the electronic device 500 may also include other hardware platforms or components, such as the tensor processing unit (“TPU”) 514, graphics processing unit (“GPU”) 515, field-programmable gate array (“FPGA”) 516, and machine learning unit (“MLU”) 517 shown. It is understood that although various hardware platforms or components are shown in the electronic device 500, they are merely exemplary and not limiting, and those skilled in the art can add or remove corresponding hardware as needed. For example, the electronic device 500 may include only a CPU, associated storage devices, and interface devices to implement the methods for training recognition models for image part recognition or the operations performed by the apparatus for image part recognition described in this application.

[0065] In some embodiments, to facilitate data transmission and interaction with external networks, the electronic device 500 of this application further includes a communication interface 518, through which it can connect to a local area network / wireless local area network (“LAN / WLAN”) 505, and further through the LAN / WLAN to connect to a local server 506 or to the Internet (“Internet”) 507. Alternatively or additionally, the electronic device 500 of this application can also directly connect to the Internet or cellular network via the communication interface 518 based on wireless communication technology, such as wireless communication technology based on 3G (“3G”), 4G (“4G”), or 5G (“5G”). In some application scenarios, the electronic device 500 of this application can also access the server 508 and database 509 of an external network as needed to obtain various known algorithms, data, and modules, and can remotely store various data, such as various data or instructions for presenting, for example, multimodal image data, feature vectors, adaptive incremental parameters, shared invariant parameters, etc.

[0066] Peripherals of the electronic device 500 may include a display device 502, an input device 503, and a data transmission interface 504. In one embodiment, the display device 502 may include, for example, one or more speakers and / or one or more visual displays, configured to provide voice prompts and / or display images and videos for training the recognition model for image part recognition or for operations performed by the device for image part recognition. The input device 503 may include, for example, a keyboard, mouse, microphone, posture capture camera, or other input buttons or controls, configured to receive audio data input and / or user commands. The data transmission interface 504 may include, for example, a serial interface, parallel interface, or Universal Serial Bus interface (“USB”), Small Computer System Interface (“SCSI”), Serial ATA, FireWire (“FireWire”), PCI Express, and High Definition Multimedia Interface (“HDMI”), configured for data transmission and interaction with other devices or systems. According to the scheme of this application, the data transmission interface 504 can receive acquired multimodal image data from radiology equipment and transmit multimodal image data or various other types of data or results to the electronic device 500.

[0067] The aforementioned CPU 511, mass storage 512, ROM 513, TPU 514, GPU 515, FPGA 516, MLU 517, and communication interface 518 of the electronic device 500 of this application can be interconnected via bus 519, and can interact with peripheral devices through this bus. In one embodiment, the CPU 511 can control other hardware components and peripheral devices in the electronic device 500 through bus 519.

[0068] The above combination Figure 5 An electronic device that can be used to execute this application is described. It should be understood that the device structure or architecture described herein is merely exemplary, and the implementation method and implementation entity of this application are not limited thereto, but can be changed without departing from the spirit of this application.

[0069] Based on the foregoing description in conjunction with the accompanying drawings, those skilled in the art will understand that the embodiments of this application can also be implemented by software programs. Therefore, this application also provides a computer-readable storage medium storing computer-readable instructions for training a recognition model for image part recognition or for image part recognition. When executed by one or more processors, these computer-readable instructions can be used to implement the embodiments of this application in conjunction with the accompanying drawings. Figure 1 The described method for training a recognition model for image part recognition or Figure 4 The operations performed by a device used for image part recognition.

[0070] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0071] It should be understood that when the terms "first," "second," "third," and "fourth," etc., are used in the claims, specification, and drawings of this application, they are used only to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0072] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0073] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for training a recognition model for image region identification, wherein the recognition model includes a feature extraction module, a feature encoding module, and a feature decision module, and the method includes: Acquire multimodal image data of labeled image regions; Based on the multimodal image data, the feature extraction module is used to extract features to obtain an initial feature vector; Based on the multimodal image data, an auxiliary model that includes dual encoding of image modality and image region is used for auxiliary encoding to obtain adaptive incremental parameters; The adaptive incremental parameter is superimposed on the shared invariant parameter of the feature encoding module, and the feature encoding module is used to encode the initial feature vector to obtain the intermediate feature vector; The intermediate feature vector is input into the feature decision module for feature decision-making, and learning is performed based on the comparison between image part labels and recognition labels, so that the recognition model outputs the image part, thereby training the recognition model for image part recognition. The feature encoding module includes at least one or more attention layers and feedforward network layers. The shared invariant parameters include a first parameter of the attention layer and a second parameter of the feedforward network layer. The adaptive incremental parameters include a third parameter of the corresponding attention layer and a fourth parameter of the corresponding feedforward network layer. The adaptive incremental parameters are superimposed on the shared invariant parameters of the feature encoding module by the following operation: superimposing the third parameter with the first parameter, and superimposing the fourth parameter with the second parameter. The auxiliary model includes a perceptron, an image part encoder, and an image modality encoder. Based on the multimodal image data, auxiliary encoding is performed using an auxiliary model that includes both image modality and image part encoding to obtain adaptive incremental parameters, including: Based on the multimodal image data, corresponding encoding is performed using the sensor, the image part encoder, and the image modality encoder, respectively; The adaptive incremental parameter is obtained based on the weighted average result of each encoding.

2. The method according to claim 1, wherein before performing feature extraction using the feature extraction module based on the multimodal image data, it further comprises: The multimodal image data is preprocessed, wherein the preprocessing operation includes at least one or more of normalization, resampling, or cropping.

3. The method according to claim 1 or 2, wherein the multimodal image data includes at least two or more of CT images, MR images, and DR images.

4. The method according to claim 1, wherein the recognition model includes the ViT model.

5. An apparatus for image region recognition, comprising: processor; as well as A memory storing computer instructions for image part recognition, which, when executed by a processor, cause the following operations to be performed: Collect image data to be identified; The image data is input into the recognition model trained by the method according to any one of claims 1-4 to identify the image parts and obtain the recognition results of the image parts.

6. An electronic device, comprising: processor; as well as A memory storing computer instructions for training a recognition model for image part recognition, which, when executed by a processor, cause the method according to any one of claims 1-4 to be performed; or storing computer instructions for image part recognition, which, when executed by a processor, cause the operation performed by the apparatus according to claim 5 to be performed.

7. A computer-readable storage medium storing computer program instructions thereon for training a recognition model for image part recognition, the computer program instructions being executed by one or more processors to cause the implementation of the method according to any one of claims 1-4; or storing computer program instructions thereon for image part recognition, the computer program instructions being executed by one or more processors to cause the implementation of the operation performed by the apparatus according to claim 5.

Images