Medical image processing method and device, equipment and storage medium

Through medical image processing methods, the encoder and autoregressive decoder of the medical image processing model are used to segment and integrate medical images, which solves the problem of low accuracy of medical image segmentation and achieves more efficient image feature extraction and segmentation Effect.

CN118887225APending Publication Date: 2024-11-01PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410917188.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

In medical image processing, it is difficult for the prior art to directly use the image segmentation model trained based on natural images to segment medical images, resulting in low segmentation accuracy, mainly due to inaccurate and insufficient number of manual annotations of training data.

Method used

A medical image processing method is proposed. By acquiring the target medical image, image segmentation is performed to obtain multiple medical image blocks, extract and integrate the features of each block, and use the encoder and autoregressive decoder of the preset medical image processing model. Perform encoding and segmentation processing, combining unsupervised training and model fine-tuning to reduce training difficulty and improve segmentation accuracy.

Benefits of technology

It improves the segmentation accuracy of medical images, reduces the difficulty of obtaining training data and model training, and enhances the ability to extract and reconstruct images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887225A_ABST
    Figure CN118887225A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a medical image processing method and device, equipment and a storage medium, and belongs to the field of digital medical treatment. The method comprises the following steps: acquiring a target medical image; performing image segmentation on the target medical image to obtain a plurality of medical image blocks arranged in sequence; for each medical image block, performing feature extraction on the medical image block to obtain an image block representation feature of the medical image block; performing integration processing on the image block characterization features of the plurality of medical image blocks to obtain a target image characterization feature of the target medical image; an encoder based on a medical image processing model encodes the target image characterization features to obtain medical image encoding features of the target medical image; and segmenting the coding features of the medical image based on the autoregression decoder of the medical image processing model to obtain the image category of each medical image block. The embodiment of the invention can improve the segmentation accuracy of the medical image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of digital medicine, and in particular to a medical image processing method and apparatus, device and storage medium. Background Art

[0002] At present, image segmentation in various business scenarios often relies on deep learning models. Among them, deep learning models can automatically learn feature representation and classification rules from a large amount of labeled data, without the need for artificially designed feature extractors or setting threshold parameters, and can achieve the effect of improving image segmentation efficiency and accuracy.

[0003] However, in the medical field, there is a large semantic gap between medical images and ordinary natural images, so it is often impossible to directly use the image segmentation model trained based on natural images to segment medical images. In this regard, the relevant technology often uses manual annotation to create training data, uses the training data to fine-tune the image segmentation model trained based on natural images, and uses the fine-tuned image segmentation model to segment medical images. Although this method can solve the above-mentioned problem of not being able to directly segment medical images to a certain extent, the manual annotation method will have the problems of inaccurate annotation of training data and a small number of annotated training data, which will affect the effect of model fine-tuning and also lead to low accuracy of medical image segmentation. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a medical image processing method and apparatus, a device and a storage medium, aiming to improve the accuracy of medical image segmentation.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a medical image processing method, the method comprising:

[0006] Acquire a target medical image;

[0007] Performing image segmentation on the target medical image to obtain a plurality of sequentially arranged medical image blocks;

[0008] For each of the medical image blocks, extracting features of the medical image block to obtain image block representation features of the medical image block;

[0009] Integrating the image block representation features of the plurality of medical image blocks to obtain a target image representation feature of the target medical image;

[0010] The encoder based on a preset medical image processing model encodes the target image representation features to obtain the medical image encoding features of the target medical image, wherein the medical image processing model also includes an autoregressive decoder; the medical image processing model is obtained by adjusting a pre-trained first model, the first model is trained based on the pixel error between a first sample medical image and a predicted medical image, the predicted medical image is obtained by inputting a masked medical image into the first model, and the masked medical image is obtained by performing masking processing on the first sample medical image;

[0011] The medical image coding features are segmented based on the autoregressive decoder to obtain image categories of each medical image block.

[0012] In some embodiments, the first model is trained by:

[0013] Acquire a first sample medical image, wherein the first sample medical image does not have an image segmentation label;

[0014] Performing mask processing on the first sample medical image to obtain a masked medical image, and performing image segmentation on the masked medical image to obtain a plurality of masked medical image blocks;

[0015] Performing feature extraction on each of the masked medical image blocks to obtain a masked image block representation feature of the masked medical image block, and integrating a plurality of the masked image block representation features to obtain a masked image representation feature;

[0016] Based on the encoder of the first model, the mask image representation feature is encoded to obtain a mask image encoding feature;

[0017] Performing image prediction on the encoded features of the mask image through the autoregressive decoder of the first model to obtain the predicted medical image;

[0018] For each of the first sample medical images, a pixel error between the predicted medical image and the first sample medical image is calculated, and the first model is trained according to the pixel error.

[0019] In some embodiments, performing mask processing on the first sample medical image to obtain a masked medical image includes:

[0020] Determining a target image region in the first sample medical image;

[0021] For each pixel point in the target image area, the pixel value of the pixel point is replaced by the first value to obtain the masked medical image.

[0022] In some embodiments, the target image area is determined by:

[0023] determining an image size of the first sample medical image, and determining a model accuracy of the medical image processing model;

[0024] Determining the area of ​​the target image region based on the image size and the model accuracy;

[0025] Based on the area of ​​the region, the target image region is determined in the first sample medical image.

[0026] In some embodiments, the medical image processing model is obtained by adjusting the first model in the following manner:

[0027] Acquire a second sample medical image and an image segmentation label of the second sample medical image;

[0028] encoding the second sample image representation feature of the second sample medical image by an encoder based on the first model to obtain a sample image encoding feature of the second sample medical image;

[0029] The autoregressive decoder based on the first model performs segmentation processing on the encoding features of the sample image to obtain a sample prediction segmentation result of the first sample medical image;

[0030] Based on the comparison of the sample prediction segmentation results of the plurality of second sample medical images and the image segmentation labels, the parameters of the first model are adjusted to obtain the medical image processing model.

[0031] In some embodiments, adjusting parameters of the first model based on the comparison of the sample prediction segmentation results of the plurality of second sample medical images and the image segmentation labels includes:

[0032] For each of the second sample medical images, calculating a sub-loss function based on the sample predicted segmentation result and the image segmentation label;

[0033] Determining a first loss function based on a plurality of sub-loss functions of the second sample medical images;

[0034] Parameters of the first model are adjusted based on the first loss function.

[0035] In some embodiments, for each of the medical image blocks, extracting features from the medical image block to obtain image block representation features of the medical image block includes:

[0036] Performing convolution processing on the medical image blocks to obtain image block convolution features of each medical image block;

[0037] The image block convolution feature is activated based on a predetermined activation function to obtain the image block representation feature.

[0038] In some embodiments, the encoder based on the pre-determined medical segmentation model encodes the target image representation feature to obtain the medical image encoding feature of the target medical image, including:

[0039] Performing a multi-head attention calculation on the target image representation feature based on the encoder to obtain an attention calculation result, wherein the attention calculation result is used to indicate the degree of correlation between the medical image blocks in the target medical image;

[0040] Mapping the attention calculation result to a predetermined vector space to obtain a target mapping vector;

[0041] The medical image coding feature is obtained based on residual splicing of the target mapping vector and the target image representation feature.

[0042] In some embodiments, the medical image coding feature comprises image block coding features of each of the medical image blocks, the plurality of medical image blocks are arranged in a first order, and the plurality of medical image blocks comprises a target image block;

[0043] The segmenting process of the medical image coding features based on the autoregressive decoder to obtain the image category of each of the medical image blocks includes:

[0044] Determining the image category of each of the image blocks that precede the target image block in the first sequence;

[0045] Based on the image categories of each of the image blocks arranged before the target image block and the image block coding features of the target image block, segmentation processing is performed by the autoregressive decoder to obtain the target image category of the target image block.

[0046] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a medical image processing device, the device comprising:

[0047] An image acquisition unit, used for acquiring a target medical image;

[0048] A segmentation unit, used for segmenting the target medical image to obtain a plurality of sequentially arranged medical image blocks;

[0049] A feature extraction unit, configured to extract features from each medical image block to obtain image block representation features of the medical image block;

[0050] an integration unit, configured to integrate the image block representation features of the plurality of medical image blocks to obtain a target image representation feature of the target medical image;

[0051] an encoding unit, configured to encode the target image representation features based on an encoder of a preset medical image processing model to obtain medical image encoding features of the target medical image, wherein the medical image processing model further includes an autoregressive decoder; the medical image processing model is obtained by adjusting a pre-trained first model, the first model is obtained by training based on a pixel error between a first sample medical image and a predicted medical image, the predicted medical image is obtained by inputting a masked medical image into the first model, and the masked medical image is obtained by performing masking processing on the first sample medical image;

[0052] A processing unit is used to perform segmentation processing on the medical image coding features based on the autoregressive decoder to obtain the image category of each of the medical image blocks.

[0053] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the medical image processing method described in the first aspect is implemented.

[0054] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the medical image processing method described in the first aspect.

[0055] The medical image processing method and device, electronic device and storage medium proposed in the present application obtain a target medical image; perform image segmentation on the target medical image to obtain a plurality of sequentially arranged medical image blocks; perform feature extraction on each medical image block to obtain image block representation features of the medical image block, which can map the image feature information of the medical image block from the pixel space to the vector space, and then integrate the image block representation features of the plurality of medical image blocks to obtain the target image representation features of the target medical image, which can improve the feature quality of the target image representation features. Furthermore, an encoder based on a preset medical image processing model encodes the target image representation features to obtain the medical image encoding features of the target medical image, and an autoregressive decoder based on the medical image processing model segmentates the medical image encoding features to obtain the image categories of each medical image block. This method can improve the segmentation accuracy of medical images by combining the encoder and the autoregressive decoder. In addition, the medical image processing model is obtained by adjusting the pre-trained first model, the first model is trained based on the pixel error between the first sample medical image and the predicted medical image, the predicted medical image is obtained by inputting the masked medical image into the first model, and the masked medical image is obtained by masking the first sample medical image. Unsupervised training is used in training the first model, which can effectively reduce the difficulty of obtaining training samples and is conducive to improving the model training effect of the first model. After training the first model, the medical image processing model is adjusted based on the first model by model fine-tuning, which can effectively reduce the model training difficulty of the medical image processing model and improve the accuracy of the medical image processing model in processing medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a medical image processing method provided by an embodiment of the present application;

[0057] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0058] Figure 3 It is a schematic diagram of a specific implementation process of generating image block representation features according to an embodiment of the present application;

[0059] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0060] Figure 5 It is a schematic diagram of a specific implementation process of generating medical image coding features according to an embodiment of the present application;

[0061] Figure 6 yes Figure 1Flow chart of step S106 in FIG.

[0062] Figure 7 is a flowchart of adjusting the first model into a medical image processing model according to an embodiment of the present application;

[0063] Figure 8 is a flowchart of training a first model according to an embodiment of the present application;

[0064] Fig. 9 is a flowchart of performing mask processing on a first sample medical image according to an embodiment of the present application;

[0065] Fig.10 is a schematic diagram of a specific implementation process of performing mask processing on a first sample medical image according to an embodiment of the present application;

[0066] Fig.11 is a schematic diagram of the structure of a medical image processing device provided in an embodiment of the present application;

[0067] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0071] First, some nouns involved in this application are analyzed:

[0072] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0074] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0075] At present, image segmentation in various business scenarios often relies on deep learning models. Among them, deep learning models can automatically learn feature representation and classification rules from a large amount of labeled data, without the need for artificially designed feature extractors or setting threshold parameters, and can achieve the effect of improving image segmentation efficiency and accuracy.

[0076] However, in the medical field, there is a large semantic gap between medical images and ordinary natural images, so it is often impossible to directly use the image segmentation model trained based on natural images to segment medical images. In this regard, the relevant technology often uses manual annotation to create training data, uses the training data to fine-tune the image segmentation model trained based on natural images, and uses the fine-tuned image segmentation model to segment medical images. Although this method can solve the above-mentioned problem of not being able to directly segment medical images to a certain extent, the manual annotation method will have the problems of inaccurate annotation of training data and a small number of annotated training data, which will affect the effect of model fine-tuning and also lead to low accuracy of medical image segmentation.

[0077] Based on this, the embodiments of the present application provide a medical image processing method, a medical image processing device, an electronic device and a storage medium, aiming to improve the accuracy of medical image segmentation.

[0078] The medical image processing method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the medical image processing method in the embodiments of the present application is described.

[0079] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0080] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0081] Medical cloud refers to the use of cloud computing to create a cloud platform for medical and health services based on new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the application of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. For example, hospital appointment registration, electronic medical records, and medical insurance are all products of the combination of cloud computing and the medical field. Medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout.

[0082] The medical image processing method provided in the embodiment of the present application relates to the field of digital medical technology. The medical image processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0085] Figure 1 is an optional flowchart of the medical image processing method provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S106.

[0086] Step S101, acquiring a target medical image;

[0087] Step S102, segmenting the target medical image to obtain a plurality of sequentially arranged medical image blocks;

[0088] Step S103, for each medical image block, extracting features of the medical image block to obtain image block representation features of the medical image block;

[0089] Step S104, integrating the image block representation features of the plurality of medical image blocks to obtain target image representation features of the target medical image;

[0090] Step S105, encoding the target image representation features based on the encoder of the preset medical image processing model to obtain the medical image encoding features of the target medical image;

[0091] Step S106, segmenting the medical image coding features based on the autoregressive decoder to obtain the image category of each medical image block.

[0092] Steps S101 to S106 shown in the embodiment of the present application are as follows: acquiring a target medical image; segmenting the target medical image to obtain a plurality of sequentially arranged medical image blocks; extracting features of each medical image block to obtain image block representation features of the medical image block, which can map the image feature information of the medical image block from the pixel space to the vector space; then, integrating and processing the image block representation features of the plurality of medical image blocks to obtain the target image representation features of the target medical image, which can improve the feature quality of the target image representation features. Furthermore, an encoder based on a preset medical image processing model encodes the target image representation features to obtain the medical image encoding features of the target medical image, and an autoregressive decoder based on the medical image processing model segmentates the medical image encoding features to obtain the image categories of each medical image block. This method can improve the segmentation accuracy of medical images by combining the encoder and the autoregressive decoder. In addition, the medical image processing model is obtained by adjusting the pre-trained first model, the first model is trained based on the pixel error between the first sample medical image and the predicted medical image, the predicted medical image is obtained by inputting the masked medical image into the first model, and the masked medical image is obtained by masking the first sample medical image. Unsupervised training is used in training the first model, which can effectively reduce the difficulty of obtaining training samples and is conducive to improving the model training effect of the first model. After training the first model, the medical image processing model is adjusted based on the first model by model fine-tuning, which can effectively reduce the model training difficulty of the medical image processing model and improve the accuracy of the medical image processing model in processing medical images.

[0093] In step S101 of some embodiments, a target medical image is acquired.

[0094] The target medical image refers to a medical image in the medical field, and the type of object contained in the target medical image is a lesion, that is, a part of the body where a disease occurs. Medical images refer to images of internal tissues, such as the stomach, abdomen, heart, knee, and brain, obtained in a non-invasive manner for medical treatment or medical research, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasound), X-ray images, electroencephalograms, and optical photographic lights generated by medical instruments.

[0095] In the specific implementation of this embodiment, the target medical image may be obtained from an existing medical database, or may be obtained by photographing with a camera or other means, without limitation.

[0096] In step S102 of some embodiments, the target medical image is segmented to obtain a plurality of sequentially arranged medical image blocks.

[0097] The medical image blocks are used to indicate the image contents of different image regions of the target medical image.

[0098] In the specific implementation of this embodiment, first, the image size of the target medical image and the predetermined segmentation size of the medical image blocks are determined, wherein the image size is used to indicate the size of the target medical image, and the predetermined segmentation size is used to indicate the size of each medical image block. Next, according to the image size of the target medical image and the predetermined segmentation size of the medical image blocks, the target medical image is segmented, and the target medical image is segmented into a plurality of medical image blocks of the same size, and the size of each medical image block is equal to the predetermined segmentation size. Further, the medical image blocks are sorted according to a predetermined sorting rule to obtain a plurality of sequentially arranged medical image blocks. The predetermined sorting rule is used to indicate that each medical image block is arranged in order from top to bottom and from left to right according to its position in the target medical image.

[0099] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:

[0100] Step S201, performing convolution processing on the medical image block to obtain image block convolution features of each medical image block;

[0101] Step S202: Activate the convolutional features of the image block based on a predetermined activation function to obtain representation features of the image block.

[0102] In step S201 of some embodiments, the image block convolution feature is used to indicate the result of mapping the image feature of the medical image block from the pixel space to the vector space.

[0103] In the specific implementation of this embodiment, for each medical image block, convolution processing is performed on the medical image block to capture the image feature information of the medical image block, and the image features of the medical image block are mapped from the pixel space to the vector space to obtain the image block convolution features of each medical image block.

[0104] In step S202 of some embodiments, a predetermined activation function is used to perform linear correction on the input feature, wherein the predetermined activation function may be a Re lu function.

[0105] In the specific implementation of this embodiment, for each medical image block, the image block convolution feature is input into a predetermined activation function, and the predetermined activation function is used to perform linear correction on the characteristic elements in the image block convolution feature to obtain the image block representation feature of the medical image block.

[0106] The advantage of this embodiment is that a stacked residual network is used to perform convolution and activation processing on each medical image block, which can map the image feature information of the medical image block from the pixel space to the vector space (latent space), achieve preliminary extraction of the image feature information of the medical image block, obtain the image block representation features of the medical image block, and improve the efficiency and accuracy of image feature extraction.

[0107] In step S104 of some embodiments, the image block representation features of multiple medical image blocks are integrated to obtain target image representation features of the target medical image.

[0108] The target image representation features are used to indicate the image content information of the target medical image in the vector space (latent space).

[0109] In the specific implementation of this embodiment, first, the sequence number of each medical image block is determined according to the order of each medical image block. Then, according to the size of the sequence number, the image block representation features of each medical image block are sequentially combined together to obtain a serialized data, and the serialized data is used as the target image representation feature of the target medical image.

[0110] like Figure 3As shown, it is a schematic diagram of the specific process of generating the target image representation features of the target medical image. Specifically, first, a target medical image with an image size of H*W*3 is segmented to obtain 16 medical image blocks, wherein the size of each medical image block is (H / 4)*(W / 4)*3, H is the height of the target medical image, W is the width of the target medical image, and 3 is the number of channels of the target medical image. Further, each medical image block is arranged from top to bottom and from left to right according to its position in the target medical image to flatten multiple medical image blocks and obtain a sequence of multiple medical image blocks. Next, in order to obtain the image representation of each medical image block, three stacked residual networks (residual network 1, residual network 2 and residual network 3) are used to extract image block features, and the image information of each medical image block is extracted into a 512-dimensional image representation to obtain the image block representation features of each medical image block. In the process of extracting image block features, each medical image block is first input into residual network 1, the output of residual network 1 is used as the input of residual network 2, the output of residual network 2 is used as the input of residual network 3, and the output of residual network 3 is used as the image block representation feature of the medical image block. Residual network 1, residual network 2 and residual network 3 are all composed of a convolutional layer of size 3*3, a batch normalization layer and a Re lu activation function. Furthermore, for an image block sequence composed of multiple medical image blocks, the image block representation features of each medical image block are integrated so that the multiple image block representation features constitute a feature sequence, wherein the sequence number of each image block representation feature in the feature sequence is the same as the sequence number of the medical image block corresponding to the image block representation feature in the image block sequence. The feature sequence is serialized data in which each element is represented by 512, and the feature sequence is determined as the target image representation feature of the target medical image, thereby realizing the conversion of the target medical image with a two-dimensional structure into the target image representation feature with a one-dimensional structure, and meeting the input requirements of the medical image processing model.

[0111] In the disclosed embodiment, the medical image processing model includes an encoder and an autoregressive decoder.

[0112] The medical image processing model is obtained by adjusting the pre-trained first model. The first model is a neural network model with feature extraction and image reconstruction functions. The input of the first model is often a medical image with masked areas, and the output of the first model is often an unmasked reconstructed medical image corresponding to the medical image.

[0113] It should be noted that the first model is trained based on the pixel error between the first sample medical image and the predicted medical image, the predicted medical image is obtained by inputting the masked medical image into the first model, and the masked medical image is obtained by masking the first sample medical image.

[0114] To save space, the specific process of training the first model and the specific process of adjusting the first model to the medical image processing model in the embodiment of the present disclosure will be described in detail below. No further description will be given here.

[0115] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S403:

[0116] Step S401, performing multi-head attention calculation on the target image representation features based on the encoder to obtain an attention calculation result;

[0117] Step S402, mapping the attention calculation result to a predetermined vector space to obtain a target mapping vector;

[0118] Step S403, obtaining a medical image coding feature based on residual splicing of the target mapping vector and the target image representation feature.

[0119] In step S401 of some embodiments, the attention calculation result is used to indicate the degree of correlation between medical image blocks in the target medical image.

[0120] In the specific implementation of this embodiment, first, the target image representation feature is input into the encoder. Then, the target image representation feature is linearly projected in the encoder to obtain the query feature, key feature and value feature corresponding to the target representation feature. Further, the query feature is multiplied by the predetermined query weight matrix to obtain the first product result; the key feature is multiplied by the predetermined key weight matrix to obtain the second product result; the value feature is multiplied by the predetermined value weight matrix to obtain the third product result. Then, the first product result is multiplied by the transposed result of the second product result to obtain the fourth product result. The fourth product result is multiplied by the division result of the square difference of the feature dimension of the target image representation feature and the third product result to obtain the target calculation result, and the softmax function is used to perform multi-head attention calculation on the target calculation result to obtain the attention calculation result.

[0121] In step S402 of some embodiments, a predetermined vector space is used to convert the attention calculation result from a data dimension to a vector dimension.

[0122] In the specific implementation of this embodiment, the attention calculation result is input into a predetermined vector space, and the attention calculation result is linearly projected in the predetermined vector space, so that the attention calculation result is mapped from the data space to the vector space, and the attention calculation result in numerical form is converted into a vector matrix form to obtain a target mapping vector.

[0123] In step S403 of some embodiments, the medical image coding feature is used to indicate image information of higher importance in the target medical image.

[0124] In the specific implementation of this embodiment, residual splicing is performed on the target mapping vector and the target image representation feature, and feature fusion is performed on the target mapping vector and the target image representation feature to obtain the medical image coding feature.

[0125] like Figure 5 As shown, it is a schematic diagram of the specific process of feature encoding using the encoder of the medical image processing model. The encoder of the embodiment of the present disclosure is a transformer encoder, which is composed of 12 transformer modules connected in sequence and a linear mapping layer. The structure of each transformer module is the same, but the parameters of the transformer module are different. Specifically, first, the target image representation feature is input into the transformer encoder, and the target image representation feature is sequentially calculated by multi-head attention through 12 transformer modules, and the output of the previous transformer module is used as the input of the next transformer module, and the output result of the last transformer module is determined as the attention calculation result. Then, the attention calculation result is input into the linear mapping layer for feature mapping to obtain a target mapping vector. Finally, the target mapping vector and the target image representation feature are residually spliced ​​to obtain the medical image coding feature.

[0126] The benefit of this embodiment is that by using the transformer encoder to perform attention calculation on the target image representation features, the model can pay more attention to the more important image feature content in the target medical image, reduce the attention to irrelevant image feature content, and improve the feature quality of the medical image coding features.

[0127] In the embodiment of the present disclosure, the medical image coding feature includes image block coding features of each medical image block, the multiple medical image blocks are arranged in a first order, and the multiple medical image blocks include a target image block.

[0128] See also Figure 6 In some embodiments, step S106 may include but is not limited to steps S601 to S602:

[0129] Step S601, determining the image category of each image block that is arranged before the target image block in the first order;

[0130] Step S602 , based on the image categories of each image block before the target image block and the image block coding feature of the target image block, segmentation processing is performed through an autoregressive decoder to obtain a target image category of the target image block.

[0131] In an embodiment of the present application, an autoregressive decoder is used to decode image block coding features of medical image blocks in sequence according to a first order.

[0132] In step S601 of some embodiments, for a target image block, the image categories of each medical image block preceding the target image block in the first order are determined based on decoding of each medical image block preceding the target image block by an autoregressive decoder.

[0133] In step S602 of some embodiments, the image block coding features of the target image block and the image categories of each medical image block arranged before the target image block are input into the autoregressive decoder together. The autoregressive decoder first merges the input image category and the image block coding features to obtain a merged image block feature, and then feature decodes the merged image block feature to obtain a target image category of the target image block.

[0134] The advantage of this embodiment is that when segmenting and annotating each medical image block, when predicting the image category of the kth medical image block (for example, the target image block, k is an integer greater than 0), it is necessary to simultaneously refer to the image block encoding features of the target image block and the image categories of the k-1 medical image blocks that have been predicted previously, so that the decoding process of the autoregressive decoder has better local continuity, which can improve the segmentation accuracy and category analysis accuracy of the autoregressive decoder for the target medical image.

[0135] See also Figure 7 In some embodiments, the medical image processing model is obtained by adjusting the first model in the following manner:

[0136] Step S701, obtaining a second sample medical image and an image segmentation label of the second sample medical image;

[0137] Step S702, encoding the second sample image representation feature of the second sample medical image based on the encoder of the first model to obtain the sample image encoding feature of the second sample medical image;

[0138] Step S703, performing segmentation processing on the sample image encoding features based on the autoregressive decoder of the first model to obtain a sample prediction segmentation result of the first sample medical image;

[0139] Step S704: Based on the sample prediction segmentation results of the plurality of second sample medical images and the comparison of the image segmentation labels, the parameters of the first model are adjusted to obtain a medical image processing model.

[0140] In step S701 of some embodiments, the image segmentation label is used to indicate the image category of each image block after the second sample medical image is segmented.

[0141] In the specific implementation of this embodiment, with authorization, a certain number of medical images with segmentation annotations are extracted from a preset medical image database, the extracted medical images are used as second sample medical images, and the segmentation annotations of the medical images are determined as image segmentation labels of the second sample medical images.

[0142] In step S702 of some embodiments, the sample image coding feature is used to indicate image information of a second sample medical image with a higher degree of importance.

[0143] In the specific implementation of this embodiment, the specific process of step S702 is similar to the specific process of steps S401 to S403 described above, and will not be described in detail to save space.

[0144] In step S703 of some embodiments, the sample prediction segmentation result is used to indicate the image category of each sample medical image block into which the second sample medical image is segmented.

[0145] In the specific implementation of this embodiment, the specific process of step S703 is similar to the specific process of steps S601 to S602, and will not be described in detail to save space.

[0146] In step S704 of some embodiments, the specific process of adjusting the parameters of the first model includes but is not limited to the following steps:

[0147] For each second sample medical image, a sub-loss function is calculated based on the sample predicted segmentation result and the image segmentation label;

[0148] Determining a first loss function based on the sub-loss functions of the plurality of second sample medical images;

[0149] Parameters of the first model are adjusted based on the first loss function.

[0150] The sub-loss function is used to indicate the difference between the sample prediction segmentation result of a single second sample medical image and the image segmentation label. The first loss function is used to indicate the overall difference between the sample prediction segmentation results of all second sample medical images and the image segmentation label.

[0151] Specifically, first, for each second sample medical image, a cross entropy loss is calculated for the sample prediction segmentation result and the image segmentation label to obtain a cross entropy loss function of the second sample medical image, and the cross entropy loss function of the second sample medical image is used as a sub-loss function of the second sample medical image. Next, the sub-loss functions of multiple second sample medical images are averaged to obtain a first loss function. Finally, the parameters of the first model are adjusted according to the first loss function, and the process returns to step S701, and iterative training is performed according to the above process until the first loss function of a certain iteration round meets the predetermined training conditions, the model parameters of the iteration round are used as the final model parameters, and the first model with the final model parameters is determined as the medical image processing model.

[0152] The advantage of this embodiment is that the first model with certain image feature extraction and image reconstruction capabilities is supervisedly trained using labeled image data, and the first model can be fine-tuned using a smaller number of second sample medical images, thereby reducing the difficulty and amount of calculation for model fine-tuning and obtaining a medical image processing model that accurately analyzes medical images.

[0153] Since there are often few medical images with segmentation annotations in the medical field, directly using supervised segmentation training often results in low image processing accuracy of the generated image generation model. Based on this, the embodiment of the present application provides a model training scheme based on self-supervised learning, which can reduce the difficulty of obtaining training data, use unlabeled data to train a first model with better image processing capabilities, and improve the training effect of the first model.

[0154] See also Figure 8 In some embodiments, the first model is trained in the following manner:

[0155] Step S801, obtaining a first sample medical image;

[0156] Step S802, performing mask processing on the first sample medical image to obtain a masked medical image, and performing image segmentation on the masked medical image to obtain a plurality of masked medical image blocks;

[0157] Step S803, extracting features from each mask medical image block to obtain mask image block representation features of the mask medical image block, and integrating multiple mask image block representation features to obtain mask image representation features;

[0158] Step S804, encoding the mask image representation features based on the encoder of the first model to obtain mask image encoding features;

[0159] Step S805, performing image prediction on the encoded features of the mask image through the autoregressive decoder of the first model to obtain a predicted medical image;

[0160] Step S806: for each first sample medical image, calculate the pixel error between the predicted medical image and the first sample medical image, and train the first model according to the pixel error.

[0161] In step S801 of some embodiments, the first sample medical image does not have an image segmentation label. The first sample medical image is obtained in a manner that is substantially the same as the manner in which the target medical image is obtained in step S101 and the manner in which the second sample medical image is obtained in step S701. To save space, the details are not described here.

[0162] It should be noted that since the first sample medical images are unlabeled data and the second sample medical images are labeled data, the total image volume of the first sample medical images is often greater than the total image volume of the second sample medical images, and the difficulty of obtaining the first sample medical images is often less than the difficulty of obtaining the second sample medical images.

[0163] In step S802 of some embodiments, the mask medical image is used to indicate an image result in which a portion of pixels of the first sample medical image are blocked.

[0164] When this embodiment is implemented, the specific process of performing mask processing on the first sample medical image will be described in detail below, but will not be described here.

[0165] Furthermore, the specific process of segmenting the masked medical image to obtain a plurality of masked medical image blocks is similar to the specific process of the above step S102, which will not be described in detail to save space.

[0166] In step S803 of some embodiments, the specific process of step S803 is similar to the above-mentioned steps S103 to S104, and will not be described in detail to save space.

[0167] In step S804 of some embodiments, the mask image encoding feature is used to indicate image information of a masked medical image with a higher degree of importance.

[0168] In the specific implementation of this embodiment, the specific process of step S804 is similar to the specific process of steps S401 to S403 described above, and will not be described in detail to save space.

[0169] In step S805 of some embodiments, the predicted medical image is used to indicate an image result of the masked medical image reconstructed by the first model according to the encoded features of the masked image in the absence of occlusion.

[0170] The mask image coding features include mask image block coding features corresponding to each mask image block of the mask medical image.

[0171] In the specific implementation of this embodiment, first, for each mask image block, an autoregressive decoder is used to perform feature decoding on the mask image block encoding features of the mask image block to achieve image block reconstruction of the mask image block, and obtain a reconstructed image block corresponding to each mask image block. Then, multiple reconstructed image blocks are combined into a complete medical image to obtain a predicted medical image.

[0172] It should be noted that, since the mask image blocks are arranged in sequence, when reconstructing the image blocks, the mask image blocks are also reconstructed in sequence according to the arrangement order.

[0173] In step S806 of some embodiments, the pixel error is used to indicate the difference between the pixel values ​​of a pixel point at the same position on the predicted medical image and the first sample medical image.

[0174] In the specific implementation of this embodiment, first, for each pixel point at the same position on the predicted medical image and the first sample medical image, the first pixel value of the pixel point on the predicted medical image and the second pixel value of the pixel point on the first sample medical image are subtracted to obtain a pixel difference value. Then, the pixel difference values ​​of all pixels are averaged to obtain a pixel error between the predicted medical image and the first sample medical image. Further, the pixel errors of all first sample medical images are averaged to obtain a model loss value. Finally, the parameters of the first model are adjusted according to the model loss value, and the process returns to step S801, and iterative training is performed according to the above process until the model loss value of a certain iteration round is less than a preset threshold value, the model parameters of the iteration round are used as the final model parameters, and the first model with the final model parameters is determined as the trained first model.

[0175] The advantage of this embodiment is that the first model is trained based on self-supervised learning, which can reduce the difficulty of obtaining training data. The first sample medical images obtained are first masked so that each first sample medical image has a part of the occluded area, and then the first model predicts a complete, unobstructed medical image (i.e., a predicted image) based on the image representation features of the masked medical image. The first model is trained based on the difference in pixels between the predicted image and the first sample medical image, so that the difference between the predicted image output by the first model and the first medical image is as small as possible, thereby achieving the effect of using unlabeled data to train a first model with better image processing capabilities. This method reduces the difficulty of training the first model while improving the training effect of the first model, so that the trained first model has better image feature extraction capabilities and image reconstruction capabilities.

[0176] See also Fig. 9In some embodiments, the specific process of performing mask processing on the first sample medical image to obtain the mask medical image may include but is not limited to steps S901 to S902:

[0177] Step S901, determining a target image region in a first sample medical image;

[0178] Step S902: for each pixel point in the target image area, the pixel value of the pixel point is replaced by the first value to obtain a masked medical image.

[0179] In step S901 of some embodiments, the target image region is used to indicate a region in the first sample medical image that needs to be masked.

[0180] In the specific implementation of this embodiment, determining the target image area in the first sample medical image includes but is not limited to the following steps:

[0181] determining an image size of the first sample medical image, and determining a model accuracy of the medical image processing model;

[0182] Determine the area of ​​the target image region based on the image size and model accuracy;

[0183] Based on the area of ​​the region, a target image region is determined in the first sample medical image.

[0184] Among them, the image size is used to indicate the size of the first sample medical image, the model accuracy is used to indicate the image processing accuracy to be achieved by the first model, and the region area is used to indicate the size of each target image region to be masked in the first sample medical image.

[0185] Specifically, first, the size of the first sample medical image is detected to obtain the image size of the first sample medical image. Then, since the model accuracy of the medical image processing model is pre-set, the preset model accuracy can be called with authorization. Further, the model accuracy is input into a preset function, and the output result of the preset function is used as the size ratio, wherein the preset function is an increasing function with the model accuracy as the independent variable and the size ratio as the dependent variable. Then, the image size and the size ratio are multiplied to obtain the area of ​​the target image area. Finally, according to the area of ​​the area, one or more image areas with an area size equal to the area of ​​the area are randomly selected in the first sample medical image as the target image area, wherein when multiple target image areas are selected, the target image areas do not overlap.

[0186] In step S902 of some embodiments, for each pixel point in the target image area, the pixel value of the pixel point is replaced by the first numerical value so that the pixel value of each pixel point in the target image area is the first numerical value, so as to mask the first sample medical image and obtain a masked medical image, wherein the masked target image area is the occluded area of ​​the masked medical image.

[0187] For example, in the template area of ​​the first sample medical image, the pixel value of each pixel is replaced with 0 to form a mask medical image.

[0188] like Fig.10 As shown in FIG. 1 , a specific process of masking a first sample medical image with an image size of 8×8 is performed. First, according to the image size and model accuracy, the area of ​​the target image region is determined to be 3×3. Then, two 3×3 image regions are randomly selected in the first sample medical image as the target image regions. Finally, for each target image region, the pixel values ​​of the 9 pixels in the target image region are replaced with 0 to obtain a masked medical image.

[0189] The advantage of this embodiment is that when determining the target image area in the first sample medical image, the image size of the first sample medical image and the model accuracy of the medical image processing model to be acquired are introduced as reference indicators, and the image size and model accuracy are comprehensively considered by function calculation and proportional distribution to determine the area size of the target image area, thereby achieving flexible adjustment of the area, location and number of areas of the target image area, and improving the mask accuracy and mask rationality of the first sample medical image.

[0190] See also Fig.11 The present application also provides a medical image processing device that can implement the above-mentioned medical image processing method. The medical image processing device includes:

[0191] An image acquisition unit 1110, configured to acquire a target medical image;

[0192] A segmentation unit 1120, configured to segment the target medical image to obtain a plurality of sequentially arranged medical image blocks;

[0193] A feature extraction unit 1130 is used to extract features of each medical image block to obtain image block representation features of the medical image block;

[0194] An integration unit 1140 is used to integrate the image block representation features of multiple medical image blocks to obtain a target image representation feature of a target medical image;

[0195] The encoding unit 1150 is used to encode the target image representation features based on the encoder of the preset medical image processing model to obtain the medical image encoding features of the target medical image, wherein the medical image processing model also includes an autoregressive decoder; the medical image processing model is obtained by adjusting a pre-trained first model, the first model is obtained by training according to the pixel error between the first sample medical image and the predicted medical image, the predicted medical image is obtained by inputting the masked medical image into the first model, and the masked medical image is obtained by performing masking processing on the first sample medical image;

[0196] The processing unit 1160 is used to perform segmentation processing on the medical image coding features based on the autoregressive decoder to obtain the image category of each medical image block.

[0197] In one embodiment, the feature extraction unit 1130 is used to:

[0198] Performing convolution processing on the medical image blocks to obtain image block convolution features of each medical image block;

[0199] The convolutional features of the image block are activated based on a predetermined activation function to obtain the representation features of the image block.

[0200] In one embodiment, the encoding unit 1150 is used to:

[0201] Based on the encoder, a multi-head attention calculation is performed on the target image representation features to obtain an attention calculation result, wherein the attention calculation result is used to indicate the degree of correlation between medical image blocks in the target medical image;

[0202] Map the attention calculation result to a predetermined vector space to obtain a target mapping vector;

[0203] Based on the residual concatenation of the target mapping vector and the target image representation features, the medical image coding features are obtained.

[0204] In one embodiment, the medical image coding feature includes image block coding features of each medical image block, the plurality of medical image blocks are arranged in a first order, and the plurality of medical image blocks include a target image block;

[0205] The processing unit 1160 is used for:

[0206] Determine the image category of each image block that is arranged before the target image block in the first order;

[0207] Based on the image categories of each image block arranged before the target image block and the image block coding feature of the target image block, segmentation processing is performed through an autoregressive decoder to obtain the target image category of the target image block.

[0208] In one embodiment, the first model is trained in the following manner:

[0209] Acquire a first sample medical image, wherein the first sample medical image does not have an image segmentation label;

[0210] Performing mask processing on the first sample medical image to obtain a masked medical image, and performing image segmentation on the masked medical image to obtain a plurality of masked medical image blocks;

[0211] Extracting features from each masked medical image block to obtain masked image block representation features of the masked medical image block, and integrating multiple masked image block representation features to obtain masked image representation features;

[0212] The encoder based on the first model encodes the representation features of the mask image to obtain a mask image encoding feature;

[0213] Performing image prediction on the encoded features of the mask image through the autoregressive decoder of the first model to obtain a predicted medical image;

[0214] For each first sample medical image, a pixel error between the predicted medical image and the first sample medical image is calculated, and the first model is trained according to the pixel error.

[0215] In one embodiment, performing mask processing on the first sample medical image to obtain a masked medical image includes:

[0216] determining a target image region in a first sample medical image;

[0217] For each pixel point in the target image area, the pixel value of the pixel point is replaced by the first value to obtain a masked medical image.

[0218] The medical image processing model is obtained by adjusting the first model in the following way:

[0219] Obtain a second sample medical image and an image segmentation label of the second sample medical image;

[0220] The encoder based on the first model performs encoding processing on the second sample image representation feature of the second sample medical image to obtain a sample image encoding feature of the second sample medical image;

[0221] The autoregressive decoder based on the first model performs segmentation processing on the encoded features of the sample image to obtain a sample prediction segmentation result of the first sample medical image;

[0222] Based on the sample prediction segmentation results of multiple second sample medical images and the comparison of image segmentation labels, the parameters of the first model are adjusted to obtain a medical image processing model.

[0223] The specific implementation of the medical image processing device is basically the same as the specific implementation of the above-mentioned medical image processing method, and will not be repeated here.

[0224] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned medical image processing method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0225] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0226] The processor 1201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0227] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the medical image processing method of the embodiments of this application;

[0228] Input / output interface 1203, used to implement information input and output;

[0229] The communication interface 1204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0230] A bus 1205 that transmits information between various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0231] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0232] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned medical image processing method.

[0233] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0234] The medical image processing method, medical image processing device, electronic device and computer-readable storage medium provided in the embodiments of the present application obtain a target medical image; perform image segmentation on the target medical image to obtain a plurality of sequentially arranged medical image blocks; perform feature extraction on each medical image block to obtain image block representation features of the medical image block, which can map the image feature information of the medical image block from the pixel space to the vector space, and then integrate the image block representation features of the plurality of medical image blocks to obtain the target image representation features of the target medical image, which can improve the feature quality of the target image representation features. Furthermore, an encoder based on a preset medical image processing model encodes the target image representation features to obtain the medical image encoding features of the target medical image, and an autoregressive decoder based on the medical image processing model segmentates the medical image encoding features to obtain the image categories of each medical image block. This method can improve the segmentation accuracy of medical images by combining the encoder and the autoregressive decoder. In addition, the medical image processing model is obtained by adjusting the pre-trained first model, the first model is trained based on the pixel error between the first sample medical image and the predicted medical image, the predicted medical image is obtained by inputting the masked medical image into the first model, and the masked medical image is obtained by masking the first sample medical image. Unsupervised training is used in training the first model, which can effectively reduce the difficulty of obtaining training samples and is conducive to improving the model training effect of the first model. After training the first model, a supervised training model fine-tuning method is used to adjust the medical image processing model based on the first model, which can effectively reduce the model training difficulty of the medical image processing model and improve the accuracy of the medical image processing model in processing medical images.

[0235] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0236] It can be understood by those skilled in the art that Figure 1-10 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0237] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0238] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0239] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0240] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0245] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A medical image processing method, characterized in that: The method comprises: Acquire a target medical image; Performing image segmentation on the target medical image to obtain a plurality of sequentially arranged medical image blocks; For each of the medical image blocks, extracting features of the medical image block to obtain image block representation features of the medical image block; Integrating the image block representation features of the plurality of medical image blocks to obtain a target image representation feature of the target medical image; The encoder based on a preset medical image processing model encodes the target image representation features to obtain the medical image encoding features of the target medical image, wherein the medical image processing model also includes an autoregressive decoder; the medical image processing model is obtained by adjusting a pre-trained first model, the first model is obtained by training according to the pixel error between a first sample medical image and a predicted medical image, the predicted medical image is obtained by inputting a masked medical image into the first model, and the masked medical image is obtained by performing masking processing on the first sample medical image; The medical image coding features are segmented based on the autoregressive decoder to obtain image categories of each medical image block.

2. The method according to claim 1, characterized in that The first model is trained in the following way: Acquire a first sample medical image, wherein the first sample medical image does not have an image segmentation label; Performing mask processing on the first sample medical image to obtain a masked medical image, and performing image segmentation on the masked medical image to obtain a plurality of masked medical image blocks; Performing feature extraction on each of the masked medical image blocks to obtain a masked image block representation feature of the masked medical image block, and integrating a plurality of the masked image block representation features to obtain a masked image representation feature; Based on the encoder of the first model, the mask image representation feature is encoded to obtain a mask image encoding feature; Performing image prediction on the encoded features of the mask image through the autoregressive decoder of the first model to obtain the predicted medical image; For each of the first sample medical images, a pixel error between the predicted medical image and the first sample medical image is calculated, and the first model is trained according to the pixel error.

3. The method according to claim 2, characterized in that The performing mask processing on the first sample medical image to obtain a masked medical image includes: Determining a target image region in the first sample medical image; For each pixel point in the target image area, the pixel value of the pixel point is replaced by the first value to obtain the masked medical image.

4. The method according to claim 1, characterized in that: The medical image processing model is obtained by adjusting the first model in the following manner: Acquire a second sample medical image and an image segmentation label of the second sample medical image; encoding the second sample image representation feature of the second sample medical image by an encoder based on the first model to obtain a sample image encoding feature of the second sample medical image; The autoregressive decoder based on the first model performs segmentation processing on the encoding features of the sample image to obtain a sample prediction segmentation result of the first sample medical image; Based on the comparison of the sample prediction segmentation results of the plurality of second sample medical images and the image segmentation labels, the parameters of the first model are adjusted to obtain the medical image processing model.

5. The method according to claim 1, characterized in that The step of extracting features from each medical image block to obtain image block representation features of the medical image block includes: Performing convolution processing on the medical image blocks to obtain image block convolution features of each medical image block; The image block convolution feature is activated based on a predetermined activation function to obtain the image block representation feature.

6. The method according to claim 1, characterized in that The encoder based on the pre-prepared medical segmentation model performs encoding processing on the target image representation feature to obtain the medical image encoding feature of the target medical image, including: Performing a multi-head attention calculation on the target image representation feature based on the encoder to obtain an attention calculation result, wherein the attention calculation result is used to indicate the degree of correlation between the medical image blocks in the target medical image; Mapping the attention calculation result to a predetermined vector space to obtain a target mapping vector; The medical image coding feature is obtained based on residual splicing of the target mapping vector and the target image representation feature.

7. The method according to any one of claims 1 to 6, characterized in that: The medical image coding features include image block coding features of each of the medical image blocks, the plurality of medical image blocks are arranged in a first order, and the plurality of medical image blocks include a target image block; The segmenting process of the medical image coding features based on the autoregressive decoder to obtain the image category of each of the medical image blocks includes: Determining the image category of each of the image blocks that precede the target image block in the first sequence; Based on the image categories of each of the image blocks arranged before the target image block and the image block coding features of the target image block, segmentation processing is performed by the autoregressive decoder to obtain the target image category of the target image block.

8. A medical image processing device, characterized in that: The device comprises: An image acquisition unit, used for acquiring a target medical image; A segmentation unit, used for segmenting the target medical image to obtain a plurality of sequentially arranged medical image blocks; A feature extraction unit, configured to extract features from each medical image block to obtain image block representation features of the medical image block; an integration unit, configured to integrate the image block representation features of the plurality of medical image blocks to obtain a target image representation feature of the target medical image; an encoding unit, configured to encode the target image representation features based on an encoder of a preset medical image processing model to obtain medical image encoding features of the target medical image, wherein the medical image processing model further includes an autoregressive decoder; the medical image processing model is obtained by adjusting a pre-trained first model, the first model is obtained by training based on a pixel error between a first sample medical image and a predicted medical image, the predicted medical image is obtained by inputting a masked medical image into the first model, and the masked medical image is obtained by performing masking processing on the first sample medical image; A processing unit is used to perform segmentation processing on the medical image coding features based on the autoregressive decoder to obtain the image category of each of the medical image blocks.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the medical image processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the medical image processing method according to any one of claims 1 to 7 is implemented.