Image processing method, device, equipment, storage medium and program product

By performing image block division and hidden state cyclic encoding of medical images, combined with Gaussian distribution feature reconstruction and nonlinear transformation, the problem of low disease grading accuracy in the prior art is solved, and higher disease grading accuracy and model are achieved Generalization.

CN118608873BActive Publication Date: 2024-11-08TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411067960.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-08
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

In the medical image disease rating, feature extraction affects the accuracy, resulting in low accuracy of disease rating. Especially when the distribution differences between different domains and data sets are different, it is difficult to achieve effective inter-domain consistency and data augmentation. .

Method used

By dividing the image to be processed, cyclically encoded based on the hidden state, the second feature sequence is obtained, and the feature reconstruction is performed using Gaussian distribution, combining nonlinear transformation and feature weight sequence fusion, improving the accuracy and generalization of disease grading .

Benefits of technology

It improves the accuracy of disease grading and generalization of the model, and can accurately identify and grade the most serious lesions in medical images between different domains, enhancing the robustness of the disease grading model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118608873B_ABST
    Figure CN118608873B_ABST
Patent Text Reader

Abstract

The present application provides an image processing method, device, equipment, storage medium and program product, which are applied to various scenarios of disease grading based on medical images; the image processing method includes: acquiring features of a target image block sequence of the image to be processed to obtain a first feature sequence; traversing the first feature sequence, performing the following processing on the kth first feature traversed, K represents the total number of target image blocks in the target image block sequence: updating the #imgabs0#th hidden state based on the kth first feature to obtain the kth hidden state, when #imgabs1#, the #imgabs2#th hidden state is the initial hidden state; mapping the kth second feature based on the kth hidden state; obtaining a second feature sequence based on the K second features obtained by traversing the first feature sequence; grading the disease based on the second feature sequence to obtain the target disease level. Through the present application, the accuracy of disease grading can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to image processing technology in the field of computing applications, and in particular to an image processing method, device, equipment, storage medium and program product. Background Art

[0002] During the development of a disease, it is often necessary to grade medical images; disease grading refers to the process of determining the severity of a disease; for example, diabetic retinopathy includes 5 disease grades, which are ranked from 0 to 4 in order of severity, with grade 0 being no diabetic retinopathy and grade 4 being the proliferative stage.

[0003] Generally speaking, in order to achieve disease classification, features are usually extracted from the entire medical image, and then the disease level is determined based on the extracted features; however, in the above-mentioned disease classification process, feature extraction is performed on the entire medical image, which affects the accuracy of the extracted features, and then, when the disease is classified based on the extracted features, the accuracy of the disease classification is affected. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of disease grading.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an image processing method, the method comprising:

[0007] Perform feature acquisition on a target image block sequence of the image to be processed to obtain a first feature sequence;

[0008] The first feature sequence is traversed, and the following processing is performed on the kth first feature traversed: , and k is an integer, K represents the total number of target image blocks in the target image block sequence:

[0009] Based on the kth first feature, update the hidden state, get the kth hidden state, when At that time, The hidden state is the initial hidden state. At that time, The hidden state is a hidden state of the target image block;

[0010] Based on the kth hidden state, mapping the kth second feature;

[0011] Obtaining a second feature sequence based on the K second features obtained by traversing the first feature sequence;

[0012] Disease classification is performed based on the second feature sequence to obtain a target disease grade.

[0013] The present application provides an image processing device, the image processing device comprising:

[0014] A feature acquisition module, used for acquiring features of a target image block sequence of the image to be processed to obtain a first feature sequence;

[0015] A cyclic encoding module is used to traverse the first feature sequence to obtain the kth first feature traversed, , and k is an integer, K represents the total number of target image blocks in the target image block sequence:

[0016] The cyclic encoding module is further used to update the kth first feature based on the kth first feature. hidden state, get the kth hidden state, when At that time, The hidden state is the initial hidden state. At that time, The hidden state is a hidden state of the target image block;

[0017] The cyclic encoding module is further used to map the kth second feature based on the kth hidden state;

[0018] The cyclic encoding module is further used to obtain a second feature sequence based on the K second features obtained by traversing the first feature sequence;

[0019] A disease grading module is used to perform disease grading based on the second feature sequence to obtain a target disease grade.

[0020] In an embodiment of the present application, the image processing device also includes a sequence determination module, which is used to divide the image to be processed into image blocks to obtain a first image block sequence and a second image block sequence, wherein the first image block sequence and the second image block sequence are in opposite order; and traverse the first image block sequence and the second image block sequence to obtain the target image block sequence.

[0021] In an embodiment of the present application, the disease grading module is further used to perform disease grading based on the second feature sequence of the first image block sequence and the second feature sequence of the second image block sequence to obtain the target disease grade.

[0022] In the embodiment of the present application, the cyclic encoding module is further used to obtain a discretized feature parameter corresponding to the continuous feature parameter based on a discretized interval parameter.

[0023] In the embodiment of the present application, the cyclic encoding module is further used to encode the kth first feature and the kth first feature based on the discretized feature parameter. The hidden states are integrated to obtain the kth hidden state, the kth first feature and the kth The integration of the hidden states is used to achieve the An update of the hidden state.

[0024] In the embodiment of the present application, the disease grading module is further used to traverse the second feature sequence and perform the following processing on the kth second feature traversed: based on the kth second feature, calculate the probability that the kth target image block belongs to the jth Gaussian distribution to obtain the matching probability, , and j is an integer, J represents the total number of the Gaussian distributions of the image to be processed; feature reconstruction is performed based on the J matching probabilities of the kth second feature and the J first Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified; disease classification is performed based on the K image block features to be classified obtained by traversing the second feature sequence to obtain the target disease level.

[0025] In an embodiment of the present application, the disease grading module is also used to calculate J posterior probabilities between the kth second feature and the J first Gaussian parameters respectively; integrate the J posterior probabilities to obtain a posterior probability integration result; obtain the target posterior probability corresponding to the jth Gaussian parameter of the Gaussian distribution and the kth second feature from the J posterior probabilities; and determine the weight of the target posterior probability in the posterior probability integration result as the matching probability.

[0026] In an embodiment of the present application, the image processing device also includes a feature reconstruction module, which is used to iteratively perform the following update processing on the first Gaussian parameter: based on the K matching probabilities for the jth first Gaussian parameter obtained by traversing the second feature sequence, and the second feature sequence, calculate the second Gaussian parameter of the jth Gaussian distribution; update the matching probability in combination with the second Gaussian parameter and the kth second feature to obtain the target matching probability; when the iteration of the update processing is completed, perform feature reconstruction based on the J target matching probabilities for the kth second feature and the J second Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified.

[0027] In an embodiment of the present application, the feature reconstruction module is also used to integrate the K matching probabilities for the jth first Gaussian parameter obtained by traversing the second feature sequence to obtain a matching probability integration result; integrate the K matching probabilities with the second feature sequence to obtain a probability feature integration result; and obtain the second Gaussian parameter that is positively correlated with the probability feature integration result and negatively correlated with the matching probability integration result.

[0028] In an embodiment of the present application, the disease grading module is further used to perform nonlinear transformation based on the first feature sequence to obtain a feature weight sequence.

[0029] In an embodiment of the present application, the disease grading module is further used to fuse the second feature sequence with the feature weight sequence to obtain a feature sequence to be integrated; integrate the first feature sequence and the feature sequence to be integrated to obtain a target feature sequence; and perform disease grading based on the target feature sequence to obtain the target disease grade.

[0030] In an embodiment of the present application, the target disease level is obtained through a disease grading model, and the image processing device also includes a model training module, which is used to use the model to be trained to grade the disease of the image sample to obtain an estimated disease level, and the model to be trained is an artificial neural network model to be trained for performing disease grading on medical images; based on the difference between the estimated disease level and the disease level label of the image sample, the target loss function value is calculated; based on the target loss function value, the model to be trained is trained to obtain the disease grading model.

[0031] In an embodiment of the present application, the model training module is further used to obtain a third Gaussian parameter set updated by each of the M image samples to obtain M third Gaussian parameter sets, wherein the third Gaussian parameter set includes J third Gaussian parameters corresponding to J Gaussian distributions. , and M is an integer; integrating the M third Gaussian parameter sets to determine a Gaussian parameter integration set; combining the initial Gaussian parameter set of the disease classification model and the Gaussian parameter integration set to obtain a Gaussian parameter update set; adjusting the state space parameters of the model to be trained based on the target loss function value to obtain state space update parameters; combining the Gaussian parameter update set and the state space update parameters to obtain the disease classification model.

[0032] In the embodiment of the present application, the feature acquisition module is further used to obtain the image to be processed in response to a disease classification request.

[0033] In the embodiment of the present application, the disease grading module is further used to present the target disease grade corresponding to the image to be processed.

[0034] An embodiment of the present application provides an electronic device for image processing, the electronic device comprising:

[0035] A memory for storing computer executable instructions or computer programs;

[0036] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0037] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program are used to implement the image processing method provided in the embodiment of the present application when executed by a processor.

[0038] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the image processing method provided by the embodiment of the present application is implemented.

[0039] The embodiments of the present application have at least the following beneficial effects: when performing disease classification on an image to be processed, the image to be processed is divided into a target image block sequence, and a first feature sequence corresponding to the target image block sequence is cyclically updated based on a hidden state to implement cyclic encoding of the target image block sequence to obtain a second feature sequence; because the most serious lesion information can be transferred between image blocks through the hidden state, the second feature sequence can accurately characterize the most serious lesion in the image to be processed; and further, when performing disease classification based on the second feature sequence, the accuracy of disease classification can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is an exemplary disease classification diagram;

[0041] Figure 2 is another exemplary disease classification diagram;

[0042] Figure 3 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0043] Figure 4 This is a method provided by the embodiment of the present application. Figure 3 A schematic diagram of the structure of the terminal in FIG.

[0044] Figure 5 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 1 ;

[0045] Figure 6This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 2 ;

[0046] Figure 7 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 3 ;

[0047] Figure 8 is an exemplary model training flow chart provided in an embodiment of the present application;

[0048] Fig. 9 is a schematic diagram of an exemplary disease development process provided in an embodiment of the present application;

[0049] Fig.10 is an exemplary disease level diagram provided in an embodiment of the present application;

[0050] Fig.11 is an exemplary medical image clustering schematic diagram provided in an embodiment of the present application;

[0051] Fig.12 is an exemplary disease classification diagram provided in an embodiment of the present application;

[0052] Fig.13 is a schematic diagram of an exemplary feature processing module provided in an embodiment of the present application;

[0053] Fig.14 is an exemplary feature reconstruction schematic diagram provided in an embodiment of the present application;

[0054] Fig.15 This is an exemplary disease grading assessment index comparison chart provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0056] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] In the following description, the terms "first\second\third, etc." are used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third, etc." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0058] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0059] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0060] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0061] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0062] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In the embodiment of the present application, disease classification can be achieved through AI.

[0063] 2) Artificial neural network model is a mathematical model that imitates the structure and function of biological neural network. In the embodiment of the present application, both the model to be trained and the disease classification model are artificial neural network models.

[0064] 3) Domain refers to a medical center or hospital. In the embodiment of the present application, the disease classification model used to implement disease classification can achieve domain-generalized disease classification. Domain-generalized disease classification means that the model to be trained is trained on a training data set corresponding to one or more domains, and is tested or applied on data corresponding to domains outside the training data set. In addition, domain-generalized disease classification can improve the generalization of the disease classification model.

[0065] 4) Hidden State: a state used to store and transmit sequence element information when processing sequence data; in the embodiment of the present application, the most serious lesion in the target image block is stored and transmitted through the hidden state.

[0066] It should be noted that, in order to achieve disease classification, the entire medical image is usually feature extracted, and then the disease level is determined based on the extracted features; however, the above disease classification process is achieved by extracting features from the entire medical image, which affects the accuracy of the extracted features, and then, when the disease is classified based on the extracted features, the accuracy of the disease classification is affected. In addition, in order to achieve disease classification, inter-domain consistency and data augmentation methods can also be used.

[0067] When using inter-domain consistency, see Figure 1 , Figure 1 is an exemplary disease classification diagram; Figure 1 As shown, during the training process, medical images 1-11 of multiple domains (for example, 1 to 3 domains) are used as input data and input into a model 1-2 (for example, a residual network ResNet50) to obtain outputs 1-3, and prediction results 1-4 are obtained based on the outputs 1-3; here, the gradient of the last fully connected layer of the model 1-2 is constrained so that the gradient covariance matrices of different domains are consistent, thereby improving the inter-domain consistency of the features, as shown in formulas (1) and (2).

[0068] (1);

[0069] (2);

[0070] in, Represents the final loss function value of model 1-2, Indicates the total number of domains, represents the loss function value corresponding to the cth domain, represents the weight, represents the gradient covariance matrix difference of the cth domain; represents the gradient covariance matrix in domain c, represents the gradient covariance matrix of all samples, Represents the calculation process of the loss function.

[0071] Continue to see Figure 1 During the test, the medical images 1-12 of other domains are input into the trained model 1-2 to obtain the output 1-8, and then the prediction result 1-9 is obtained based on the output 1-8, so as to determine the test result of the trained model 1-2 based on the prediction result 1-9.

[0072] It should be noted that although the inter-domain consistency method constrains the gradient of the final fully connected layer of the model during training to make the gradient covariance matrices of different domains consistent. However, different datasets have different sample proportions of different disease levels; for example, the samples of the five disease levels in dataset 1 are evenly distributed, while the samples of disease level 1 in dataset 2 account for 60% of the total number of samples; and because the differences in sample distribution of datasets will be reflected in the covariance matrix, it is impossible to achieve the consistency of the gradient covariance matrices of different domains, which in turn affects the accuracy of disease grading.

[0073] When using data augmentation, see Figure 2 , Figure 2 is another exemplary disease classification diagram; Figure 2 As shown, the medical image 2-11 is first transformed (for example, brightness adjustment, transparency adjustment) to obtain the medical image 2-12; the medical image 2-12 is then processed in turn through the modules 2-21 and 2-22 in the model 2-2 to obtain the feature 2-13; then, the disease level 2-3 is obtained based on the feature 2-13; finally, two loss function values ​​(loss function value 2-41 and loss function value 2-42) are calculated based on the disease level 2-3 and the label, and the model 2-2 is trained based on the two loss function values.

[0074] The data augmentation based method simulates the image style differences between domains through image transformation. However, image transformation cannot guarantee the authenticity of the generated medical images, and the derived model is overfitted, which affects the accuracy of disease classification.

[0075] Based on this, the embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of disease grading. The following describes an exemplary application of an electronic device for image processing (referred to as an image processing device) provided in an embodiment of the present application. The image processing device provided in an embodiment of the present application can be implemented as various types of terminals such as robots, smart phones, smart watches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart car devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable game devices and smart speakers. It can also be implemented as a server, or it can be implemented as a combination of the two. Below, an exemplary application of the image processing device when it is implemented as a terminal will be described.

[0076] See also Figure 3 , Figure 3 Schematic diagram of the architecture of the image processing system provided in the embodiment of the present application; Figure 3 As shown, to support an image processing application, in the image processing system 100, the terminal 400 (the terminal 400-1 and the terminal 400-2 are shown as examples) is connected to the server 200 via the network 300; the network 300 can be a wide area network or a local area network, or a combination of the two; the server 200 is used to provide services to the terminal 400 via the network 300, for example, providing storage services for the corresponding information of the image to be processed and the target disease level. In addition, the image processing system 100 also includes a database 500 for providing data support to the server 200; and, Figure 3 What is shown in FIG. 5 is a case where the database 500 is independent of the server 200. In addition, the database 500 may also be integrated in the server 200, which is not limited in the embodiment of the present application.

[0077] Terminal 400 is used to obtain features of a target image block sequence of the image to be processed to obtain a first feature sequence; traverse the first feature sequence, and perform the following processing on the kth first feature traversed: update the kth first feature based on the kth first feature; hidden states to obtain the kth hidden state; based on the kth hidden state, map the kth second feature; based on the K second features obtained by traversing the first feature sequence, obtain a second feature sequence; perform disease grading based on the second feature sequence to obtain a target disease level; present the target disease level corresponding to the image to be processed (exemplarily showing the image to be processed and the target disease level: level 3 presented by the graphical interface 400-11, and the image to be processed and the target disease level: level 3 presented by the graphical interface 400-21).

[0078] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0079] See also Figure 4 , Figure 4 This is a method provided by the embodiment of the present application. Figure 3 The schematic diagram of the terminal structure in FIG. Figure 4 As shown, the terminal 400 includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 4 Various buses are labeled as bus system 440 .

[0080] The processor 410 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0081] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0082] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0083] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiment of the present application is intended to include any suitable type of memory.

[0084] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0085] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0086] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), etc.;

[0087] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0088] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0089] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 4 An image processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, including the following software modules: a feature acquisition module 4551, a cyclic encoding module 4552, a disease classification module 4553, a sequence determination module 4554, a feature reconstruction module 4555, and a model training module 4556. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0090] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the image processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0091] In some embodiments, the terminal or server can implement the image processing method provided in the embodiments of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions may be microprogram-level commands, machine instructions or software instructions. The computer program may be a native program or software module in the operating system; it may be a native application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a disease grading APP or a diagnosis APP; it may also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions may be instructions in any form, and the above-mentioned computer program may be an application, module or plug-in in any form.

[0092] The image processing method provided by the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the image processing device provided by the embodiment of the present application. In addition, the image processing method provided by the embodiment of the present application is applied to various scenarios of disease grading based on medical images.

[0093] See also Figure 5 , Figure 5 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 1 ;in, Figure 5 The execution subject of each step in the above is the image processing device. Figure 5 The steps shown are explained.

[0094] Step 101: Acquire features of a target image block sequence of an image to be processed to obtain a first feature sequence.

[0095] In an embodiment of the present application, when a request is made to grade a disease on an image to be processed, the image processing device divides the image to be processed into image blocks, and based on the order in which the image blocks appear in the image to be processed, combines the divided image blocks into data in the form of a sequence, thereby obtaining a target image block sequence. Since the target image block sequence is composed of a combination of target image blocks, the image processing device performs feature acquisition on the target image block sequence, that is, performs feature acquisition on each target image block in the target image block sequence, and the feature of the target image block obtained is the first feature. Based on the sequence order of the target image blocks in the target image block sequence, the first features obtained are combined into a sequence, thereby obtaining a first feature sequence.

[0096] It should be noted that the image to be processed is a medical image to be graded for disease, such as a medical image of a fracture, a medical image of diabetic retinopathy, etc. The target image block sequence is a sequence consisting of K target image blocks corresponding to the image to be processed, for example, the target image block sequence is an image block sequence sorted from the upper left corresponding to the image presentation orientation; wherein K is an integer greater than 1, and K represents the total number of target image blocks in the target image block sequence; the K target image blocks are the image block division results of the image to be processed; in addition, when the image to be processed includes one image block sequence, the target image block sequence is this image block sequence; and when the image to be processed includes multiple image block sequences, the target image block sequence is any one of the multiple image block sequences.

[0097] Here, feature acquisition can be feature extraction, or it can be a process of acquiring features based on the features of image blocks that have completed feature extraction, and the embodiments of the present application do not limit this; for example, when an image block sequence is obtained for the image to be processed, or the target icon block sequence is the first image block sequence to be feature acquired among multiple image block sequences, feature acquisition is feature extraction; and when the target image block sequence is an image block sequence among multiple image block sequences except the first image block sequence that has completed feature extraction, feature acquisition is a process of acquiring features based on the features of image blocks that have completed feature extraction.

[0098] It should also be noted that the first feature sequence corresponds to the target image block sequence, that is, a first feature in the first feature sequence is a feature of the target image block in the corresponding sequence order in the target image block sequence; in addition, the first feature can be an initial feature such as an embedded representation of the target image block.

[0099] In the embodiment of the present application, the image processing device traverses the first feature sequence and performs the following processing (steps 102 to 104) on the kth first feature traversed; wherein k is a variable for traversing the first feature sequence, , and k is an integer.

[0100] It should be noted that the image processing is based on the sequence order of the first feature sequence, and encodes each first feature to achieve cyclic encoding of the first feature sequence; thus, the image processing device uses k to traverse the first feature sequence.

[0101] Step 102: Update the kth first feature based on the kth first feature. hidden states, and get the kth hidden state.

[0102] In the embodiment of the present application, the image processing device updates the kth first feature based on the kth first feature. hidden state, and update the The first hidden state is called the kth hidden state. Here, the image processing device updates the first feature based on the kth first feature. The hidden state is to store the feature of the most serious lesion represented by the kth first feature into the in a hidden state.

[0103] It should be noted that when At that time, The hidden state is the initial hidden state, where the initial hidden state is a default or preset hidden state or a hidden state learned through an artificial neural network model. At that time, The hidden state is The hidden state of the target image patch, and the The hidden state of the target image block is based on the The hidden state updated by the first feature.

[0104] In the embodiment of the present application, when the image processing device obtains the discretized feature parameters corresponding to the continuous feature parameters based on the discretized interval parameters, the image processing device updates the kth first feature based on the kth first feature in step 102. hidden state, and obtain the kth hidden state, including: based on the discretized feature parameters, the kth first feature and the kth hidden states are integrated to obtain the kth hidden state; thus, the kth first feature and the kth The integration of hidden states is used to achieve the A hidden state update.

[0105] It should be noted that the discretization interval parameter refers to the interval used to achieve discretization, which can be preset or a parameter trained by an artificial neural network model, etc., and the embodiments of the present application do not limit this. When the image processing device includes a continuous feature parameter, since the continuous feature parameter refers to a parameter for updating a hidden state based on a continuous feature, the image processing device uses the discretization interval parameter as a parameter for converting from continuous to discrete, and converts the continuous feature parameter into a discrete feature parameter; wherein the discretization feature parameter refers to a parameter for updating a hidden state based on a discrete feature; in addition, the continuous feature parameter includes a first parameter for the feature, and also includes a second parameter for the hidden state, so that the discretization feature parameter includes a third parameter determined by the first parameter and the discretization interval parameter, and also includes a fourth parameter determined by the second parameter and the discretization interval parameter; therefore, the image processing device fuses the third parameter with the kth first feature to obtain a first fusion result, and fuses the third parameter with the kth first feature to obtain a first fusion result. The hidden states are fused to obtain a second fusion result, and finally, the combination result of the first fusion result and the second fusion result is determined as the kth hidden state.

[0106] Step 103: Map the kth second feature based on the kth hidden state.

[0107] It should be noted that the image processing device includes a mapping parameter for obtaining the encoding result of the corresponding target image block based on the hidden state, so that after the image processing device obtains the kth hidden state, it maps the kth hidden state to the encoding result of the kth target image block based on the mapping parameter, and obtains the kth second feature. Among them, the mapping parameter can be preset, or it can be a parameter trained by an artificial neural network model, etc., which is not limited in the embodiment of the present application.

[0108] It can be understood that by storing the feature of the most serious lesion in the kth first feature to the hidden states to obtain the kth hidden state, and then obtain the kth second feature based on the kth hidden state; such cyclic encoding makes the kth second feature include the most serious lesions in the first k target image blocks, which can improve the recognition accuracy of the most serious lesions.

[0109] Step 104: Obtain a second feature sequence based on the K second features obtained by traversing the first feature sequence.

[0110] It should be noted that since the image processing device can obtain a second feature for each first feature traversed, K second features can be obtained for the K first features in the first feature sequence. Here, the image processing device combines the K second features into a second feature sequence based on the sequence order of the first feature sequence, so that the second feature sequence corresponds to the first feature sequence based on the sequence order.

[0111] Step 105: Perform disease classification based on the second feature sequence to obtain a target disease grade.

[0112] In the embodiment of the present application, the image processing device may use the second feature sequence as the final feature corresponding to the image to be processed to perform disease classification based on the second feature sequence; the second feature sequence may also be optimized to perform disease classification based on the optimized second feature sequence; the embodiment of the present application is not limited to this. Here, the image processing device refers to the disease classification result obtained as the target disease level.

[0113] It should be noted that the target disease level refers to the disease level of the image to be processed. In addition, the image processing device can perform disease level classification on a specified disease, and can also perform disease level classification on multiple diseases, which is not limited in the embodiments of the present application.

[0114] See also Figure 6 , Figure 6 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 2 ;in, Figure 6 The execution subject of each step in is the image processing device, such as Figure 6 As shown, in the embodiment of the present application, step 105 can be implemented through step 1051; that is, the image processing device performs disease classification based on the second feature sequence to obtain the target disease level, including step 1051, which is described below.

[0115] Step 1051 : perform disease classification based on the second feature sequence of the first image block sequence and the second feature sequence of the second image block sequence to obtain a target disease grade.

[0116] It should be noted that when the image to be processed includes multiple image block sequences, if the multiple image block sequences include a first image block sequence and a second image block sequence, then when the target image block sequence is the first image block sequence, the second feature sequence obtained is the second feature sequence of the first image block sequence; and when the target image block sequence is the second image block sequence, the second feature sequence obtained is the second feature sequence of the second image block sequence; at this time, the device to be processed combines the second feature sequence of the first image block sequence and the second feature sequence of the second image block sequence to perform disease classification. Among them, the first image block sequence and the second image block sequence are in opposite order; for example, the first image block sequence is {image block 1, image block 2, image block 3, image block 4, image block 5, image block 6}, and the second image block sequence is {image block 6, image block 5, image block 4, image block 3, image block 2, image block 1}.

[0117] It can be understood that by acquiring the second feature from different sequence orders, the most serious lesion can be transmitted in both forward and reverse directions on the image block, which can improve the extraction accuracy of the most serious lesion.

[0118] See also Figure 7 , Figure 7 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 3 ;in, Figure 7 The execution subject of each step in is the image processing device, such as Figure 7 As shown, in the embodiment of the present application, step 105 can be implemented through steps 1052 to 1054; that is, the image processing device performs disease classification based on the second feature sequence to obtain the target disease level, including steps 1052 to 1054, and each step is described below.

[0119] In the embodiment of the present application, the image processing device traverses the second feature sequence and performs the following processing (step 1052 and step 1053) on the kth second feature traversed.

[0120] Step 1052: Based on the kth second feature, calculate the probability that the kth target image block belongs to the jth Gaussian distribution to obtain a matching probability.

[0121] It should be noted that the image processing device is provided with J Gaussian distributions for the image to be processed, so as to reconstruct the second features based on the J Gaussian distributions to eliminate the image style differences between different domains; wherein the image style differences are determined by at least one of the following: differences between medical image acquisition devices, differences between medical image acquisition operations. Since the kth second feature is the second feature of the kth target image block, the image processing device can calculate the matching probability that the kth target image block belongs to the jth Gaussian distribution based on the kth second feature. wherein, , and j is an integer, J represents the total number of Gaussian distributions of the image to be processed; the matching probability represents the probability that the kth target image block belongs to the jth Gaussian distribution, so it can also be recorded as the kjth matching probability.

[0122] In step 1052 of the embodiment of the present application, the image processing device calculates the probability that the kth target image block belongs to the jth Gaussian distribution based on the kth second feature, and obtains the matching probability, including: the image processing device first calculates J posterior probabilities between the kth second feature and the J first Gaussian parameters respectively; then integrates the J posterior probabilities to obtain a posterior probability integration result; then, from the J posterior probabilities, obtains the target posterior probability corresponding to the first Gaussian parameter of the jth Gaussian distribution and the kth second feature; finally, the weight of the target posterior probability in the posterior probability integration result is determined as the matching probability.

[0123] It should be noted that the posterior probability integration result may be the cumulative result of J posterior probabilities; the weight of the target posterior probability in the posterior probability integration result may be the ratio of the target posterior probability to the posterior probability integration result.

[0124] Step 1053 , perform feature reconstruction based on the J matching probabilities of the kth second feature and the J first Gaussian parameters of the J Gaussian distributions to obtain features of the image block to be classified.

[0125] In an embodiment of the present application, since the image to be processed corresponds to J Gaussian distributions, the image processing device can calculate the J matching probabilities that the kth target image block distribution belongs to the J Gaussian distributions based on the kth second feature; at this time, the image processing device will integrate the J matching probabilities of the kth second feature and the J first Gaussian parameters, and the obtained integration result is the image block feature to be classified.

[0126] It should be noted that the first Gaussian parameter is a parameter of the Gaussian distribution; here, feature reconstruction can be achieved by correspondingly multiplying the J matching probabilities and the J first Gaussian parameters, and accumulating the obtained J products. The feature of the image block to be classified is the feature reconstruction result of the kth second feature, which represents the reconstructed feature of the kth second feature.

[0127] Step 1054: perform disease classification based on the K to-be-classified image block features obtained by traversing the second feature sequence to obtain a target disease grade.

[0128] It should be noted that, since a corresponding image block feature to be classified can be obtained for each second feature, when the second feature sequence is traversed, K image block features to be classified can be obtained for the K second features in the second feature sequence; at this time, the image processing device performs disease classification based on the K image blocks to be classified, and the disease classification may be performed directly based on the K image blocks to be classified, or the disease classification may be performed based on the combination of the K image block features to be classified and the K second features, and so on. The embodiments of the present application are not limited to this.

[0129] It can be understood that, by reconstructing the second feature through the parameters of the Gaussian distribution, the medical image style information in the second feature is eliminated, and the information accuracy of the reconstructed image block feature to be classified is improved.

[0130] Continue to see Figure 7 In the embodiment of the present application, step 1052 also includes steps 1055 to 1057; that is, the image processing device calculates the probability that the kth target image block belongs to the jth Gaussian distribution based on the kth second feature, and after obtaining the matching probability, the image processing method also includes steps 1055 to 1057, and each step is described below.

[0131] In the embodiment of the present application, the image processing device iteratively performs the following update process (step 1055 and step 1056 ) on the first Gaussian parameter.

[0132] Step 1055: Calculate the second Gaussian parameter of the jth Gaussian distribution based on the K matching probabilities for the jth first Gaussian parameter obtained by traversing the second feature sequence and the second feature sequence.

[0133] It should be noted that the image processing device can directly use the first Gaussian parameter to reconstruct the second feature, and can also first update the first Gaussian parameter, and then reconstruct the second feature based on the updated first Gaussian parameter, which is not limited in the embodiments of the present application. Here, by traversing the second feature sequence, J matching probabilities with J Gaussian distributions are calculated for each second feature, so that K matching probabilities corresponding to the jth first Gaussian parameter and K second features can be obtained; then, the image processing device updates the first Gaussian parameter based on the K matching probabilities and the K second features in the second feature sequence, and the result obtained is the updated first Gaussian parameter, which is called the second Gaussian parameter. Among them, the update of the first Gaussian parameter can be iterative, and the iteration end condition can be the update amount, the number of iterations, the iteration time, etc., which is not limited in the embodiments of the present application.

[0134] In step 1055 of the embodiment of the present application, the image processing device calculates the second Gaussian parameter of the j-th Gaussian distribution based on the K matching probabilities for the j-th first Gaussian parameter obtained by traversing the second feature sequence and the second feature sequence, including: the image processing device integrates the K matching probabilities of the j-th first Gaussian parameter obtained by traversing the second feature sequence to obtain a matching probability integration result; and integrates the K matching probabilities with the second feature sequence to obtain a probability feature integration result; finally, obtains a second Gaussian parameter that is positively correlated with the probability feature integration result and negatively correlated with the matching probability integration result.

[0135] It should be noted that the image processing device can integrate K matching probabilities by accumulation, can also integrate K matching probabilities by weighted summation, can also integrate K matching probabilities by multiplication, and so on, which is not limited in the embodiments of the present application; the matching probability integration result is the integration result of the K matching probabilities. The image processing device can first multiply the K matching probabilities with the second feature sequence correspondingly, and then obtain the probability feature integration result by accumulating the K products. Among them, the second Gaussian parameter is positively correlated with the probability feature integration result and negatively correlated with the matching probability integration result. The second Gaussian parameter is, for example, the ratio of the probability feature integration result to the matching probability integration result.

[0136] Step 1056: Update the matching probability by combining the second Gaussian parameter and the kth second feature to obtain the target matching probability.

[0137] It should be noted that the process of the image processing device updating the matching probability in combination with the second Gaussian parameter and the kth second feature is similar to the process of calculating the matching probability that the kth target image block belongs to the jth Gaussian distribution based on the kth second feature in step 1052, and the embodiments of the present application will not be repeated here.

[0138] Step 1057: When the iteration of the update process is completed, feature reconstruction is performed based on the J target matching probabilities for the kth second feature and the J second Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified.

[0139] It should be noted that the image processing device iteratively executes steps 1055 and 1056 to implement iterative updates of the first Gaussian parameters, and in the iterative update process of the first Gaussian parameters, iterative updates of the matching probabilities are also implemented. Here, the process of the image processing device combining the J target matching probabilities and the J second Gaussian parameters to perform feature reconstruction is similar to the process of performing feature reconstruction based on the J matching probabilities and the J first Gaussian parameters in step 1053, and the embodiment of the present application will not be repeated here.

[0140] It can be understood that by iteratively updating the first Gaussian parameter through the second feature to obtain the second Gaussian parameter, the correlation between the second feature and the first Gaussian parameter is improved; furthermore, when feature reconstruction is performed based on the second Gaussian parameter, the accuracy of feature reconstruction can be improved.

[0141] It can also be understood that when performing disease classification on an image to be processed, the image to be processed is divided into a target image block sequence, and a first feature sequence corresponding to the target image block sequence is cyclically updated based on a hidden state to implement cyclic encoding of the target image block sequence to obtain a second feature sequence; since the most serious lesions can be transferred between image blocks through the hidden state, the second feature sequence can accurately characterize the most serious lesions in the image to be processed; and further, when performing disease classification based on the second feature sequence, the accuracy of disease classification can be improved.

[0142] Continue to see Figure 6 In the embodiment of the present application, step 106 and step 107 are also included before step 101; that is, before the image processing device performs feature acquisition on the target image block sequence of the image to be processed and obtains the first feature sequence, the image processing method also includes steps 106 and 107. Each step is described below.

[0143] Step 106: Divide the image to be processed into image blocks to obtain a first image block sequence and a second image block sequence.

[0144] It should be noted that the image processing device divides the image to be processed into image blocks, and after obtaining a first image block sequence based on the image block division result, obtains a reverse sequence of the first image block sequence, thereby obtaining a second image block sequence.

[0145] Step 107: traverse the first image block sequence and the second image block sequence to obtain a target image block sequence.

[0146] It should be noted that the image processing device traverses the first image block sequence and the second image block sequence. When the traversed image block sequence is the first image block sequence, the first image block sequence is the target image block sequence, and when the traversed image block sequence is the second image block sequence, the second image block sequence is the target image block sequence.

[0147] Before step 105 of the embodiment of the present application, that is, before the image processing device performs disease classification based on the second feature sequence to obtain the target disease level, the image processing method also includes: the image processing device first performs nonlinear conversion based on the first feature sequence to obtain a feature weight sequence.

[0148] It should be noted that the image processing device can implement nonlinear transformation of the first feature sequence through an activation function; here, the nonlinear transformation result of the first feature sequence obtained is the feature weight sequence.

[0149] Correspondingly, in step 105, the image processing device performs disease classification based on the second feature sequence to obtain a target disease level, including: the image processing device fuses the second feature sequence with the feature weight sequence to obtain a feature sequence to be integrated; and integrates the first feature sequence and the feature sequence to be integrated to obtain a target feature sequence; finally, performs disease classification based on the target feature sequence to obtain a target disease level.

[0150] It should be noted that the feature sequence to be integrated is the fusion result of the second feature sequence and the feature weight sequence, and the image processing device can realize the fusion of the second feature sequence and the feature weight sequence by point-by-point multiplication; the target feature sequence is the integration result of the first feature sequence and the feature sequence to be integrated, and the integration of the first feature sequence and the feature sequence to be integrated can be realized by point-by-point addition.

[0151] It can be understood that by integrating the obtained second feature sequence with the initial first feature sequence, the comprehensiveness of the information represented by the target feature sequence is improved; furthermore, when the disease is graded based on the target feature sequence, the accuracy of the disease classification can be improved.

[0152] In the embodiment of the present application, the target disease level is obtained through a disease grading model, that is, steps 101 to 105 can be implemented through a disease grading model; wherein the disease grading model refers to an artificial neural network model for grading diseases in medical images; see Figure 8 , Figure 8 is an exemplary model training flow chart provided in the embodiment of the present application; Figure 8 As shown, the disease classification model is obtained through training from step 108 to step 110, and each step is described below.

[0153] Step 108: Use the model to be trained to perform disease classification on the image samples to obtain an estimated disease grade.

[0154] It should be noted that the model to be trained is an artificial neural network model to be trained for disease classification of medical images. In addition, the process of the image processing device using the model to be trained to obtain the image sample to estimate the disease level is similar to the process of obtaining the target disease level of the image to be processed, and the embodiment of the present application does not limit this. Among them, the image sample and the image to be processed can belong to the same domain or different domains, and the embodiment of the present application does not limit this.

[0155] Step 109: Calculate the target loss function value based on the difference between the estimated disease level and the disease level label of the image sample.

[0156] It should be noted that the image processing device uses a loss function to calculate the difference between the estimated disease level and the disease level label, thereby calculating the target loss function value.

[0157] Step 110: Train the model to be trained based on the target loss function value to obtain a disease grading model.

[0158] In an embodiment of the present application, the image processing device performs back propagation in the model to be trained based on the target loss function value to adjust the model parameters of the model to be trained, thereby adjusting the model to be trained; the model to be trained after the adjustment is a disease grading model. In addition, the training of the model to be trained can be performed iteratively, and when the iterative training ends, the model to be trained trained by the current iterative training is the disease grading model. Here, when the image processing device determines that the iterative training meets the training end condition, it determines that the iterative training ends; otherwise, iterative training continues.

[0159] It should be noted that the training end condition may be reaching an accuracy index threshold, reaching an iteration number threshold, reaching an iteration duration threshold, or a combination of the above, etc., and the embodiments of the present application do not limit this.

[0160] Continue to see Figure 8 , step 110 can be implemented through steps 1101 to 1105; that is, the image processing device trains the model to be trained based on the target loss function value to obtain a disease grading model, including steps 1101 to 1105, and each step is described below.

[0161] Step 1101: Obtain a third Gaussian parameter set updated for each image sample in M ​​image samples to obtain M third Gaussian parameter sets.

[0162] It should be noted that the third Gaussian parameter set includes J third Gaussian parameters corresponding to J Gaussian distributions. , and M is an integer; M represents the total number of batch image samples in the model training process; the process of obtaining the third Gaussian parameter is similar to the process of obtaining the second Gaussian parameter, and the embodiment of the present application is not limited to this.

[0163] Step 1102: Integrate the M third Gaussian parameter sets to determine an integrated Gaussian parameter set.

[0164] In an embodiment of the present application, the image processing device may determine the mean of M third Gaussian parameter sets as the integrated Gaussian parameter set, or may determine the mode of M third Gaussian parameter sets as the integrated Gaussian parameter set, and so on, which is not limited in this embodiment of the present application.

[0165] Step 1103: Combine the initial Gaussian parameter set and the Gaussian parameter integration set of the disease grading model to obtain a Gaussian parameter update set.

[0166] In an embodiment of the present application, the model parameters of the model to be trained include an initial Gaussian parameter set, and for the initial Gaussian parameter set, a sliding average method can be used to implement parameter adjustment; thus, the image processing device can obtain a Gaussian parameter update set by weighted summing the initial Gaussian parameter set and the Gaussian parameter integration set. The initial Gaussian parameter set is the parameters of the Gaussian distribution obtained before disease grading for the image sample. The Gaussian parameter update set is the parameters of the updated Gaussian distribution obtained during the model training process, which is used for feature reconstruction of disease grading of the next medical image.

[0167] Step 1104: adjust the state space parameters of the model to be trained based on the target loss function value to obtain the state space update parameters.

[0168] It should be noted that the parameters of the model to be trained also include state space parameters, which are used to cyclically encode the image block sequence of the medical image; here, the image processing device uses back propagation to adjust the state space parameters, and the adjusted state space parameters are the state space update parameters.

[0169] In the embodiment of the present application, the execution order of step 1102 and step 1103 is independent of the execution order of step 1104. Figure 8 An exemplary execution order is shown.

[0170] Step 1105: Combine the Gaussian parameter update set and the state space update parameters to obtain a disease classification model.

[0171] It should be noted that by updating the initial Gaussian parameter set in the model to be trained to the Gaussian parameter update set and adjusting the state space parameters in the model to be trained to the state space update parameters, the training of the model to be trained is completed and the disease classification model is obtained.

[0172] It is understandable that when training the model to be trained, the state space parameters are adjusted by back propagation, while the parameters of the Gaussian distribution are adjusted by sliding average. This can improve the stability of the model parameter adjustment, thereby improving the accuracy of model training and the generalization of the disease grading model.

[0173] In an embodiment of the present application, the image processing device performs feature acquisition on a target image block sequence of the image to be processed, and before obtaining the first feature sequence, the image processing method further includes: the image processing device responds to a disease grading request and obtains the image to be processed. Correspondingly, the image processing device performs disease grading based on the second feature sequence, and after obtaining the target disease grade, the image processing device further includes: the image processing device presents the target disease grade corresponding to the image to be processed.

[0174] It should be noted that when a request is made to perform disease grading on an image to be processed, the image processing device also receives the disease grading request; at this time, the image processing device responds to the disease grading request and obtains the medical image requested for disease grading, i.e., the image to be processed. Here, after obtaining the target disease grade, the image processing device presents the target disease grade to complete the response to the disease grading request.

[0175] Below, an exemplary application of the embodiment of the present application in an actual application scenario will be described. The exemplary application describes the process of disease grading of medical images. It is easy to know that the image processing method provided by the embodiment of the present application is applicable to any scenario of disease grading based on medical images. Here, the disease grading of diabetic retinopathy medical images and the disease grading of fatigue fracture medical images are used as examples for description.

[0176] It should be noted that disease classification is used to assess the severity of the lesion area based on medical images. Since the disease development process is a transitional, continuous, and time-dependent process, there is ambiguity in the classification between and within different disease grades. Fig. 9 , Fig. 9 is an exemplary disease development process diagram provided in the embodiments of the present application; Fig. 9 As shown, three disease levels corresponding to disease 9-1 and disease 9-2 are shown (disease level 9-11 to disease level 9-13 corresponding to disease 9-1, and disease level 9-21 to disease level 9-23 corresponding to disease 9-2), showing the changes in lesions during the development of the disease.

[0177] In addition, different medical images within the same disease level may correspond to different disease progressions; and, since some lesions of disease levels persist during the course of disease progression, similarities may exist between medical images of different disease levels. Fig.10 , Fig.10 is an exemplary disease level diagram provided in the embodiments of the present application; Fig.10As shown, the diversity of lesions within the same disease grade (disease grade 10-11, disease grade 10-12) and the similarity between different disease grades (disease grade 10-21 to disease grade 10-26) are shown.

[0178] It should also be noted that the number of sample images of different disease levels in different domains is unevenly distributed; and the differences in scanners, imaging parameters, and clinical operations in different domains make the feature distribution of medical images between different domains also different; thus, medical images from the same domain are clustered together in the feature space. Fig.11 , Fig.11 is an exemplary medical image clustering schematic diagram provided in an embodiment of the present application; Fig.11 As shown, domains 11-11 to 11-14 correspond to disease levels 11-21 to 11-24 respectively, wherein medical images of various disease levels in category 11-31 belong to domain 11-11, medical images of various disease levels in category 11-32 belong to domain 11-12, medical images of various disease levels in category 11-33 belong to domain 11-13, and medical images of various disease levels in category 11-34 belong to domain 11-14.

[0179] Based on the above description, in the disease grading scenario, in order to improve the accuracy of disease grading, it can be achieved by identifying the most serious lesions related to disease grading to achieve cross-domain disease grading. However, the most serious lesion may be located in an area in the medical image, corresponding to different shapes, and affected by the domain style; therefore, the embodiment of the present application proposes an image processing method, which is a cyclic modeling method based on severity perception, which can accurately identify the most serious lesions related to disease grading and can achieve cross-domain processing. The method is described below.

[0180] See also Fig.12 , Fig.12 is an exemplary disease classification diagram provided in the embodiments of the present application; Fig.12As shown, the hierarchical classification model 12-1 includes an embedding representation module 12-11, a first stage 12-12, a downsampling module 12-13, a second stage 12-14, a downsampling module 12-15, a third stage 12-16, a downsampling module 12-17, a fourth stage 12-18 and a hierarchical classification module 12-19; ​​the embedding representation module 12-11 is first used to embed the medical image 12-21 in the domain 12-2 to obtain a feature 12-31; the feature 12-31 is sequentially processed through the first stage 12-12, the downsampling module 12-13, the second stage 12-14, the downsampling module 12-15, the third stage 12-16, the downsampling module 12-17, the fourth stage 12-18 and the hierarchical classification module 12-19. The sampling module 12-13, the second stage 12-14, the down-sampling module 12-15, the third stage 12-16, the down-sampling module 12-17 and the fourth stage 12-18 obtain the feature 12-32; then, the grading module 12-19 is used to grade the feature 12-32 to obtain the graded result (for example, disease level 1 to disease level 5) 12-4; finally, the graded model 12-1 is trained based on the difference between the graded result 12-4 and the label 12-5 of the medical image 12-21.

[0181] It should be noted that in Fig.12 In the example, the first stage 12-12 includes two (denoted as ⤬2) feature processing modules (Samba Block) 12-121, the second stage 12-14 includes two (denoted as ⤬2) feature processing modules 12-141, the third stage 12-16 includes nine (denoted as ⤬9) feature processing modules 12-161, and the fourth stage 12-18 includes two (denoted as ⤬2) feature processing modules 12-181. Since the model structures of the feature processing modules are the same, any feature processing module will be described below.

[0182] In the embodiment of the present application, the feature processing module adopts a state space model to encode the image blocks of the medical image in a cyclic manner. The state space model combines the E-dimensional input and hidden state Mapping the output , as shown in formula (3); and, the hidden state can be used The derivative of represents the change in the hidden state, as shown in formula (4).

[0183] (3);

[0184] (4);

[0185] Among them, A is The parameter matrix of B, C and D are t is the parameter matrix of continuous data; during model training, A, B, C, and D are all learnable parameters and are updated by gradient descent.

[0186] It should be noted that due to It is a skip connection, which is used to pass data directly to the next step, so here In addition, since the data corresponding to medical images are discrete data, for medical images, the input The discretization expression of is shown in formula (5).

[0187] (5);

[0188] Among them, k is the sequence index of discrete data, Represents the interval (called the discretization interval parameter), which is a learnable parameter; is the kth image block .

[0189] In the embodiment of the present application, based on formula (6) to formula (8), the feature of the kth image block of the medical image can be determined: (called the first feature) corresponds to the output of the spatial state model (called the second characteristic), as shown in formulas (6) to (8).

[0190] (6);

[0191] (7);

[0192] (8);

[0193] Among them, A, B (where A and B are called continuous characteristic parameters) and C are estimated in the way of zero-order hold to obtain the corresponding discretization representations , (in, and is called the discretized characteristic parameter) and . Indicates The hidden state corresponding to the image block is Indicates The hidden state corresponding to each image patch.

[0194] It should be noted that zero-order hold is a signal processing technique used to reconstruct a continuous signal based on discrete sampled data. By adopting zero-order hold, the value of the signal is obtained at each sampling moment and this value is kept unchanged until the next sampling moment.

[0195] Based on formula (6) to formula (8), see Fig.13 , Fig.13 is a schematic diagram of an exemplary feature processing module provided in an embodiment of the present application; Fig.13 As shown, the feature processing module 13-1 includes a batch normalization module 13-11, a linear layer 13-12, a linear layer 13-13, a forward encoding module 13-14, a reverse encoding module 13-15, a weight determination module 13-16 and a linear layer 13-17. First, the feature 13-21 of the image block (referred to as the first feature) passes through the batch normalization module 13-11, the linear layer 13-12 and the forward encoding module 13-14 in sequence to obtain the feature 13-22 (referred to as the second feature); in addition, the feature 13-21 passes through the batch normalization module 13-11, the linear layer 13-12 and the reverse encoding module 13-15 in sequence to obtain the feature 13-23 (referred to as the second feature); and the feature 13-21 passes through the batch normalization module 13-11, the linear layer 13-13 and the weight determination module 13-16 in sequence to obtain the weight 13-24 (referred to as the feature weight sequence). Then, feature 13-22 is dot-multiplied by weight 13-24 to obtain feature 13-25, and feature 13-23 is dot-multiplied by weight 13-24 to obtain feature 13-26. Feature 13-25 and feature 13-26 are point-wise summed, and the point-wise summation result passes through linear layer 13-17 to obtain feature 13-27. Feature 13-27 and feature 13-21 are point-wise summed to obtain feature 13-28.

[0196] It should be noted that the forward encoding modules 13-14 and the reverse encoding modules 13-15 are implemented based on formulas (6) to (8) when performing encoding.

[0197] It can be understood that the use of state space to process medical images can determine the parameter matrix based on the input, and then accurately obtain the context information of the image block, thereby improving the accuracy of disease grading. In addition, by dividing the medical image into a sequence of image blocks, the lesions in the image block can be stored in the hidden state, so that the lesions can be transferred to the subsequent image blocks; and, through the forward encoding module and the reverse encoding module, the lesions can be transferred to each image block, realizing the transfer of lesions from local (image block) to global (medical image), so that based on the globally perceived lesions, the update of the hidden state can selectively extract the most serious lesions, thereby improving the recognition accuracy of the most serious lesions.

[0198] In the embodiment of the present application, the feature processing module uses a feature correction module, which can map the features of image blocks in different domains to the same feature space, thereby improving the generalization performance of disease classification. The feature correction module uses a Gaussian mixture model to model the features of the image blocks, as shown in formula (9).

[0199] (9);

[0200] in, For the The features of the image blocks, is the modeling result, J is the total number of Gaussian functions, For the The image block belongs to the jth ( ) Gaussian function, For the The mean of the Gaussian functions, is the standard matrix I, is the distribution of the Gaussian function.

[0201] It should be noted that in the distribution of Gaussian function, is the standard matrix I, so the Gaussian function can be obtained by Here, is called the severity base, which is a trainable parameter. Here, in order to improve the accuracy of the features of medical images, the expectation maximization algorithm is used to update the severity base. First, based on and Estimate (called matching probability), as shown in formula (10).

[0202] (10);

[0203] in, For the The mean of the Gaussian functions; is the posterior probability , which can be achieved by formula (11).

[0204] (11);

[0205] Here, based on the dth Update the dth (called the first Gaussian parameter), as shown in formula (12).

[0206] (12);

[0207] in, For the Second (called the second Gaussian parameter), For the dth , For the dth ,and and and similar, is the total number of image blocks.

[0208] Based on the iterative update of formula (10) to formula (12), when the convergence conditions such as the set maximum number of steps are met, we get and ;in, To finally obtain , To finally obtain At this time, the reconstructed features of image block k are (called the image block feature to be classified) is shown in formula (13).

[0209] (13);

[0210] For example, see Fig.14 , Fig.14 is an exemplary feature reconstruction schematic diagram provided in an embodiment of the present application; Fig.14 As shown, the feature correction module 14-1 includes a linear layer 14-11, an iteration module 14-12, a linear layer 14-13 and a batch normalization module 14-14. The feature 14-21 of the image block first passes through the linear layer 14-11 to obtain the feature 14-22. The iteration module 14-12 uses formula (10) to formula (12) to iteratively update the Gaussian parameters and estimated probabilities in the iteration module 14-12 based on the feature 14-22 to obtain the feature 14-23; then, the feature 14-23 passes through the linear layer 14-13 and the batch normalization module 14-14 in turn to obtain the feature 14-24; finally, the feature 14-24 and the feature 14-21 are summed point by point to obtain the feature 14-25.

[0211] It should be noted that the mean difference distribution of the Gaussian function is implemented by a sliding average method, as shown in formula (14).

[0212] (14);

[0213] The left side of the arrow Represents the updated J , on the right side of the arrow Indicates the J before the update , Represents the M corresponding to the batch of medical images The average value of .

[0214] It should be noted that when performing disease grading for diabetic retinopathy, diabetic retinopathy includes six data sets: Dataset 1 (DeepDR), Dataset 2 (Messidor), Dataset 3 (IDRID), Dataset 4 (APTOS), Dataset 5 (FGADR) and Dataset 6 (RLDR). Here, baseline model 1, baseline model 2 and the disease grading model provided in the embodiment of the present application (hereinafter referred to as the model of the present application) are obtained by training with data set 1, and the disease grading model is tested on data sets 2 to 6, using the precise score (F1) and accuracy (Accuracy ACC) as the evaluation indicators of the test, and calculating the average of each evaluation indicator of the 5 data sets, as shown in Table 1.

[0215] Table 1

[0216]

[0217] As can be seen from Table 1, compared with baseline model 1 and baseline model 2, the model of the present application is larger in various evaluation indicators and average items on different data sets; the average F1 of the model of the present application is 8.7 higher than the average F1 of baseline model 2, and the average ACC of the model of the present application is 22.5 higher than the average ACC of baseline model 2.

[0218] It should be noted that when disease classification is performed for fatigue fracture, fatigue fracture includes two data sets: data set 7 and data set 8, where data set 7 and data set 8 belong to different domains; here, the disease classification model is trained on data set 7, and tested on data set 8; and the area under the curve (AUC), F1 and ACC are used as evaluation indicators.

[0219] See also Fig.15 , Fig.15 is an exemplary disease grading assessment index comparison chart provided in the embodiments of the present application; Fig.15 As shown, compared Figure 15-1 To comparison Figure 15-6 The horizontal axes of are the iteration times of Gaussian parameters; Figure 15-1 and contrast Figure 15-4 The vertical axes are all AUC. Figure 15-2 and contrast Figure 15-5 The vertical coordinates are all ACC. Figure 15-3 and contrast Figure 15-6 The vertical coordinate is F1.

[0220] contrast Figure 15-1 To comparison Figure 15-3What is described is the comparison result before and after adding feature reconstruction; wherein, curve 15-41 represents the AUC corresponding to different numbers of iterations before adding feature reconstruction, and curve 15-42 represents the AUC corresponding to different numbers of iterations after adding feature reconstruction; curve 15-43 represents the ACC corresponding to different numbers of iterations before adding feature reconstruction, and curve 15-44 represents the ACC corresponding to different numbers of iterations after adding feature reconstruction; curve 15-45 represents the F1 corresponding to different numbers of iterations before adding feature reconstruction, and curve 15-46 represents the F1 corresponding to different numbers of iterations after adding feature reconstruction.

[0221] contrast Figure 15-4 To comparison Figure 15-6 Described are the comparison results of using random initialization, gradient back propagation and sliding average methods respectively; wherein, curve 15-51 represents the AUC of different iterations corresponding to updating the Gaussian parameters in the model by sliding average, curve 15-53 represents the AUC of different iterations corresponding to updating the Gaussian parameters in the model by random initialization, curve 15-52 represents the AUC of different iterations corresponding to updating the Gaussian parameters in the model by gradient back propagation; curve 15-54 represents the ACC of different iterations corresponding to updating the Gaussian parameters in the model by sliding average, curve Line 15-56 represents the ACC of different iterations corresponding to updating the Gaussian parameters in the model by random initialization, and curve 15-55 represents the ACC of different iterations corresponding to updating the Gaussian parameters in the model by gradient back propagation; curve 15-57 represents the F1 of different iterations corresponding to updating the Gaussian parameters in the model by sliding average, curve 15-59 represents the F1 of different iterations corresponding to updating the Gaussian parameters in the model by random initialization, and curve 15-58 represents the F1 of different iterations corresponding to updating the Gaussian parameters in the model by gradient back propagation.

[0222] It should be noted that by comparison Figure 15-1 To comparison Figure 15-3 It can be determined that adding feature reconstruction can improve various evaluation indicators. Figure 15-4 To comparison Figure 15-6 It can be determined that compared with the random initialization method and the gradient back propagation method, the sliding average method can improve various evaluation indicators.

[0223] See Table 2, which describes the data of baseline models 3 to 7 and the model of the present application under the three evaluation indicators of AUC, F1 and ACC when performing disease grading for fatigue fractures.

[0224] Table 2

[0225] Model ACC AUC F1 Baseline Model 3 39.8 50.2 18.6 Baseline Model 4 43.3 61.8 20.9 Baseline Model 5 45.7 65.1 22.4 Baseline Model 6 50.0 69.3 26.5 Baseline Model 7 52.7 70.4 28.7 This application model 76.2 81.4 45.8

[0226] As can be seen from Table 2, the model of this application is better than baseline model 3 to baseline model 7 in terms of the three evaluation indicators of AUC, F1 and ACC.

[0227] It is understandable that the embodiment of the present application represents the medical image as a sequence of image blocks, and uses a bidirectional encoding method to cyclically encode the image block sequence, which can accurately extract the lesion severity information in the image block, and can also transfer lesion-related information between image blocks, which can improve the accuracy of extracting the most serious lesions. In addition, by reducing the impact of inter-domain differences through feature correction, the robustness of the disease grading model in the domain generalization scenario can be improved, and the accuracy of disease grading in the domain generalization scenario can be improved.

[0228] The following is a description of an exemplary structure of the image processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 3 As shown, the software modules stored in the image processing device 455 of the memory 450 may include:

[0229] A feature acquisition module 4551 is used to acquire features of a target image block sequence of the image to be processed to obtain a first feature sequence;

[0230] The cyclic encoding module 4552 is used to traverse the first feature sequence to obtain the kth first feature traversed. , and k is an integer, K represents the total number of target image blocks in the target image block sequence:

[0231] The cyclic encoding module 4552 is further configured to update the kth first feature based on the kth first feature. hidden state, get the kth hidden state, when At that time, The hidden state is the initial hidden state. At that time, The hidden state is a hidden state of the target image block;

[0232] The cyclic encoding module 4552 is further used to map the kth second feature based on the kth hidden state;

[0233] The cyclic encoding module 4552 is further configured to obtain a second feature sequence based on the K second features obtained by traversing the first feature sequence;

[0234] The disease grading module 4553 is used to perform disease grading based on the second feature sequence to obtain a target disease grade.

[0235] In an embodiment of the present application, the image processing device 455 also includes a sequence determination module 4554, which is used to divide the image to be processed into image blocks to obtain a first image block sequence and a second image block sequence, wherein the first image block sequence and the second image block sequence are in opposite order; and traverse the first image block sequence and the second image block sequence to obtain the target image block sequence.

[0236] In this embodiment of the present application, the disease grading module 4553 is further used to perform disease grading based on the second feature sequence of the first image block sequence and the second feature sequence of the second image block sequence to obtain the target disease grade.

[0237] In the embodiment of the present application, the cyclic encoding module 4552 is further used to obtain a discretized feature parameter corresponding to the continuous feature parameter based on a discretization interval parameter.

[0238] In the embodiment of the present application, the cyclic encoding module 4552 is further used to encode the kth first feature and the kth first feature based on the discretized feature parameter. The hidden states are integrated to obtain the kth hidden state, the kth first feature and the kth The integration of the hidden states is used to achieve the An update of the hidden state.

[0239] In the embodiment of the present application, the disease grading module 4553 is further used to traverse the second feature sequence and perform the following processing on the kth second feature traversed: based on the kth second feature, calculate the probability that the kth target image block belongs to the jth Gaussian distribution to obtain the matching probability, , and j is an integer, J represents the total number of the Gaussian distributions of the image to be processed; feature reconstruction is performed based on the J matching probabilities of the kth second feature and the J first Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified; disease classification is performed based on the K image block features to be classified obtained by traversing the second feature sequence to obtain the target disease level.

[0240] In an embodiment of the present application, the disease grading module 4553 is also used to calculate J posterior probabilities between the kth second feature and the J first Gaussian parameters respectively; integrate the J posterior probabilities to obtain a posterior probability integration result; obtain the target posterior probability corresponding to the jth Gaussian parameter of the Gaussian distribution and the kth second feature from the J posterior probabilities; and determine the weight of the target posterior probability in the posterior probability integration result as the matching probability.

[0241] In an embodiment of the present application, the image processing device 455 also includes a feature reconstruction module 4555, which is used to iteratively perform the following update processing for the first Gaussian parameter: based on the K matching probabilities for the jth first Gaussian parameter obtained by traversing the second feature sequence, and the second feature sequence, calculate the second Gaussian parameter of the jth Gaussian distribution; update the matching probability in combination with the second Gaussian parameter and the kth second feature to obtain the target matching probability; when the iteration of the update processing is completed, perform feature reconstruction based on the J target matching probabilities for the kth second feature and the J second Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified.

[0242] In an embodiment of the present application, the feature reconstruction module 4555 is also used to integrate the K matching probabilities for the jth first Gaussian parameter obtained by traversing the second feature sequence to obtain a matching probability integration result; integrate the K matching probabilities with the second feature sequence to obtain a probability feature integration result; and obtain the second Gaussian parameter that is positively correlated with the probability feature integration result and negatively correlated with the matching probability integration result.

[0243] In the embodiment of the present application, the disease grading module 4553 is further used to perform nonlinear transformation based on the first feature sequence to obtain a feature weight sequence.

[0244] In an embodiment of the present application, the disease grading module 4553 is also used to fuse the second feature sequence with the feature weight sequence to obtain a feature sequence to be integrated; integrate the first feature sequence and the feature sequence to be integrated to obtain a target feature sequence; and perform disease grading based on the target feature sequence to obtain the target disease grade.

[0245] In an embodiment of the present application, the target disease level is obtained through a disease grading model, and the image processing device 455 also includes a model training module 4556, which is used to use the model to be trained to grade the disease of the image sample to obtain an estimated disease level, and the model to be trained is an artificial neural network model to be trained for performing disease grading on medical images; based on the difference between the estimated disease level and the disease level label of the image sample, the target loss function value is calculated; based on the target loss function value, the model to be trained is trained to obtain the disease grading model.

[0246] In the embodiment of the present application, the model training module 4556 is further used to obtain a third Gaussian parameter set updated by each of the M image samples, to obtain M third Gaussian parameter sets, wherein the third Gaussian parameter set includes J third Gaussian parameters corresponding to J Gaussian distributions. , and M is an integer; integrating the M third Gaussian parameter sets to determine a Gaussian parameter integration set; combining the initial Gaussian parameter set of the disease classification model and the Gaussian parameter integration set to obtain a Gaussian parameter update set; adjusting the state space parameters of the model to be trained based on the target loss function value to obtain state space update parameters; combining the Gaussian parameter update set and the state space update parameters to obtain the disease classification model.

[0247] In the embodiment of the present application, the feature acquisition module 4551 is further used to obtain the image to be processed in response to a disease classification request.

[0248] In the embodiment of the present application, the disease grading module 4553 is also used to present the target disease grade corresponding to the image to be processed.

[0249] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer-readable storage medium. The processor of the image processing device reads the computer executable instructions or computer programs from the computer-readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the image processing device performs the above-mentioned image processing method of the embodiment of the present application.

[0250] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the image processing method provided by the present application embodiment, for example, Figure 5 The image processing method is shown.

[0251] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0252] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0253] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0254] As an example, computer-executable instructions may be deployed to be executed on one electronic device (in which case, the one electronic device is the image processing device), or on multiple electronic devices located at one location (in which case, the multiple electronic devices located at one location are the image processing devices), or on multiple electronic devices distributed at multiple locations and interconnected by a communication network (in which case, the multiple electronic devices distributed at multiple locations and interconnected by a communication network are the image processing devices).

[0255] It is understandable that in the embodiments of the present application, data related to medical images, etc. are involved. When the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the information subject and carry out subsequent data use and processing within the scope of authorization of laws, regulations and the information subject; and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and comply with the principles of legality, legitimacy and necessity. It does not involve the acquisition of data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0256] In summary, when performing disease classification on an image to be processed, the embodiment of the present application divides the image to be processed into a target image block sequence, and cyclically updates the first feature sequence corresponding to the target image block sequence based on the hidden state to realize cyclic encoding of the target image block sequence and obtain a second feature sequence; because the most serious lesion information can be transferred between image blocks through the hidden state, the second feature sequence can accurately characterize the most serious lesion in the image to be processed; and thus, when performing disease classification based on the second feature sequence, the accuracy of disease classification can be improved. In addition, by obtaining the second feature sequence corresponding to the first image block sequence and the second image block sequence respectively, bidirectional transmission of the most serious lesion can be realized, and thus the recognition accuracy of the most serious lesion can be improved; and when a disease classification model is used for disease classification, the spatial state parameters in the model to be trained are trained by gradient back propagation, and the Gaussian parameters in the model to be trained are adjusted by sliding average, which can improve the generalization of the disease classification model.

[0257] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Perform feature acquisition on a target image block sequence of the image to be processed to obtain a first feature sequence; The first feature sequence is traversed, and the following processing is performed on the kth first feature traversed: , and k is an integer, K represents the total number of target image blocks in the target image block sequence: Based on the kth first feature, update the hidden state, get the kth hidden state, when At that time, The hidden state is the initial hidden state. At that time, The hidden state is a hidden state of the target image block; Based on the kth hidden state, mapping the kth second feature; Obtaining a second feature sequence based on the K second features obtained by traversing the first feature sequence; Performing disease grading based on the second feature sequence to obtain a target disease grade; The step of performing disease grading based on the second feature sequence to obtain a target disease grade includes: traversing the second feature sequence, and performing the following processing on the kth second feature traversed: based on the kth second feature, calculating the probability that the kth target image block belongs to the jth Gaussian distribution, and obtaining a matching probability; , and j is an integer, J represents the total number of the Gaussian distributions of the image to be processed; feature reconstruction is performed based on the J matching probabilities of the kth second feature and the J first Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified; disease classification is performed based on the K image block features to be classified obtained by traversing the second feature sequence to obtain the target disease level.

2. The method according to claim 1, characterized in that Before acquiring features of the target image block sequence of the image to be processed to obtain the first feature sequence, the method further includes: Dividing the image to be processed into image blocks to obtain a first image block sequence and a second image block sequence, wherein the first image block sequence and the second image block sequence are in opposite order; Traversing the first image block sequence and the second image block sequence to obtain the target image block sequence; The performing disease grading based on the second feature sequence to obtain a target disease grade includes: Disease grading is performed based on the second feature sequence of the first image block sequence and the second feature sequence of the second image block sequence to obtain the target disease grade.

3. The method according to claim 1, characterized in that The updating of the kth first feature based on the kth first feature hidden state, before obtaining the kth hidden state, the method further comprises: Based on the discretization interval parameter, obtaining a discretization feature parameter corresponding to the continuous feature parameter; The updating of the kth first feature based on the kth first feature hidden states, and obtain the kth hidden state, including: Based on the discretized feature parameters, the kth first feature and the The hidden states are integrated to obtain the kth hidden state, the kth first feature and the kth The integration of the hidden states is used to achieve the An update of the hidden state.

4. The method according to claim 1, characterized in that: The calculating, based on the kth second feature, the probability that the kth target image block belongs to the jth Gaussian distribution to obtain the matching probability includes: Calculate J posterior probabilities between the k-th second feature and J first Gaussian parameters respectively; Integrate the J posterior probabilities to obtain a posterior probability integration result; From the J posterior probabilities, obtain a target posterior probability corresponding to the first Gaussian parameter of the j-th Gaussian distribution and the k-th second feature; The weight of the target posterior probability in the posterior probability integration result is determined as the matching probability.

5. The method according to claim 1, characterized in that The step of performing feature reconstruction based on the J matching probabilities of the kth second feature and the J first Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified includes: For each of the J first Gaussian parameters, the following updating process is iteratively performed: Calculate the second Gaussian parameter of each Gaussian distribution based on the K matching probabilities for each of the first Gaussian parameters obtained by traversing the second feature sequence and the second feature sequence; Combining the second Gaussian parameter of each Gaussian distribution and the kth second feature, updating the matching probability that the kth target image block belongs to each Gaussian distribution, to obtain a target matching probability; When the iteration of the update processing for the J first Gaussian parameters is completed, feature reconstruction is performed based on the J target matching probabilities for the kth second feature and the J second Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified.

6. The method according to claim 5, characterized in that The calculating the second Gaussian parameter of each Gaussian distribution based on the K matching probabilities for each of the first Gaussian parameters obtained by traversing the second feature sequence and the second feature sequence includes: Integrating the K matching probabilities for each of the first Gaussian parameters obtained by traversing the second feature sequence to obtain a matching probability integration result; Integrate the K matching probabilities and the second feature sequence to obtain a probability feature integration result; The second Gaussian parameter of each of the Gaussian distributions that is positively correlated with the probability feature integration result and negatively correlated with the matching probability integration result is obtained.

7. The method according to any one of claims 1 to 6, characterized in that: Before performing disease grading based on the second feature sequence to obtain a target disease grade, the method further includes: Performing nonlinear transformation based on the first feature sequence to obtain a feature weight sequence; The performing disease grading based on the second feature sequence to obtain a target disease grade includes: fusing the second feature sequence with the feature weight sequence to obtain a feature sequence to be integrated; Integrating the first feature sequence and the feature sequence to be integrated to obtain a target feature sequence; Disease classification is performed based on the target feature sequence to obtain the target disease grade.

8. The method according to any one of claims 1 to 6, characterized in that: The target disease grade is obtained by a disease grade model, and the disease grade model is trained by the following steps: Using a model to be trained to perform disease classification on image samples to obtain an estimated disease grade, wherein the model to be trained is an artificial neural network model to be trained for performing disease classification on medical images; Calculating a target loss function value based on a difference between the estimated disease level and the disease level label of the image sample; The model to be trained is trained based on the target loss function value to obtain the disease grading model.

9. The method according to claim 8, characterized in that The step of training the model to be trained based on the target loss function value to obtain the disease grading model includes: Obtaining a third Gaussian parameter set updated by each of the M image samples to obtain M third Gaussian parameter sets, wherein the third Gaussian parameter set includes J third Gaussian parameters corresponding to J Gaussian distributions, , and M is an integer; Integrate the M third Gaussian parameter sets to determine an integrated Gaussian parameter set; Combining the initial Gaussian parameter set of the disease grading model and the Gaussian parameter integration set to obtain a Gaussian parameter update set; Adjust the state space parameters of the model to be trained based on the target loss function value to obtain state space update parameters; The disease classification model is obtained by combining the Gaussian parameter update set and the state space update parameters.

10. The method according to any one of claims 1 to 6, characterized in that: Before acquiring features of the target image block sequence of the image to be processed to obtain the first feature sequence, the method further includes: In response to a disease classification request, obtaining the image to be processed; After performing disease grading based on the second feature sequence to obtain a target disease grade, the method further includes: The target disease level corresponding to the image to be processed is presented.

11. An image processing device, characterized in that: The image processing device comprises: A feature acquisition module, used for acquiring features of a target image block sequence of the image to be processed to obtain a first feature sequence; A cyclic encoding module is used to traverse the first feature sequence to obtain the kth first feature traversed, , and k is an integer, K represents the total number of target image blocks in the target image block sequence: The cyclic encoding module is further used to update the kth first feature based on the kth first feature. hidden state, get the kth hidden state, when At that time, The hidden state is the initial hidden state. At that time, The hidden state is a hidden state of the target image block; The cyclic encoding module is further used to map the kth second feature based on the kth hidden state; The cyclic encoding module is further used to obtain a second feature sequence based on the K second features obtained by traversing the first feature sequence; A disease grading module, used for performing disease grading based on the second feature sequence to obtain a target disease grade; The disease grading module is further used to traverse the second feature sequence and perform the following processing on the kth second feature traversed: based on the kth second feature, calculate the probability that the kth target image block belongs to the jth Gaussian distribution to obtain a matching probability, , and j is an integer, J represents the total number of the Gaussian distributions of the image to be processed; feature reconstruction is performed based on the J matching probabilities of the kth second feature and the J first Gaussian parameters of the J Gaussian distributions to obtain the image block features to be classified; disease classification is performed based on the K image block features to be classified obtained by traversing the second feature sequence to obtain the target disease level.

12. An electronic device for image processing, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the image processing method according to any one of claims 1 to 10 when executing the computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.