Image processing model training method and device, electronic equipment and storage medium

Abstract

The application provides a training method and device of an image processing model, an electronic device and a storage medium. The method comprises: acquiring a plurality of three-dimensional image samples and annotation data, the types of the three-dimensional image samples comprising annotated image samples and unannotated image samples, and the annotation data comprising actual segmentation positions of the annotated image samples; calling the image processing model based on the plurality of three-dimensional image samples to perform image segmentation processing, obtaining a first predicted segmentation position, and updating the image processing model based on a first loss between the first predicted segmentation position and the annotation data; taking the first predicted segmentation position of the unannotated image sample as an actual segmentation position of the unannotated image sample; updating the first loss based on the actual segmentation position of each three-dimensional image sample to obtain a second loss; and updating the image processing model updated based on the first loss based on the second loss. Through the application, the accuracy of segmentation of three-dimensional images can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and in particular to an image processing model training method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Computer vision technology (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, locate and measure targets, and further process images to make computer processing more suitable for human observation or transmission to instrument detection.

[0003] For the segmentation processing of three-dimensional images, the related technology usually converts three-dimensional images into two-dimensional images for processing, such as optical coherence tomography (OCT) images. Each slice image in the image is processed. In the process of training the related image processing model, the training samples usually need to be manually annotated, which consumes a lot of time and computing resources.

[0004] The related technology has no good solution to improve the accuracy of segmenting three-dimensional images. SUMMARY

[0005] The embodiments of the present application provide an image processing model training method and device, an electronic device, a computer readable storage medium, and a computer program product, which can improve the accuracy of segmenting three-dimensional images.

[0006] The technical solutions of the embodiments of the present application are as follows:

[0007] The embodiments of the present application provide an image processing model training method, comprising:

[0008] Obtain a plurality of three-dimensional image samples and annotation data, wherein the types of the three-dimensional image samples include annotated image samples and unannotated image samples, and the annotation data includes the actual segmentation position of the annotated image samples;

[0009] Call the image processing model based on the plurality of three-dimensional image samples to perform image segmentation processing, obtain a first predicted segmentation position, and update the image processing model based on a first loss between the first predicted segmentation position and the annotation data;

[0010] the first predicted segmentation position of the unlabeled image sample as the actual segmentation position of the unlabeled image sample, wherein the first predicted segmentation position is generated by calling the image processing model;

[0011] updating the first loss based on the actual segmentation position of each three-dimensional image sample to obtain a second loss;

[0012] updating the image processing model updated based on the first loss based on the second loss.

[0013] Embodiments of the present application provide an image processing method, which comprises:

[0014] obtaining a three-dimensional image to be segmented;

[0015] calling an image processing model based on the three-dimensional image to perform image segmentation processing to obtain a segmentation position in the three-dimensional image, wherein the image processing model is obtained by the training method of the image processing model of the embodiments of the present application.

[0016] Embodiments of the present application provide a training device of an image processing model, which comprises:

[0017] a sample acquisition module configured to acquire a plurality of three-dimensional image samples and annotation data, wherein the types of the three-dimensional image samples include labeled image samples and unlabeled image samples, and the annotation data includes the actual segmentation position of the labeled image samples;

[0018] a pre-training module configured to call the image processing model based on the plurality of three-dimensional image samples to perform image segmentation processing to obtain a first predicted segmentation position, and update the image processing model based on a first loss between the first predicted segmentation position and the annotation data;

[0019] a model updating module configured to take the first predicted segmentation position of the unlabeled image sample as the actual segmentation position of the unlabeled image sample, wherein the first predicted segmentation position is generated by calling the image processing model;

[0020] the model updating module is further configured to update the first loss based on the actual segmentation position of each three-dimensional image sample to obtain a second loss;

[0021] the model updating module is further configured to update the image processing model updated based on the first loss based on the second loss.

[0022] Embodiments of the present application provide an image processing device, which comprises:

[0023] An image receiving module is configured to acquire a three-dimensional image to be segmented.

[0024] An image processing module is configured to call an image processing model based on the three-dimensional image to perform image segmentation processing to obtain a segmentation position in the three-dimensional image, wherein the image processing model is obtained by the training method of the image processing model provided in the embodiments of the present application.

[0025] The embodiments of the present application provide an electronic device, comprising:

[0026] A memory is configured to store computer executable instructions.

[0027] A processor is configured to execute the computer executable instructions stored in the memory to implement the training method of the image processing model provided in the embodiments of the present application.

[0028] The embodiments of the present application provide a computer readable storage medium storing computer executable instructions for causing a processor to execute the training method of the image processing model provided in the embodiments of the present application.

[0029] The embodiments of the present application provide a computer program product comprising a computer program or computer executable instructions, which, when executed by a processor, implement the training method of the image processing model provided in the embodiments of the present application.

[0030] The embodiments of the present application have the following beneficial effects:

[0031] By training in two stages, the accuracy of the trained image processing model is improved. The unlabeled training data is labeled based on the first predicted segmentation position obtained by prediction in a sparse labeling manner to update the loss of the image processing model, and the image processing model is optimized and trained based on the updated loss to improve the accuracy of the trained image processing model and save the computing resources required for labeling the training data. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is an application mode schematic diagram of the training method of the image processing model provided in the embodiments of the present application;

[0033] Figure 2A is a structural schematic diagram of a server provided in the embodiments of the present application;

[0034] Figure 2B is a structural schematic diagram of a server provided in the embodiments of the present application;

[0035] Figure 2C is a structural schematic diagram of an image processing model provided in the embodiments of the present application;

[0036] Figures 3A to 31 is a flowchart of a training method of an image processing model provided by an embodiment of the present application;

[0037] Figure 4 is a structural diagram of an image processing model provided by an embodiment of the present application;

[0038] Figure 5 is a flowchart of a training method of an image processing model provided by an embodiment of the present application;

[0039] Figure 6A is an application mode diagram of a training method of an image processing model provided by an embodiment of the present application;

[0040] Figure 6B is an application mode diagram of an image processing method provided by an embodiment of the present application;

[0041] Figures 7A to 7D is a table of experimental results provided by an embodiment of the present application;

[0042] Figures 8A to 8D is a diagram of an image processing result provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the present application in conjunction with the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0044] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0045] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] It should be noted that in the embodiments of the present application, the information related to the user, the user feedback data and the like, when the embodiments of the present application are applied to specific products or technologies, need to obtain the permission or consent of the user, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0048] Before the embodiments of the present application are further described, the terms and names involved in the embodiments of the present application are explained, and the terms and names involved in the embodiments of the present application are applicable to the following explanations.

[0049] 1) Image Segmentation: Image segmentation is a key process in computer vision. It involves segmenting visual input into pieces to simplify image analysis. A piece represents an object or a part of an object and consists of a set of pixels or "superpixels". Image segmentation organizes pixels into larger parts, eliminating the need to treat individual pixels as units of observation. Image segmentation is used to identify parts of an image and understand what they belong to, and is the basis for object detection and classification. Image segmentation can be applied in the fields of face detection, medical imaging, autonomous driving, etc.

[0050] 2) Optical Coherence Tomography (OCT): A technology that obtains tomographic information in the depth direction based on the principle of low-coherence interference, and reconstructs two-dimensional or three-dimensional images of the internal structure of biological tissues or materials by scanning.

[0051] 3) 3D Coherent Layer Segmentation: An algorithm for three-dimensional segmentation of three-dimensional OCT images to obtain the positions of retinal layers.

[0052] 4) B-Scan and A-Scan: B-Scan refers to transverse scanning, and a transverse slice image is obtained by extracting optical coherence tomography transverse scanning. A-Scan refers to axial scanning, and a column in the transverse slice image is obtained.

[0053] 5) B-Scan Alignment: Eliminate the misalignment between the transverse slice images obtained by B-Scan scanning, so that all the transverse slice images of B-Scan are aligned.

[0054] 6) Convolutional Neural Networks (CNN): A class of feed-forward neural networks containing convolutional computation and having a deep structure, is one of the representative algorithms of deep learning.

[0055] 7) 2D-3D hybrid network: a hybrid network containing both 2D convolutional neural network and 3D convolutional neural network.

[0056] 8) Spatial Transformation Module (STM): a module that transforms the original image or feature map by using spatial transform parameters, which supports backpropagation.

[0057] 9) Sparse Annotation: Sparse annotation refers to that only part of the training data is labeled in the training data, and part of the training data is not labeled. For example, only part of the slices in a set of three-dimensional OCT data is labeled, and the other slice images are not labeled.

[0058] The embodiment of the present application provides a kind of training method of image processing model, training device of image processing model, electronic equipment and computer readable storage medium and computer program product, can improve the accuracy of three-dimensional image segmentation.

[0059] The exemplary application of the electronic device provided by the embodiment of the present application will be described below. The electronic device provided by the embodiment of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), a vehicle terminal, etc. Various types of user terminals, and can also be implemented as a server. Below, the exemplary application when the device is implemented as a server will be described.

[0060] Reference Figure 1 , Figure 1 is the application mode schematic diagram of the training method of image processing model provided by the embodiment of the present application; in the example, Figure 1 The training server 200-1, the image processing server 200-2, the network 300 and the terminal device 400 are involved in the embodiment. The training server 200-1 and the image processing server 200-2 communicate through the network 300 or other means, and the terminal device 400 connects the image processing server 200-2 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0061] In some embodiments, the user is a scientific researcher or a medical staff, and the OCT three-dimensional image to be processed is an image processing server 200-2 of a medical platform. The segmentation result is a lesion (e.g., edge) existing in the OCT three-dimensional image. The user can use the segmentation result as a basis for diagnosis. The above examples will be described below.

[0062] For example, the training server 200-1 obtains training samples, calls the initialized image processing model to perform the training method of the image processing model provided in the embodiments of the present application based on the training samples, obtains the trained image processing model, synchronizes the trained image processing model to the image processing server 200-2, the image processing server 200-2 receives the OCT three-dimensional image to be processed uploaded by the user through the terminal device 400, performs segmentation processing on the OCT three-dimensional image to be processed, and returns the segmentation result to the terminal device 400. The user takes the segmentation result as a diagnostic basis to diagnose the patient corresponding to the OCT three-dimensional image.

[0063] In some embodiments, the user can be a technician, the terminal device 400 can be a computer, the training server 200-1 is a server of a cloud platform providing model training services, and the above examples are explained and described in combination.

[0064] For example, the technician uploads specific training samples to the training server 200-1 through the computer, the training server 200-1 runs the training method of the image processing model provided in the embodiments of the present application based on the training samples, obtains the trained image processing model, realizes the individual customization of the image processing model, and sends the trained model to the computer, so that the technician uses the trained image processing model in the computer.

[0065] For example, the image processing model trained by the training method of the image processing model provided in the embodiments of the present application can be applied in the following scenarios:

[0066] (1) Medical image processing, for example: the training samples include OCT images of human blood vessels with lesions and OCT images of healthy human blood vessels, the trained image processing model can be an image processing model for segmenting the OCT images of human blood vessels, the segmentation result is the edge of the lesion of the human blood vessel, and the user can take the segmentation result as a diagnostic basis.

[0067] (2) Industrial field, for example: the training samples include electron microscope images (cross section or longitudinal section) of alloy materials with different degrees of deformation, the trained image processing model is used to segment the edges of the defect regions in the alloy materials with different degrees of deformation, and the technician can analyze the tensile degree, elongation and other parameters of the alloy material through the segmentation result, and improve the efficiency of developing the alloy material.

[0068] For another example: the training samples include OCT images of opaque plastic products, the trained image processing model is used to segment the edges of the defect regions of the opaque plastic products, and the technician can quickly determine the defects existing in the plastic products and determine the plastic products that do not meet the quality standards through the segmentation result, thereby improving the efficiency of quality inspection.

[0069] (3) Face detection, for example: the training sample includes: a plurality of images containing faces, the annotation data is the actual position of the face in the image, the trained image processing model is used for segmenting the face region in the image, and the trained image processing model can be used to provide face recognition services.

[0070] (4) Autonomous driving, for example: the training sample includes: a street view image including a plurality of obstacles (for example: vehicles, pedestrians, roadblocks, etc.), the annotation data is the edge contour of the obstacle in the street view image, and the trained image processing model is used for segmenting the obstacle in the street view image, so that the autonomous driving vehicle determines a safe driving area based on the area of the obstacle.

[0071] The embodiments of the present application can be implemented through blockchain technology, and the image processing model trained by the embodiments of the present application can be uploaded to the blockchain for storage, and the reliability of the image processing model can be ensured through a consensus algorithm. The blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The blockchain (Blockchain) is essentially a decentralized database, which is a series of data blocks associated using cryptography. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0072] The embodiments of the present application can be implemented through database technology. In short, a database (Database) can be regarded as an electronic file cabinet for storing electronic files. Users can add, query, update, delete and other operations on the data in the file. The so-called "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs.

[0073] A database management system (DBMS) is a computer software system designed for managing databases, typically with functions such as storage, retrieval, security, backup, and other basic functions. DBMSs can be classified by the database model they support, such as relational, XML; by the type of computer they support, such as server clusters, mobile phones; by the query language they support, such as SQL, XQuery; by the performance focus, such as maximum size, maximum speed; or by other classification methods. Regardless of the classification method, some DBMSs can cross categories, such as supporting multiple query languages.

[0074] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool, be used on demand, and be flexible and convenient. Cloud computing technology will become an important support. The background service of a technical network system requires a large amount of computing and storage resources, such as video websites, picture websites, and more portals. With the high development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce, and open collaboration, every item may have its own hash code identification mark in the future, which needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support. Only through cloud computing can this be achieved.

[0075] In some embodiments, the training server 200-1 and the image processing server 200-2 can be integrated into one independent physical server.

[0076] In some embodiments, the training server 200-1 or the image processing server 200-2 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The electronic device can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device and the server can be connected directly or indirectly through wired or wireless communication, and the embodiments of the present application do not limit the connection method.

[0077] Referring to Figure 2A , Figure 2A is a structural schematic diagram of a server provided by an embodiment of the present application, Figure 2A The training server 200-1 shown includes at least one processor 410, a memory 450, and at least one network interface 420. The various components in the training server 200-1 are coupled together by a bus system 440. It can be understood that the bus system 440 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity in Figure 2A , all the buses are marked as the bus system 440.

[0078] The processor 410 can be an integrated circuit chip having a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0079] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memories, hard disk drives, optical disk drives, etc. The memory 450 can optionally include one or more storage devices that are physically located away from the processor 410.

[0080] The memory 450 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0081] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0082] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0083] The network communication module 452 is configured to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, such as Bluetooth, WiFi, Universal Serial Bus (USB), and the like.

[0084] In some embodiments, the training apparatus of the image processing model provided by the embodiments of the present application can be implemented in a software manner, Figure 2A The training apparatus 455 of the image processing model stored in the memory 450 can be software in the form of programs and plug-ins, and includes the following software modules: a sample acquisition module 4551, a pre-training module 4552, and a model updating module 4553. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0085] Referring to Figure 2B , Figure 2B is a structural schematic diagram of a server provided by the embodiments of the present application, Figure 2B The image processing server 200-2 shown in FIG. 2 includes at least one processor 410, a memory 450, and at least one network interface 420. The various components in the image processing server 200-2 are coupled together by a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between the components. The bus system 440 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 440 in Figure 2B .

[0086] The processor 410 can be an integrated circuit chip having a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and the like. The general-purpose processor can be a microprocessor or any conventional processor.

[0087] The memory 450 can be removable, non-removable, or a combination thereof. The exemplary hardware devices include a solid-state memory, a hard disk drive, an optical disk drive, and the like. The memory 450 optionally includes one or more storage devices physically separated from the processor 410.

[0088] The memory 450 includes a volatile memory or a nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile memory can be a read-only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0089] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0090] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0091] The network communication module 452 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), and the like.

[0092] In some embodiments, the training device of the image processing model provided by the embodiments of the present application can be realized in a software manner, Figure 2B An image processing device 456 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: an image receiving module 4554 and an image processing module 4555, which are logical, and thus can be combined or further split according to the functions implemented. The functions of each module will be described below.

[0093] The training method of the image processing model provided by the embodiments of the present application will be described in conjunction with exemplary applications and implementations of the server provided by the embodiments of the present application. Referring to Figure 3A , Figure 3A is a flowchart of the training method of the image processing model provided by the embodiments of the present application, taking the server as the execution subject, which will be described in conjunction with Figure 3A the steps shown.

[0094] In step 301, a plurality of three-dimensional image samples and annotation data are obtained.

[0095] For example, the types of the three-dimensional image samples include labeled image samples and unlabeled image samples, and the labeled data includes actual segmentation positions of the labeled image samples. The three-dimensional image sample can be an OCT image including a plurality of B-scan images (lateral scan images), and embodiments of the present application take retinal layer images as examples for explanation and description.

[0096] For example, given A three-dimensional OCT image can be written as a real-valued function V(x, y, z): Ω→R, where x and y axes are rows and columns in a B-scan image, and z axis is orthogonal to the B-scan image. V can be regarded as an ordered set of all lateral scan slice images B-Scans in an OCT image: V(b) = {I b}, where I b : Φ→R is the bth B-scan image, and b ∈ [1, N B ], N B is the total number of B-Scans in an OCT image. A retinal layer can be represented as S = {r b,a}, where a ∈ [1, N A ], N A is the number of A-Scans in a B-scan image, and r b,a is the position of the retinal surface on the ath A-Scan in the bth B-Scan. It should be noted that a surface only intersects each A-Scan once. The training target of the embodiments of the present application is to enable the image processing model to locate a set of smooth retinal layer surfaces in a three-dimensional OCT image.

[0097] In some embodiments, step 301 is implemented by obtaining three-dimensional image data and labeled data, performing flattening processing (processing of flattening data into a structure suitable for a neural network) on the three-dimensional image data, and dividing the flattened three-dimensional image data into a plurality of three-dimensional image samples. For example, the size of the whole three-dimensional image data is 320x400x41. During training, due to limited video memory, the image is cropped along the x-axis to obtain a plurality of small images of 320x40x41, and the plurality of small images are taken as three-dimensional image samples.

[0098] In the embodiments of the present application, by performing flattening processing and cropping into a plurality of image samples on the three-dimensional image samples, the required computing resources in the training process are saved, and the efficiency of training the image processing model is improved.

[0099] In step 302, the image processing model is called based on the plurality of three-dimensional image samples to perform image segmentation processing, to obtain first predicted segmentation positions, and the image processing model is updated based on a first loss between the first predicted segmentation positions and the labeled data.

[0100] In an example, the updating the image processing model is updating parameters of the image processing model.

[0101] In some embodiments, the first loss includes: a first alignment loss and a first segmentation loss; the reference Figure 3B , Figure 3B is a flowchart of a method for training an image processing model provided by an embodiment of the present application, step 302 can be implemented by the following steps 3021 to step 3025, which are described in detail below.

[0102] In step 3021, based on each three-dimensional image sample, the image processing model is called to perform the following processing: obtaining the alignment features of the three-dimensional image sample.

[0103] In some embodiments, the reference Figure 3C , Figure 3C is a flowchart of a method for training an image processing model provided by an embodiment of the present application, step 3021 can be implemented by the following steps 30211 to step 30213, which are described in detail below.

[0104] In step 30211, the three-dimensional image sample is subjected to feature extraction processing to obtain image features.

[0105] In an example, step 3021 will be explained and described in combination with the structure of the image processing model, referring to Figure 2C , Figure 2C is a structural diagram of an image processing model provided by an embodiment of the present application; the image processing model 201C includes: a feature extraction network 210C, an alignment network 220C, a spatial displacement module 230C, and a segmentation network 240C; the feature extraction network 210C is used to perform feature extraction processing, the alignment network 220C is used to perform displacement processing, the spatial displacement module 230C is used to perform alignment processing, and the segmentation network 240C is used to perform segmentation processing. The feature extraction network 210C can be an encoder composed of multiple levels of two-dimensional convolutional neural networks. The alignment network 220C and the segmentation network 240C are respectively decoders composed of multiple levels of three-dimensional convolutional neural networks.

[0106] In an example, the feature extraction network 210C performs feature extraction processing on the three-dimensional image sample to obtain two-dimensional image features. For example, the feature extraction network 210C extracts two-dimensional image features from the B-Scan image.

[0107] In step 30212, the image features are subjected to displacement processing to obtain displacement vector features.

[0108] The displacement processing is implemented by the alignment network 220C, which is configured to generate a displacement vector feature based on the two-dimensional image features, and the displacement vector feature can be represented as an alignment displacement field. The alignment displacement field is configured to indicate a displacement of each B-Scan image in the y-axis direction.

[0109] In step 30213, the displacement vector feature is aligned with the image feature to obtain an alignment feature of the three-dimensional image sample.

[0110] In an example, each three-dimensional image sample is acquired separately in the image acquisition process. Due to the global alignment and the inevitable motion of the human body organs (for example, eye movement, blood vessel pulsation, etc.), the consecutive B-Scan images in an OCT image can be misaligned, and the misaligned images can be aligned through the alignment processing.

[0111] In an example, the alignment network 220C and the segmentation network 240C are coupled through the spatial displacement module 230C, and the spatial displacement module 230C is configured to perform the alignment processing, that is, to perform the displacement processing on the image feature based on the displacement distance indicated by the alignment displacement field to obtain the aligned image feature, that is, the alignment feature.

[0112] Continuing to refer to Figure 3B In step 3022, the alignment feature is segmented to obtain a first predicted segmentation position corresponding to the three-dimensional image sample.

[0113] In an example, the segmentation network is a decoder composed of multiple three-dimensional convolutional networks, which is configured to predict the alignment feature to obtain a region of interest in the three-dimensional image sample as the first predicted segmentation position, for example, a hierarchical position between retinal layers.

[0114] In an example, the segmentation network has two output heads. The first output head outputs a distribution of retinal layer surfaces, and the second output head outputs a pixel-level segmentation label (layer wise map). The second output head is configured to output the pixel-level supervision information (segmentation label), so as to better train the image processing model. In the stage of applying the image processing model, the output content of the second output head can be ignored.

[0115] The embodiments of the present application improve the segmentation accuracy by the alignment processing and the segmentation processing after the alignment processing, avoid the misalignment of the segmentation result caused by the misalignment between the transverse slice images in the three-dimensional image sample in the image acquisition process, and do not need to perform the alignment processing in the preprocessing stage of the training sample, thereby saving the computing resources.

[0116] Continuing to refer to Figure 3BIn step 3023, a first alignment loss is determined based on each alignment feature and the annotation data.

[0117] For example, the first alignment loss includes a smooth alignment loss and a cross-correlation loss. For reference Figure 3D , Figure 3D is a flowchart of a training method of an image processing model provided by an embodiment of the present application. Step 3023 can be implemented by the following steps 30231 to 30234, which are described in detail below.

[0118] In step 30231, for any two adjacent three-dimensional image samples in the plurality of three-dimensional image samples, the following processing is performed: obtaining a sub-cross-correlation loss between the alignment features of the two three-dimensional image samples.

[0119] For example, the cross-correlation loss is used to represent the correlation between the adjacent aligned images. The principle of obtaining the cross-correlation loss is similar to that of obtaining the correlation coefficient. The cross-correlation loss may be represented by the following formula (1):

[0120]

[0121] wherein, is the sub-cross-correlation loss corresponding to the adjacent two three-dimensional image samples b and b+1.

[0122] In some embodiments, the two three-dimensional image samples are denoted as a first three-dimensional image sample (for example: three-dimensional image sample b) and a second three-dimensional image sample (for example: three-dimensional image sample b+1). For reference Figure 3E , Figure 3E is a flowchart of a training method of an image processing model provided by an embodiment of the present application. Step 30231 can be implemented by the following steps 2311 to 2313, which are described in detail below.

[0123] In step 2311, a first average value of a plurality of pixel values around each pixel in the first three-dimensional image sample is extracted from the alignment features of the first three-dimensional image sample.

[0124] For example, a pixel pair includes a first pixel in the first three-dimensional image sample and a second pixel in the second three-dimensional image sample that has the same position as the first pixel, and p represents all pixels in the image space Φ, is the bth B-scan image according to the corresponding displacement field d b image after displacement, is the image k pixels in the image the image obtained by subtracting the local average value from the image after displacement, the first average value in formula (1) can be converted to the second average value in formula (1) can be converted to that is, the average value of n 2 pixels around the pixel p k . 2 The first average value in formula (1) is the average value of n pixels around the pixel p k . 2 The first average value in formula (1) is the average value of n

[0125] In step 2312, a second average value of a plurality of pixel values around each pixel in the second three-dimensional image sample is extracted from the aligned features of the second three-dimensional image sample.

[0126] For example, the second average value is represented in formula (1) as The principle of obtaining the second average value is the same as that of obtaining the first average value in step 2311.

[0127] In step 2313, any one pixel in the first three-dimensional image sample and a pixel at the same position in the second three-dimensional image sample are taken as a pixel pair, and a sub-cross-correlation loss of the two three-dimensional image samples is determined based on the difference between the first average value and the second average value of the pixel pair.

[0128] For example, for each pixel pair, the following processing is performed: determining a first product of the first average value of the first pixel and the second average value of the second pixel in the pixel pair, and the first product is represented in formula (1) as The square of the first average value is taken as a second square term, and the square of the second average value is taken as a third square term, and the second square term is represented in formula (1) as The third square term is represented in formula (1) as A first sum between each first product is obtained, and the square of the first sum is obtained; the square of the first sum is represented in formula (1) as A second product of the sum between each second square term and the sum between each third square term is determined, and the second product is represented in formula (1) as The ratio of the square of the first sum to the second product is taken as the sub-cross-correlation loss of the two three-dimensional image samples.

[0129] For example, the square of the first sum is divided by the second product, and the sub-cross-correlation loss between the adjacent two three-dimensional image samples is obtained, and the cross-correlation loss is obtained by adding the sub-cross-correlation losses corresponding to all B-scan three-dimensional image sample pairs. As shown in formula (1) above.

[0130] With reference back to Figure 3DIn step 30232, a first difference value between an actual segmentation position and a displacement vector feature of each three-dimensional image sample is obtained, a second difference value between two first difference values corresponding to two three-dimensional image samples respectively is obtained, and a square of the second difference value is taken as a first square term.

[0131] The smooth alignment loss The smooth alignment loss can be represented by the following formula (2):

[0132]

[0133] wherein r g is a labeled position of a retinal layer, d b represents a displacement field corresponding to the bth B-Scan, r b,a that is, a position of a retinal surface on the ath A-Scan in the bth B-Scan, is an actual segmentation position of the labeled retinal surface on the ath A-Scan in the bth B-Scan. That is, the first difference value of the three-dimensional image sample b, That is, the first difference value of the three-dimensional image sample b+1, and the second difference value is represented by The first square term is represented by

[0134] In step 30233, each sub-cross-correlation loss is added to obtain a cross-correlation loss, and each first square term is added to obtain a smooth alignment loss.

[0135] For example, the process of obtaining the cross-correlation loss is shown in the above explanation of formula (1), and the process of obtaining the smooth alignment loss is shown in the above explanation of formula (2).

[0136] In step 30234, the sum of the cross-correlation loss and the smooth alignment loss is taken as a first alignment loss.

[0137] For example, the first alignment loss is represented by:

[0138] With reference to the foregoing description Figure 3B In step 3024, a first segmentation loss is determined based on each first predicted segmentation position and the labeled data.

[0139] For example, the first segmentation loss includes a cross-entropy loss, a mean absolute error (MAE) loss, and a global smoothness loss. The mean absolute error loss is also referred to as a smooth L1 loss.

[0140] In some embodiments, the result of the segmentation processing includes: a distribution probability of each position in the three-dimensional image sample (the accuracy of each position is a pixel level), the first predicted segmentation position; and Figure 3F , Figure 3F is a flowchart of a training method of an image processing model provided in an embodiment of the present application, and step 3024 can be implemented through the following steps 30241 to 30244, which are described in detail below.

[0141] In step 30241, a cross-entropy loss is determined based on the distribution probability of each actual segmentation position in the three-dimensional image sample and the first predicted segmentation position.

[0142] The cross-entropy loss may be represented by the following formula (3):

[0143]

[0144] where q represents the distribution probability, q b，a (r|V; q) is the distribution probability of the retinal layer position in the A-Scan, R is the height of the A-Scan, 1(x) is an indicator function, and x represents the actual segmentation position whether the predicted first predicted segmentation position r is equal, that is, whether the labeled data and the predicted position are equal, and when x is true, the value of 1(x) is 1, otherwise it is 0.

[0145] In step 30242, an absolute mean error loss is determined based on the difference between each first predicted segmentation position and the actual segmentation position.

[0146] The mean error loss, that is, the smooth L1 loss (Mean Absolute Error, MAE), is used to directly guide the predicted surface position to be close to the true value, as shown in the following formula (4).

[0147]

[0148] where, represents the difference between the optimal predicted position and the actual position; obtained by a soft-argmax function: is the r that maximizes the distribution probability.

[0149] In step 30243, a surface two-dimensional gradient of each position in the three-dimensional image sample is determined based on the position difference between two adjacent first predicted segmentation positions, and the sum of the norms of each surface two-dimensional gradient is taken as a global smoothness loss.

[0150] The embodiment of the present application adopts a global smoothness loss to guide the predicted curved surface position to have more three-dimensional smoothness, refer to formula (5):

[0151]

[0152] The embodiment of the present application uses the difference between the positions of adjacent points on the predicted curved surface position to represent the two-dimensional gradient of the curved surface, refer to formula (8) below:

[0153]

[0154] wherein, and respectively represent the pixel spacing in the z-axis direction and the y-axis direction in the OCT image.

[0155] In step 30244, the cross-entropy loss, the average error loss, and the global smoothness loss are added to obtain the first segmentation loss.

[0156] The first segmentation loss can be summarized as: cross-entropy loss smooth L1 loss and the sum of the global smoothness loss . The global smoothness loss is multiplied by λ and added to the other losses to obtain the total segmentation loss

[0157] wherein λ is a hyperparameter for controlling the weight of the global smoothness loss. The value of λ is selected according to the actual smoothness of each layer. Specifically, for the lth layer of the retinal curved surface, the embodiment of the present application calculates λ using the following formula (7):

[0158]

[0159] wherein is the label of the lth layer of the retinal curved surface. The smoother the surface is, the greater the penalty for the loss of its overall coherence. The embodiment of the present application uses the arithmetic mean of λ l of the OCT data in all training data to calculate λ l .

[0160] Continuing to refer to Figure 3B , in step 3025, the image processing model is updated based on the first alignment loss and the first segmentation loss.

[0161] In some embodiments, step 3025 can be implemented by: performing parameter update processing on the alignment network and the spatial displacement module based on the first alignment loss, to obtain the parameter-updated alignment network and the parameter-updated spatial displacement module; and performing parameter update processing on the segmentation network based on the first segmentation loss, to obtain the parameter-updated segmentation network.

[0162] For example, with reference to Figure 2C , the image processing model includes multiple different networks and modules, and different losses are used for parameter update of the networks with different functions.

[0163] In some embodiments, the segmentation network includes a first output head and a second output head; the second output head is configured to output a distribution probability of each position in the three-dimensional image sample, and the first output head is configured to output a first predicted segmentation position; the parameter update processing on the segmentation network based on the first segmentation loss can be implemented by: performing parameter update processing on the first output head based on a cross-entropy loss, a global smoothing loss, and an average error loss, to obtain a parameter-updated first output head; and performing parameter update processing on the second output head based on a cross-entropy loss, to obtain a parameter-updated second output head.

[0164] The embodiments of the present application perform parameter update processing on the networks with different functions in the image processing model by using different losses, thereby improving the accuracy of the image processing model in segmenting images, and saving the computing resources required for training the model by using different losses for parameter update of part of the networks.

[0165] For example, with reference to Figure 3A , in step 303, the first predicted segmentation position of the unlabeled image sample is taken as the actual segmentation position of the unlabeled image sample.

[0166] For example, the first predicted segmentation position is generated by calling the image processing model.

[0167] The embodiments of the present application save the human and computing resources required for labeling the image samples and save the time for obtaining the training samples by taking the predicted segmentation position obtained in the pre-training stage as the actual segmentation position of the image sample without labeled data in the semi-supervised manner of sparse labeling, while maintaining the accuracy of the trained image processing model on the basis of saving the computing resources.

[0168] For example, with reference to Figure 3A , in step 304, the first loss is updated based on the actual segmentation position of each three-dimensional image sample, to obtain a second loss.

[0169] For example, the first loss includes: a first alignment loss and a first segmentation loss; for example, with reference to Figure 3G ,Figure 3G is a flowchart of a training method of an image processing model provided by an embodiment of the present application. Step 304 can be implemented through the following steps 3041 to 3043, which are described in detail below.

[0170] In step 3041, the actual segmentation position in the smooth alignment loss included in the first alignment loss is updated based on the actual segmentation position of each three-dimensional image sample after re-labeling, to obtain a second alignment loss.

[0171] For example, the smooth alignment loss uses the surface position in the segmentation decoder G s to replace the predicted position, to obtain the following sparse-labeled smooth alignment loss

[0172]

[0173] When the corresponding B-Scan is labeled, and when the corresponding B-Scan is not labeled, that is, the prediction result of the corresponding segmentation decoder. That is, the predicted segmentation position obtained by pre-training prediction is used as the label of the unlabeled image. After the second alignment loss is updated, it is

[0174] In step 3042, the predicted segmentation position in the global smooth loss included in the first segmentation loss is updated based on the actual segmentation position of each three-dimensional image sample after re-labeling, to obtain a second segmentation loss.

[0175] For example, for the global smooth loss, the alignment network trained by the updated second alignment loss obtains alignment features, the updated alignment features are segmented to obtain a second predicted segmentation position, and the global smooth loss is updated based on the second predicted segmentation position. The updated global smooth loss can be represented as

[0176] In step 3043, the second alignment loss and the second segmentation loss are combined to obtain a second loss.

[0177] For example, the second alignment loss and the second segmentation loss are combined and used to update different networks in the image processing model.

[0178] In an embodiment of the present application, the loss is updated in the case of sparse labeling, which improves the efficiency of training the image processing model, and further enables the image processing model to more accurately perform image segmentation processing, while saving the computing resources required for training the model.

[0179] Continuing to refer to Figure 3AIn step 305, the image processing model updated based on the first loss is updated based on the second loss.

[0180] In some embodiments, the image processing model comprises a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network; the second loss comprises a second alignment loss and a second segmentation loss; the reference Figure 3H , Figure 3H is a flowchart of a training method of an image processing model provided by an embodiment of the present application. Step 305 can be implemented by the following steps 3051 to 3052, which are described in detail below.

[0181] In step 3051, the alignment network and the spatial displacement module are updated in parameters based on the second alignment loss, to obtain the alignment network and the spatial displacement module updated in parameters.

[0182] For example, the parameters of the alignment network and the spatial displacement module are updated in a back propagation manner based on the second alignment loss, to obtain the alignment network and the spatial displacement module updated in parameters.

[0183] In step 3052, the segmentation network is updated in parameters based on the second segmentation loss, to obtain the segmentation network updated in parameters.

[0184] For example, the parameters of the segmentation network are updated in a back propagation manner based on the second segmentation loss, to obtain the segmentation network updated in parameters. Each network updated in parameters is combined together to obtain the image processing model trained.

[0185] In some embodiments, the training method of the image processing model provided by an embodiment of the present application can be applied in the training process of other image models. For example, for an image processing model used for image recognition, a predicted recognition region is obtained through pre-training, a training sample with missing label data is labeled in the predicted recognition region through sparse labeling, an optimized loss is obtained, and the pre-trained model is processed in a back propagation manner based on the optimized loss to obtain a trained model.

[0186] An image processing method is also provided by an embodiment of the present application. Referring to Figure 3I , Figure 3I is a flowchart of an image processing method provided by an embodiment of the present application. The server is taken as an execution subject, and the following steps shown in Figure 3A will be described.

[0187] In step 306, a three-dimensional image to be segmented is obtained.

[0188] For example, the three-dimensional image to be segmented can be an OCT image of blood vessels of a human body. The three-dimensional image to be segmented is flattened to convert the three-dimensional image to be segmented into a structure suitable for a neural network, and the flattened three-dimensional image is divided into a plurality of small patches. A plurality of small patches are processed simultaneously in each batch to save computing resources of an image processing server.

[0189] In step 307, an image processing model is called based on the three-dimensional image to perform image segmentation processing, to obtain a segmentation position in the three-dimensional image.

[0190] For example, the image processing model is obtained by the training method of the image processing model provided in the embodiments of the present application. Step 302I is implemented by the following manner: based on the three-dimensional image, calling the image processing model to perform the following processing: performing feature extraction processing on the three-dimensional image to obtain image features of the three-dimensional image; performing displacement processing on the image features to obtain displacement vector features; performing alignment processing on the displacement vector features and the image features to obtain aligned features of the three-dimensional image; and performing segmentation processing on the aligned features to obtain a segmentation position corresponding to the three-dimensional image.

[0191] The embodiments of the present application improve the accuracy of the trained image processing model by training in two stages. The unlabeled training data is labeled based on the first predicted segmentation position obtained by prediction in a sparse labeling manner, to update the loss of the image processing model, and the image processing model is optimized and trained based on the updated loss, to improve the accuracy of the trained image processing model and save the computing resources required for labeling the training data.

[0192] In the following, an example application of the training method of the image processing model provided in the embodiments of the present application in an actual application scenario will be described.

[0193] In the related art, a full-supervised 2D OCT image retinal layer segmentation method treats each slice image (B-Scan) in the OCT data as an independent two-dimensional picture for segmentation. However, in fact, the OCT image is composed of a plurality of B-Scans, covering a three-dimensional area of the fundus. Such a method does not utilize the prior knowledge that the retinal layer is a smooth three-dimensional curved surface, which may cause misalignment of the segmentation results between B-scans and affect subsequent analysis. Meanwhile, such a method is based on full-supervised training, that is, each B-Scan in the three-dimensional OCT data needs to be manually labeled, which is time-consuming and laborious. The energy function with spatial gradient continuity constraint is used for retinal layer segmentation and parameter measurement in the optic nerve head region. The traditional method for three-dimensional retinal layer segmentation of OCT data has a poorer segmentation result than the segmentation result of the convolutional neural network model technical solution, and does not consider the misalignment between adjacent B-Scans, and cannot realize end-to-end training. The uncertain guided semi-supervised segmentation method of the retinal layer in the OCT image is also a 2D semi-supervised segmentation method, which also does not utilize the three-dimensional continuity of the retinal layer, and the segmentation results between B-scans will be misaligned.

[0194] To solve the problems in the related art, the training method of the image processing model provided in the embodiments of the present application can achieve the following effects:

[0195] 1) The scheme provided in the embodiments of the present application can align the misaligned B-Scans in the OCT data. The original OCT data is input, and the embodiments of the present application can generate an up-down displacement field of the B-scans. The displacement field is applied to the original OCT data through a spatial transformation module (STM), and the aligned OCT data is obtained.

[0196] 2) The embodiments of the present application can segment the three-dimensional OCT data without any alignment preprocessing to obtain a three-dimensional continuous retinal layer, obtain a more accurate segmentation result, and the three-dimensional continuous retinal layer segmentation result has more clinical significance in subsequent quantitative analysis of related diseases than the segmentation result based on the two-dimensional segmentation method.

[0197] 3) The embodiments of the present application can perform alignment and retinal layer segmentation of three-dimensional OCT data based on sparse labeling, save the calculation resources required for labeling of the training model, ensure good results, and increase the clinical application value. Compared with full supervision, the 3D continuity of the retinal layer can be better utilized in the case of sparse labeling.

[0198] The training method of the image processing model provided in the embodiments of the present application will be described below with reference to Figure 5 , Figure 5 is a flowchart of the training method of the image processing model provided in the embodiments of the present application. Reference is made to Figure 4 , Figure 4 is a structural diagram of the image processing model provided in the embodiments of the present application; and the steps in Figure 5 and the structure in Figure 4 will be described.

[0199] In step 501, the training data is preprocessed.

[0200] For example, automatic segmentation of retinal layers is an important and challenging problem in optical coherence tomography (OCT) analysis. Given A three-dimensional OCT image can be written as a real-valued function V(x, y, z): Ω→R, where x and y axes are rows and columns in a B-Scan image, and z axis is orthogonal to the B-Scan image. V can be regarded as an ordered set of all transverse scan slice images B-Scans in an OCT image: V(b) = {I b}, where I b : Φ→R is the bth B-Scan image, and b∈[1, N B ], N B is the total number of B-Scans in an OCT image. A retinal layer can be represented as S = {r b,a}, where a∈[1, N A ], N A is the number of A-Scans in a B-Scan image, and r b,a is the position of the retinal surface on the ath A-Scan in the bth B-Scan. It should be noted that a surface only intersects each A-Scan once. The goal of the embodiments of the present application is to locate the interested {S} in V, i.e., a set of smooth retinal layer surfaces, to accurately segment the layers.

[0201] For example, a flattening algorithm (an algorithm for flattening data into a structure suitable for a neural network) is used to flatten the retinal layers in the original OCT data for training, and the flattened OCT image is divided into small images.

[0202] The embodiments of the present application are respectively trained and verified on two public data sets: A2A data set and JHH data set. That is, normal OCT images and lesion OCT images are obtained from different public data sets as training samples. Each OCT image is segmented to obtain a plurality of image blocks included in each OCT image, and each image is processed based on the image processing model to obtain a segmentation result.

[0203] The A2A dataset includes OCT data of 265 normal people and 115 people with age-related macular degeneration (AMD). The center region of size 512x400x41 is cropped from each data for training and testing. After the flattening algorithm, the overall 3D data size is 320x400x41. During training, due to limited memory, the image is cropped along the x-axis to obtain multiple small images of 320x40x41. During testing, the results of multiple small images are spliced to obtain the segmentation result of the overall image. The data resolution is 3.24 pm (y), 6.7 pm (x), and 67 pm (z). The data provides artificial annotation of three retinal layer surfaces; the inner boundary of the inner limiting membrane (ILM), the inner part of the retinal pigment epithelium drusen complex (IRPE), and the outer part of the Bruchs membrane (OBM).

[0204] The JHH dataset includes OCT data collected from 14 normal people and 21 people with multiple sclerosis (MS). The data is collected by the Spectralis OCT system (Heidelberg Engineering, Heidelberg, Germany), which has its own motion correction, registration, and averaging algorithm, so the OCT images in the JHH dataset can be considered completely aligned between B-Scans. Each OCT data includes 49 B-Scans, each of which is 496*1024 in size. After preprocessing by the flattening algorithm, the overall 3D data size is 128x1024x49, and the data size is 128x48x49 after cropping along the x-axis. The data resolution is 3.9 pm (y), 5.8 pm (x), and 123.6 pm (z). The last 6 normal samples and the last 9 MS samples in the JHH dataset are used as the training set, and the other data is used as the test set. There are 9 corneal layer surfaces in the JHH dataset, which are annotated as follows: retinal nerve fiber layer (RNFL); ganglion cell layer (GCL) combined with inner plexiform layer (IPL), denoted as GCIP; inner nuclear layer (INL); outer plexiform layer (OPL); outer nuclear layer (ONL); inner segment (IS); outer segment (OS); and retinal pigment epithelium (RPE). The surfaces between these layers are represented by hyphens, such as RNFL-GCL, which represents the surface before the RNFL layer and the GCL layer.

[0205] In step 502, an initialized image processing model is called based on the training data to perform a training task.

[0206] In an example, the encoder G f Features of the 2D B-Scans are extracted, and then the network G aThe alignment decoder) will generate the displacement field of the B-Scans These displacement fields will act on the encoder G f Among the generated features at each level, the alignment features are obtained, which will be generated by the segmentation network G s The segmentation decoder) generates the segmentation results of the retinal layers. For the alignment network, the normalized cross-corelation loss (Normalized cross-corelation loss) And the smooth alignment loss (Smooth alignment loss) The model is trained together.

[0207] In an example, the embodiment of the present application adopts a two-dimensional convolutional neural network encoder G f Feature extraction is performed on the B-Scans in the OCT data to obtain two-dimensional feature input into the alignment decoder. The embodiment of the present application will adopt a 2D convolutional neural network, and in order to reduce the parameter amount of the network, the embodiment of the present application halves the channel number of each layer of the neural network in the original network.

[0208] In the image acquisition process, each B-Scan is acquired separately, and due to the global alignment and inevitable eye movement, the consecutive B-Scans in an OCT image may be misaligned. Misalignment mainly occurs in the A-Scan (y-axis) direction, which is also the direction considered by the embodiment of the present application. The embodiment of the present application proposes a B-Scans alignment decoder G a composed of a 3D convolutional neural network, which takes the 2D features generated by the feature extraction network G f as input and generates a displacement vector indicating the displacement of each B-Scan in the y-axis direction The embodiment of the present application uses the normalized cross-corelation loss as the loss function of the alignment network G a , as shown in the following formula (1).

[0209]

[0210] Where p traverses all pixels in the image space Φ, is the image after the displacement of the bth B-scan image according to the corresponding displacement field d b , is the image in the kth pixel.

[0211] is the image obtained by subtracting the local average value from the image after displacement, p kRepresenting the traversal of p around n 2 In this embodiment, n = 9 pixels. The local average value is represented as...

[0212] On the other hand, for well-aligned B-Scans, the positions of the retinal layers between adjacent B-Scans should be relatively close. Therefore, this application proposes to use the position markings of the retinal layers as a supervisory aid for alignment, referring to formula (2):

[0213]

[0214] Where, r g This is the marked location of the retinal layer, d b Characterizing the displacement field corresponding to the b-th B-Scan, r b,a That is, the position of the retinal surface on the a-scan within the b-th B-scan. This refers to the labeled position of the retinal surface on the a-th A-scan within the b-th B-scan. Finally, the loss function of the alignment network... In summary:

[0215] For example, the encoder G of this application uses a two-dimensional convolutional neural network. f Feature extraction is performed on B-scans in the OCT data to obtain two-dimensional features, which are then input into the alignment decoder. This embodiment employs a 2D convolutional neural network, and to reduce the number of network parameters, the number of channels in each layer of the original network is halved.

[0216] The segmentation network G in this application embodiment s This application extends the framework of Fully Convolutional Boundary Regression (FCBR). For 3D surface segmentation, embodiments of this application employ a 2D-3D hybrid network. Embodiments of this application propose a global smoothing loss to make the planes predicted by the model more 3D smooth.

[0217] To make the framework more complete and to allow the alignment and segmentation modules to benefit each other, embodiments of this application couple the alignment network and the segmentation network together. Alignment decoder G a Based on feature extraction network (encoder G) f The B-Scan displacement field generated from the extracted features will be used for alignment and transmitted to the segmentation decoder G. s The features are designed to ensure that the alignment decoder can regress a smooth surface. To achieve this, embodiments of this application propose adding a Spatial Transformer Module (STM) to the network. This module will transfer the encoder G... fAll the features of the segmentation decoder G s are aligned (including the highest level feature and the straight connected features of each level). In order to align the straight connected features of each level, the displacement field Δd needs to be adjusted to the corresponding size. In addition, the process of alignment can allow the back propagation of gradient in the training, so as to ensure that the whole optimization process can be carried out end to end. The embodiments of the present application align the network G a and the segmentation network G s through the spatial movement module STM, which simplifies the whole framework while the experiments prove that it improves the performance of segmentation.

[0218] The segmentation network G s has two output heads. The first output head 401A outputs the distribution of the retinal layer surface of each A-Scan, and the second output head 402A outputs the pixel-level segmentation label (layer wise map). The second head is only added to provide an additional task for the network, so that the network can be better trained by using the pixel-level supervision information (segmentation label), and the output of the second head is not considered in the test. The embodiments of the present application use Dice and cross-entropy loss as the training loss of the second output head.

[0219] For the first output head 401A, its output is the distribution probability q b,a (r|V; θ) of the retinal layer position in each A-Scan in the bth B-Scan, where θ is the parameter of the network, and the higher the distribution probability is, the higher the probability of the position r on the retinal surface is. The embodiments of the present application use cross-entropy loss to train the first output head 401A, as shown in the following formula (3):

[0220]

[0221] where q represents the distribution probability, q b,a (r|V; θ) is the distribution of the retinal layer position in the A-Scan, R is the height of the A-Scan, and 1(x) is an indicator function, x represents whether r is equal to r, that is, whether the labeled position is equal to the predicted position, and when x is true, the value of 1(x) is 1, otherwise it is 0. In addition, the smooth L1 loss (Mean Absolute Error, MAE) is used to directly guide the predicted surface position to be close to the true value, as shown in the following formula (4).

[0222]

[0223] wherein, characterize the difference between the optimal predicted position and the actual position; is obtained by a soft-argmax function: is the r that maximizes the distribution probability.

[0224] The embodiment of the present application adopts a global smoothing loss to guide the predicted surface position to have more three-dimensional smoothness, referring to formula (5):

[0225]

[0226] The embodiment of the present application uses the difference between the positions of adjacent points on the predicted surface position to represent the two-dimensional gradient of the surface, referring to the following formula (8):

[0227]

[0228] wherein, and respectively represent the pixel spacing in the z-axis direction and the y-axis direction in the OCT image. In summary, the total segmentation loss of the segmentation decoder (the first segmentation loss) can be summarized as: the cross-entropy loss the smoothing L1 loss and the global smoothing loss added together. Wherein, the global smoothing loss is multiplied by λ and added to other losses to obtain the total segmentation loss

[0229] wherein, λ is a hyperparameter for controlling the weight of the global smoothing loss. The value of λ is selected according to the actual smoothing degree of each layer. Specifically, for the lth retinal surface, the embodiment of the present application calculates λ using the following formula (7):

[0230]

[0231] wherein is the label of the lth retinal surface. The smoother the surface is, the greater the penalty for its overall coherence loss. In some embodiments, the embodiment of the present application uses the arithmetic mean of λ l of the OCT data in all training sets to calculate λ l .

[0232] In step 503, the training data is sparsely labeled, and the image processing model is trained based on the sparsely labeled training data.

[0233] For example, in the case of sparse labeling, the smooth alignment loss of the alignment network becomes For the segmentation network, cross-entropy loss is used Smooth L1 loss Global smooth loss Train its first output branch (Primary) using Dice and cross-entropy loss Train its second output branch (Secondary).

[0234] To reduce the dependence on full annotation and improve the clinical application value of the embodiments of the present application, the embodiments of the present application are further extended to the semi-supervised case based on sparse annotation. Since the scheme of the embodiments of the present application couples B-Scan alignment and retinal layer segmentation, by further utilizing the smoothness of the retinal layer, only a slight change is needed to the loss function , and the image processing model can be trained on OCT data with only part of the slices being annotated. Specifically, for the B-Scan alignment decoder, the curved surface position used in the smooth alignment loss is changed to the predicted position of the segmentation decoder G s , obtaining the following sparse annotation smooth alignment loss

[0235]

[0236] When the corresponding B-Scan is annotated, and when the corresponding B-Scan is not annotated, that is, the prediction result of the corresponding segmentation decoder. That is, the predicted segmentation position obtained by pre-training prediction is used as the label of the image without annotation.

[0237] The loss function of the alignment network becomes It is worth noting that the quality of the retinal layer prediction result output by the segmentation decoder G s , so the embodiments of the present application do not calculate the segmentation loss in the first five rounds of training to prevent poor segmentation results from affecting the quality of alignment.

[0238] For the segmentation network, cross-entropy loss is used Smooth L1 loss Global smooth loss Train its first output branch (Primary) using Dice and cross-entropy loss Train its second output branch (Secondary).

[0239] In some embodiments, the experimental implementation of the embodiments of the present application is completed on the PyTorch (1.4.0) neural network framework. The feature extraction network (encoder G f ) of the embodiments of the present application is a two-dimensional convolutional network, and in order to reduce the parameter quantity of the network, the channel number of each convolutional layer is halved. The Adam algorithm is used as the optimizer during network training, and the training round number is 120 rounds. The initial learning rate is 1e-3, and during training, if the loss function does not decrease for ten rounds, the learning rate is halved. The model is trained on three 2080Ti Nvidia graphics cards, and the batch size is 9.

[0240] In some embodiments, referring to Figure 6A , Figure 6A is an application mode schematic diagram of the training method of the image processing model provided by the embodiments of the present application. The training method of the image processing model provided by the embodiments of the present application runs in a cloud server.

[0241] The terminal device sends three-dimensional OCT data and sparse annotations to the cloud server, and the cloud server executes the training method of the image processing model to generate a corresponding image processing model.

[0242] In some embodiments, referring to Figure 6B , Figure 6B is an application mode schematic diagram of the image processing method provided by the embodiments of the present application; the image processing model trained by the training method of the image processing model provided by the embodiments of the present application is stored in a cloud server.

[0243] The terminal device sends three-dimensional OCT data to the cloud server, and the cloud server calls the trained image processing model to perform image processing to obtain aligned three-dimensional OCT data and segmentation results.

[0244] In some embodiments, the embodiments of the present application can use other segmentation networks, such as Res-UNet or a segmentation network based on Transformer; can be used for other modal 3D data with layer structure, referring to Figure 8D , Figure 8D is a schematic diagram of the image processing result provided by the embodiments of the present application. Figure 8D The image processing result of the intravascular ultrasound (IVUS) image is shown. The segmentation line 801D represents the lumen on the IVUS, and the segmentation line 802D represents the media.

[0245] Effects of the embodiments of the present application:

[0246] (1) The embodiment of the present application can be trained in a weakly supervised scene based on sparse annotation, and the obtained model can simultaneously align and segment the OCT data to obtain aligned OCT data and smoothed retinal layer surface.

[0247] (2) The embodiment of the present application is verified on two public data sets, and the test results based on full annotation and sparse annotation are better than other SOTA methods.

[0248] The effects of the embodiment of the present application are explained and described below in combination with experimental data.

[0249] Reference Figure 8A , Figure 8A is a schematic diagram of the image processing result provided by the embodiment of the present application. It shows the alignment results of different methods (Fig. d is the alignment result corresponding to the embodiment of the present application) on the A2A data test set, including three-dimensional visualization diagram, OCT base map (obtained by averaging the pixel values between OBM surface and IRPE surface) and cross-B-Scan cross-section diagram. The arrow indicates the place where the alignment visualization effect of the method of the embodiment of the present application is better.

[0250] Reference Figure 7A , Figure 7A is an experimental result table provided by the embodiment of the present application; it characterizes the B-Scan alignment result on the A2A OCT data, which corresponds to the segmentation result in Figure 8A . The table of Figure 7A quantitatively analyzes the alignment degree of the alignment result. The embodiment of the present application uses the average distance of the manually annotated retinal layer position between adjacent B-Scans to measure the good and bad of the alignment (the better the alignment, the closer the retinal layer of adjacent B-Scans, and the lower the value), and the average normalized cross-correlation (NCC) of adjacent layers is also given in the table (the higher, the higher the correlation, that is, the more aligned). Figure 7A The quantitative results in the table of

[0251] Reference Figure 7D , Figure 7D is an experimental result table provided by the embodiment of the present application; in Figure 7DThe document presents a comparison of the segmentation results of the proposed embodiment and other technical solutions on two datasets based on full supervision. The evaluation metric is the mean absolute distance (MAD) between the predicted and labeled surface locations, and the variance and significance of the differences are also given. ReLayNet and MGU-Net are the results of testing on the dataset used in the proposed embodiment using the officially released code, FCBR is the result of reproducing the 2D segmentation scheme from the literature, and Proposed is the test result using the proposed embodiment.

[0252] Figure 7D This paper presents the surface segmentation results of different methods on two public datasets, quantified by the mean absolute distance (MAD(μm)) between the predicted and labeled surface locations. An asterisk indicates a statistically significant difference between the corresponding method and the method proposed in this application (*: p < 0.05; **: p < 0.01; ***: p < 0.001).

[0253] refer to Figure 7B as well as Figure 7C , Figure 7B as well as Figure 7C This is a table of experimental results provided in the embodiments of this application; Figure 7B as well as Figure 7C This demonstrates the results in two datasets ( Figure 7B Corresponding to the A2A dataset, Figure 7C (Corresponding to the JHH dataset), the mean absolute distance (MADs (μm)) between the predicted and labeled surface positions in the test set obtained using different sampling ratios is used (the black line represents the standard deviation). The dashed line indicates the performance of the method in this embodiment when the sampling ratio is 1 / 8. The asterisk indicates a statistically significant difference between the corresponding method and the method proposed in this embodiment (*: p < 0.05; **: p < 0.01; ***: p < 0.001).

[0254] Figure 7B as well as Figure 7C The paper presents a comparison of B-Scan alignment results using different methods in OCT data. Based on the embodiments of this application, it can be determined that there is a significant misalignment between B-Scans in the unaligned data. Among the three alignment methods (Montuoro, NoRMCore, and the method of this application embodiment), NoRMCore and the method of this application embodiment can make the B-Scans significantly more aligned.

[0255] Depend on Figure 7DThe embodiments of the present application can draw several conclusions: first, compared with ReLayNet and MGU-Net, the average error of the method of the embodiments of the present application is obviously improved on two data sets (most p values are less than 0.05), and compared with the current best algorithm FCBR based on 2D segmentation, it can be seen that the algorithm of the embodiments of the present application has obtained lower mean absolute error on two data sets. The segmentation results of the sample data on two data sets of the technical solutions of the embodiments of the present application and the technical solutions in FCBR are shown.

[0256] In order to simulate sparse annotations of different sparsity, the embodiments of the present application uniformly sample the original slice annotations with different fraction factors. For example, a fraction factor of 1 / 8 means that only one B-Scan out of every 8 B-Scans is annotated. Note that in order to avoid unnecessary boundary effects, the annotations of the first and last B-Scans in the OCT three-dimensional data are always included. In the extreme case, for a three-dimensional OCT data, only three B-Scans are annotated, namely the first, the last and the middle B-Scan. In addition to the SOTA fully supervised methods compared above, the embodiments of the present application also include an uncertainty-guided semi-supervised method developed specifically for OCT corneal layer segmentation, namely U-SLS, for comparison in a semi-supervised environment.

[0257] In some embodiments, the sparse annotation results on the A2A and JHH data sets are as shown in Figure 8B and Figure 8C . Referring to Figure 8B and Figure 8C , Figure 8B is a schematic diagram of an image processing result provided by the embodiments of the present application, Figure 8B shows the segmentation result visualization of the A2A data set: the segmentation result of the artificial annotation, the segmentation result of FCBR, and the segmentation result of the technical solution of the embodiments of the present application. The upper row is an AMD patient sample, and the lower row is a normal patient sample. The arrows indicate the positions where the segmentation result of the embodiments of the present application is better than other segmentation results. Figure 8C shows the segmentation result visualization of the JHH data set: the segmentation result of the artificial annotation 801C, the segmentation result of FCBR 802C, and the segmentation result of the technical solution of the embodiments of the present application 803C. The arrows indicate the positions where the segmentation result of the embodiments of the present application is better than other segmentation results.

[0258] As the number of annotated B-Scans decreases, the performance of all methods degrades. However, the degree of degradation varies, and the method of embodiments of the present application maintains the best performance for all evaluated score factors in both datasets. Specifically, for score factors below 1 / 12, the performance of the method of embodiments of the present application is relatively stable, and in three extreme cases of annotation (1 / 20 and 1 / 24 score factors on the A2A and JHH datasets), the advantage of the method of embodiments of the present application over other methods becomes more prominent, even better than the performance of ReLayNet, MGU-Net and U-SLS in the full annotation case, demonstrating the practicability of the method of embodiments of the present application in the case of sparse annotation. The superior performance of the method is attributed to the effective use of unannotated B-Scan images by strengthening the 3D surface coherence of the retinal layers and coupling B-Scan segmentation and motion correction.

[0259] The following continues to illustrate an exemplary structure of the implementation of the training device 455 of the image processing model as a software module, in some embodiments, as shown in Figure 2A The software module stored in the training device 455 of the image processing model in the memory 450 can include: a sample acquisition module 4551 configured to acquire a plurality of three-dimensional image samples and annotation data, wherein the types of three-dimensional image samples include annotated image samples and unannotated image samples, and the annotation data includes the actual segmentation position of the annotated image samples; a pre-training module 4552 configured to call the image processing model based on the plurality of three-dimensional image samples to perform image segmentation processing to obtain a first predicted segmentation position, and update the image processing model based on a first loss between the first predicted segmentation position and the annotation data; a model updating module 4553 configured to take the first predicted segmentation position of the unannotated image sample as the actual segmentation position of the unannotated image sample, wherein the first predicted segmentation position is generated by calling the image processing model; the model updating module 4553 is further configured to update the first loss based on the actual segmentation position of each three-dimensional image sample to obtain a second loss; the model updating module 4553 is further configured to update the image processing model based on the first loss after the first loss is updated.

[0260] In some embodiments, the first loss includes a first alignment loss and a first segmentation loss; the pre-training module 4552 is configured to call the image processing model based on each three-dimensional image sample to perform the following processing: acquiring an alignment feature of the three-dimensional image sample; performing segmentation processing on the alignment feature to obtain a first predicted segmentation position corresponding to the three-dimensional image sample; determining a first alignment loss based on each alignment feature and the annotation data; determining a first segmentation loss based on each first predicted segmentation position and the annotation data; and performing parameter update processing on the image processing model based on the first alignment loss and the first segmentation loss.

[0261] In some embodiments, the pre-training module 4552 is configured to perform feature extraction processing on the three-dimensional image sample to obtain image features, perform displacement processing on the image features to obtain displacement vector features, and perform alignment processing on the displacement vector features and the image features to obtain aligned features of the three-dimensional image sample.

[0262] In some embodiments, the image processing model comprises: a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network.

[0263] The feature extraction network is configured to perform feature extraction processing, the alignment network is configured to perform displacement processing, the spatial displacement module is configured to perform alignment processing, and the segmentation network is configured to perform segmentation processing.

[0264] In some embodiments, the pre-training module 4552 is configured to perform the following processing on any two adjacent three-dimensional image samples in the plurality of three-dimensional image samples: obtain a sub-cross-correlation loss between the aligned features of the two three-dimensional image samples; obtain a first difference value between the actual segmentation position of each three-dimensional image sample and the displacement vector features, obtain a second difference value between two first difference values corresponding to the two three-dimensional image samples, and take the square of the second difference value as a first square term; add each sub-cross-correlation loss to obtain a cross-correlation loss, and add each first square term to obtain a smooth alignment loss; and take the sum of the cross-correlation loss and the smooth alignment loss as a first alignment loss.

[0265] In some embodiments, the pre-training module 4552 is configured to extract a first average value of a plurality of pixel values around each pixel in the first three-dimensional image sample from the aligned features of the first three-dimensional image sample, extract a second average value of a plurality of pixel values around each pixel in the second three-dimensional image sample from the aligned features of the second three-dimensional image sample, take any one pixel in the first three-dimensional image sample and a pixel at the same position in the second three-dimensional image sample as a pixel pair, and determine a sub-cross-correlation loss of the two three-dimensional image samples based on a difference between the first average value and the second average value of the pixel pair.

[0266] In some embodiments, the result of the segmentation processing comprises: a distribution probability of each position in the three-dimensional image sample, and a first predicted segmentation position.

[0267] The pre-training module 4552 is configured to determine a cross-entropy loss based on the distribution probability of each actual segmentation position in the three-dimensional image sample and the first predicted segmentation position, determine an absolute average error loss based on the difference between each first predicted segmentation position and the actual segmentation position, determine a surface two-dimensional gradient of each position in the three-dimensional image sample based on the position difference between two adjacent first predicted segmentation positions, and take the sum of the norms of each surface two-dimensional gradient as a global smoothing loss, and add the cross-entropy loss, the average error loss, and the global smoothing loss to obtain the first segmentation loss.

[0268] In some embodiments, the pre-training module 4552 is configured to perform parameter update processing on the alignment network and the spatial displacement module based on the first alignment loss to obtain the parameter updated alignment network and the spatial displacement module, and perform parameter update processing on the segmentation network based on the first segmentation loss to obtain the parameter updated segmentation network.

[0269] In some embodiments, the segmentation network includes a first output head and a second output head, the second output head is configured to output the distribution probability of each position in the three-dimensional image sample, and the first output head is configured to output the first predicted segmentation position; the pre-training module 4552 is configured to perform parameter update processing on the first output head based on the cross-entropy loss, the global smoothing loss, and the average error loss to obtain the parameter updated first output head, and perform parameter update processing on the second output head based on the cross-entropy loss to obtain the parameter updated second output head.

[0270] In some embodiments, the first loss includes the first alignment loss and the first segmentation loss; the model updating module 4553 is configured to update the actual segmentation position in the smooth alignment loss included in the first alignment loss based on the actual segmentation position of each three-dimensional image sample re-labeled to obtain a second alignment loss, update the predicted segmentation position in the global smoothing loss included in the first segmentation loss based on the actual segmentation position of each three-dimensional image sample re-labeled to obtain a second segmentation loss, and combine the second alignment loss and the second segmentation loss to obtain a second loss.

[0271] In some embodiments, the image processing model includes a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network; the second loss includes a second alignment loss and a second segmentation loss; and the model updating module 4553 is configured to perform parameter update processing on the alignment network and the spatial displacement module based on the second alignment loss to obtain the parameter updated alignment network and the spatial displacement module, and perform parameter update processing on the segmentation network based on the second segmentation loss to obtain the parameter updated segmentation network.

[0272] The embodiment of the present application also provides an image processing device. The following continues to describe an exemplary structure of the image processing device 456 provided by the embodiment of the present application in the form of a software module. In some embodiments, as shown in the figure, the software module stored in the image processing device 456 of the memory 450 can include: an image receiving module 4554, configured to acquire a three-dimensional image to be segmented; and an image processing module 4555, configured to call an image processing model based on the three-dimensional image to perform image segmentation processing, to obtain a segmentation position in the three-dimensional image, wherein the image processing model is obtained by the training method of the image processing model provided by the embodiment of the present application. Figure 2B

[0273] In some embodiments, the image processing module 4555 is configured to call the image processing model based on the three-dimensional image to perform the following processing: performing feature extraction processing on the three-dimensional image to obtain image features of the three-dimensional image; performing displacement processing on the image features to obtain displacement vector features; performing alignment processing on the displacement vector features and the image features to obtain aligned features of the three-dimensional image; and performing segmentation processing on the aligned features to obtain the segmentation position corresponding to the three-dimensional image.

[0274] The embodiment of the present application provides a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. The processor of a computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the training method of the image processing model provided by the embodiment of the present application, or the image processing method provided by the embodiment of the present application.

[0275] The embodiment of the present application provides a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions are stored in the computer readable storage medium. When the computer executable instructions are executed by the processor, the processor will execute the training method of the image processing model provided by the embodiment of the present application, for example, the training method of the image processing model as shown in the figure. Figure 3A Or the processor will execute the image processing method provided by the embodiment of the present application.

[0276] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0277] ​In some embodiments, the computer-executable instructions can be in the form of programs, procedures, modules, scripts, or code, and can be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0278] By way of example, a computer-executable instruction can be, but is not limited to, a file in a file system, can be stored in a part of a file that holds other programs or data, for example, one or more scripts stored in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, sub programs, or code portions.

[0279] By way of example, a computer-executable instruction can be deployed to execute on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected through a communication network.

[0280] In summary, the embodiments of the present application improve the accuracy of training the image processing model by training in two stages, label the unlabeled training data based on the first predicted segmentation position obtained by prediction in a sparse labeling manner to update the loss of the image processing model, and optimize and train the image processing model based on the updated loss, thereby improving the accuracy of the image processing model obtained by training and saving the computing resources required for labeling the training data.

[0281] The above merely describes the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training an image processing model, characterized in that, The method comprises: obtaining a plurality of three-dimensional image samples and annotation data, wherein the types of the three-dimensional image samples include annotated image samples and unannotated image samples, and the annotation data includes actual segmentation positions of the annotated image samples; calling the image processing model based on the plurality of three-dimensional image samples to perform image segmentation processing to obtain first predicted segmentation positions, and updating the image processing model based on a first loss between the first predicted segmentation positions and the annotation data, wherein the first loss includes a first alignment loss, and the first alignment loss is determined by performing the following processing: for any two adjacent three-dimensional image samples in the plurality of three-dimensional image samples, performing the following processing: extracting a first average value of a plurality of pixel values around each pixel in a first three-dimensional image sample from alignment features of the first three-dimensional image sample; extracting a second average value of a plurality of pixel values around each pixel in a second three-dimensional image sample from alignment features of the second three-dimensional image sample; taking any one pixel in the first three-dimensional image sample and a pixel at the same position in the second three-dimensional image sample as a pixel pair, and determining a sub-cross-correlation loss of the two three-dimensional image samples based on a difference between the first average value and the second average value of the pixel pair; obtaining a first difference value between an actual segmentation position and a displacement vector feature of each three-dimensional image sample, obtaining a second difference value between two first difference values corresponding to two three-dimensional image samples, and taking a square of the second difference value as a first square term; adding each sub-cross-correlation loss to obtain a cross-correlation loss, and adding each first square term to obtain a smooth alignment loss; and taking a sum of the cross-correlation loss and the smooth alignment loss as the first alignment loss; taking the first predicted segmentation position of the unannotated image sample as an actual segmentation position of the unannotated image sample, wherein the first predicted segmentation position is generated by calling the image processing model; updating the first loss based on the actual segmentation position of each three-dimensional image sample to obtain a second loss; updating the image processing model based on the second loss based on the first loss.

2. The method of claim 1, wherein, The first loss further includes a first segmentation loss; The method comprises: calling the image processing model based on each three-dimensional image sample to perform the following processing: obtaining alignment features of the three-dimensional image sample; performing segmentation processing on the alignment features to obtain a first predicted segmentation position corresponding to the three-dimensional image sample; determining the first segmentation loss based on each first predicted segmentation position and the annotation data; performing parameter update processing on the image processing model based on the first alignment loss and the first segmentation loss.

3. The method of claim 2, wherein, The acquiring the alignment feature of the three-dimensional image sample comprises: performing feature extraction processing on the three-dimensional image sample to obtain image features; performing displacement processing on the image features to obtain displacement vector features; performing alignment processing on the displacement vector features and the image features to obtain the alignment feature of the three-dimensional image sample.

4. The method of claim 3, wherein, The image processing model comprises: a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network. The feature extraction network is configured to perform the feature extraction processing, the alignment network is configured to perform the displacement processing, the spatial displacement module is configured to perform the alignment processing, and the segmentation network is configured to perform the segmentation processing.

5. The method of claim 4, wherein, The result of the segmentation processing comprises: a distribution probability of each position in the three-dimensional image sample and the first predicted segmentation position. The determining a first segmentation loss based on each of the first predicted segmentation positions and the annotation data comprises: determining a cross-entropy loss based on the distribution probability of each actual segmentation position in the three-dimensional image sample and the first predicted segmentation position; determining an absolute average error loss based on a difference between each of the first predicted segmentation positions and an actual segmentation position; determining a surface two-dimensional gradient of each position in the three-dimensional image sample based on a position difference value between two adjacent first predicted segmentation positions, and taking a sum of norms of each of the surface two-dimensional gradients as a global smoothness loss; adding the cross-entropy loss, the average error loss, and the global smoothness loss to obtain the first segmentation loss.

6. The method of claim 5, wherein, The performing parameter update processing on the image processing model based on the first alignment loss and the first segmentation loss comprises: performing parameter update processing on the alignment network and the spatial displacement module based on the first alignment loss to obtain the alignment network and the spatial displacement module after parameter update; performing parameter update processing on the segmentation network based on the first segmentation loss to obtain the segmentation network after parameter update.

7. The method of claim 6, wherein, The segmentation network comprises a first output head and a second output head; the second output head is configured to output the distribution probability of each position in the three-dimensional image sample, and the first output head is configured to output the first predicted segmentation position. The performing parameter update processing on the segmentation network based on the first segmentation loss to obtain the segmentation network after parameter update comprises: performing parameter update processing on the first output head based on the cross-entropy loss, the global smoothness loss, and the average error loss to obtain the first output head after parameter update; performing parameter update processing on the second output head based on the cross-entropy loss to obtain the second output head after parameter update.

8. The method of claim 1, wherein: the first loss comprises a first alignment loss and a first segmentation loss; the updating the first loss based on each actual segmentation position of the three-dimensional image sample to obtain a second loss comprises: updating, based on the actual segmentation position of each of the three-dimensional image samples after re-labeling, the predicted segmentation position in the global smoothing loss included in the first segmentation loss to obtain a second segmentation loss; updating, based on the actual segmentation position of each of the three-dimensional image samples after re-labeling, the actual segmentation position in the smooth alignment loss included in the first alignment loss to obtain a second alignment loss; combining the second alignment loss and the second segmentation loss to obtain a second loss.

9. The method of claim 1, wherein the image processing model comprises a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network; and the second loss comprises a second alignment loss and a second segmentation loss. updating the image processing model updated based on the first loss based on the second loss comprises: updating the parameters of the alignment network and the spatial displacement module based on the second alignment loss to obtain the alignment network and the spatial displacement module after parameter updating; and updating the parameters of the segmentation network based on the second segmentation loss to obtain the segmentation network after parameter updating.

10. An image processing method characterized by, The method comprises: obtaining a three-dimensional image to be segmented; calling an image processing model based on the three-dimensional image to perform image segmentation processing to obtain a segmentation position in the three-dimensional image, wherein the image processing model is obtained by the training method of the image processing model of any one of claims 1 to 9.

11. The method of claim 10, wherein, The calling of the image processing model based on the three-dimensional image to perform image segmentation processing to obtain a segmentation position in the three-dimensional image comprises: calling the image processing model based on the three-dimensional image to perform the following processing: performing feature extraction processing on the three-dimensional image to obtain image features of the three-dimensional image; performing displacement processing on the image features to obtain displacement vector features; performing alignment processing on the displacement vector features and the image features to obtain aligned features of the three-dimensional image; performing segmentation processing on the aligned features to obtain a segmentation position corresponding to the three-dimensional image.

12. An apparatus for training an image processing model, comprising: The training device of the image processing model comprises: a sample acquisition module configured to acquire a plurality of three-dimensional image samples and annotation data, wherein the types of the three-dimensional image samples include labeled image samples and unlabeled image samples, and the annotation data includes actual segmentation positions of the labeled image samples; The pre-training module is configured to call the image processing model based on the plurality of three-dimensional image samples to perform image segmentation processing, to obtain a first predicted segmentation position, and to update the image processing model based on a first loss between the first predicted segmentation position and the annotation data, wherein the first loss comprises a first alignment loss, and the first alignment loss is determined by performing the following processing: for any two adjacent three-dimensional image samples in the plurality of three-dimensional image samples, performing the following processing: extracting a first average value of a plurality of pixel values around each pixel in the first three-dimensional image sample from the alignment feature of the first three-dimensional image sample; extracting a second average value of a plurality of pixel values around each pixel in the second three-dimensional image sample from the alignment feature of the second three-dimensional image sample; taking any one pixel in the first three-dimensional image sample and a pixel at the same position in the second three-dimensional image sample as a pixel pair, and determining a sub-cross-correlation loss of the two three-dimensional image samples based on a difference between the first average value and the second average value of the pixel pair; obtaining a first difference value between the actual segmentation position of each three-dimensional image sample and the displacement vector feature, obtaining a second difference value between two first difference values corresponding to the two three-dimensional image samples, and taking a square of the second difference value as a first square term; adding each sub-cross-correlation loss to obtain a cross-correlation loss, and adding each first square term to obtain a smooth alignment loss; and taking a sum of the cross-correlation loss and the smooth alignment loss as the first alignment loss; The model updating module is configured to take the first predicted segmentation position of the unlabeled image sample as an actual segmentation position of the unlabeled image sample, wherein the first predicted segmentation position is generated by calling the image processing model; The model updating module is further configured to update the first loss based on the actual segmentation position of each three-dimensional image sample to obtain a second loss; The model updating module is further configured to update the image processing model based on the second loss, which is updated based on the first loss.

13. The apparatus of claim 12, wherein, The first loss further comprises a first segmentation loss; The pre-training module is further configured to call the image processing model based on each three-dimensional image sample to perform the following processing: obtaining an alignment feature of the three-dimensional image sample; performing segmentation processing on the alignment feature to obtain a first predicted segmentation position corresponding to the three-dimensional image sample; and determining the first segmentation loss based on each first predicted segmentation position and the annotation data; The image processing model is updated based on the first alignment loss and the first segmentation loss.

14. The apparatus of claim 13, wherein The pre-training module is further configured to perform feature extraction processing on the three-dimensional image sample to obtain an image feature, and to perform displacement processing on the image feature to obtain a displacement vector feature; The displacement vector feature and the image feature are aligned to obtain an alignment feature of the three-dimensional image sample.

15. The apparatus of claim 14, wherein, The image processing model comprises a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network. The feature extraction network is configured to perform the feature extraction processing, the alignment network is configured to perform the displacement processing, the spatial displacement module is configured to perform the alignment processing, and the segmentation network is configured to perform the segmentation processing.

16. The apparatus of claim 15, wherein, The result of the segmentation processing comprises a distribution probability of each position in the three-dimensional image sample and the first predicted segmentation position. The pre-training module is further configured to determine a cross-entropy loss based on the distribution probability of each actual segmentation position in the three-dimensional image sample and the first predicted segmentation position, determine an absolute average error loss based on a difference between each first predicted segmentation position and an actual segmentation position, determine a surface two-dimensional gradient of each position in the three-dimensional image sample based on a position difference between two adjacent first predicted segmentation positions, and take a sum of norms of each surface two-dimensional gradient as a global smoothness loss, and add the cross-entropy loss, the average error loss, and the global smoothness loss to obtain the first segmentation loss.

17. The apparatus of claim 16, wherein The pre-training module is further configured to perform parameter update processing on the alignment network and the spatial displacement module based on the first alignment loss to obtain the parameter-updated alignment network and the parameter-updated spatial displacement module. The pre-training module is further configured to perform parameter update processing on the segmentation network based on the first segmentation loss to obtain the parameter-updated segmentation network.

18. The apparatus of claim 17, wherein, The segmentation network comprises a first output head and a second output head, the second output head is configured to output the distribution probability of each position in the three-dimensional image sample, and the first output head is configured to output the first predicted segmentation position. The pre-training module is further configured to perform parameter update processing on the first output head based on the cross-entropy loss, the global smoothness loss, and the average error loss to obtain the parameter-updated first output head, and perform parameter update processing on the second output head based on the cross-entropy loss to obtain the parameter-updated second output head.

19. The apparatus of claim 12, wherein, The first loss comprises a first alignment loss and a first segmentation loss. The model update module is further configured to update an actual segmentation position in a smooth alignment loss included in the first alignment loss based on the re-labeled actual segmentation position of each three-dimensional image sample to obtain a second alignment loss, update a predicted segmentation position in a global smoothness loss included in the first segmentation loss based on the re-labeled actual segmentation position of each three-dimensional image sample to obtain a second segmentation loss, and combine the second alignment loss and the second segmentation loss to obtain a second loss.

20. The apparatus of claim 12, wherein, The image processing model comprises a feature extraction network, an alignment network, a spatial displacement module, and a segmentation network, and the second loss comprises a second alignment loss and a second segmentation loss. The model updating module is further configured to perform parameter updating processing on the alignment network and the spatial displacement module based on the second alignment loss to obtain the parameter-updated alignment network and the parameter-updated spatial displacement module, and perform parameter updating processing on the segmentation network based on the second segmentation loss to obtain the parameter-updated segmentation network.

21. An image processing apparatus characterized by comprising: The image processing apparatus comprises: An image receiving module configured to acquire a three-dimensional image to be segmented; An image processing module configured to call an image processing model based on the three-dimensional image to perform image segmentation processing and obtain a segmentation position in the three-dimensional image, wherein the image processing model is obtained through the training method of the image processing model in any one of claims 1 to 9.

22. The apparatus of claim 21, wherein The image processing module is further configured to call an image processing model based on the three-dimensional image to perform the following processing: performing feature extraction processing on the three-dimensional image to obtain image features of the three-dimensional image; performing displacement processing on the image features to obtain displacement vector features; performing alignment processing on the displacement vector features and the image features to obtain aligned features of the three-dimensional image; and performing segmentation processing on the aligned features to obtain a segmentation position corresponding to the three-dimensional image. The electronic device comprises:

23. An electronic device, comprising: A memory for storing computer executable instructions; A processor for executing the computer executable instructions stored in the memory to implement the training method of the image processing model in any one of claims 1 to 9 or the image processing method in any one of claims 10 to 11. The computer executable instructions are executed by the processor to implement the training method of the image processing model in any one of claims 1 to 9 or the image processing method in any one of claims 10 to 11. 24.A computer readable storage medium storing computer executable instructions, wherein, The computer program or computer executable instructions are executed by the processor to implement the training method of the image processing model in any one of claims 1 to 9 or the image processing method in any one of claims 10 to 11.

25. A computer program product comprising computer programs or computer executable instructions, characterized in that, ​

Images