Image processing, image processing model training methods, devices, equipment and media

Abstract

This application discloses an image processing method, apparatus, device, and medium for training an image processing model, belonging to the field of artificial intelligence technology, specifically computer vision technology within the field of artificial intelligence. The image processing method includes: acquiring an image to be processed and an image processing model, wherein the image processing model includes an encoding model and a decoding model, the encoding model includes an attention model and a convolutional model, the attention model encodes based on global information, and the convolutional model encodes based on local information; calling the attention model and the convolutional model to encode the image to be processed based on global and local information to obtain target encoded features; calling the decoding model to decode the target encoded features to obtain target image features; and obtaining a segmentation result based on the target image features. In this method, the target image features are obtained by comprehensively focusing on both global and local information, resulting in richer information focus, higher reliability, and improved accuracy of the segmentation results.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and medium for training an image processing model. Background Technology

[0002] With the development of artificial intelligence technology, there are more and more application scenarios for image processing. One such application scenario is: an image contains sub-images of a reference object, and an image processing model is called to process the image to segment the sub-images and obtain the segmentation result of the image.

[0003] In related technologies, the image features used to obtain image segmentation results are obtained by focusing only on local information. The limited information focused on makes the reliability of image features poor, which in turn leads to poor accuracy of the obtained segmentation results. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, and medium for training an image processing model, which can be used to improve the reliability of image features, thereby improving the accuracy of the obtained segmentation results. The technical solution is as follows:

[0005] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0006] The image to be processed and the image processing model are obtained. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information.

[0007] The attention model and the convolution model are invoked to encode the image to be processed based on global and local information to obtain target encoded features;

[0008] The decoding model is invoked to decode the target encoded features to obtain the target image features;

[0009] Based on the target image features, the segmentation result of the image to be processed is obtained.

[0010] A method for training an image processing model is also provided, the method comprising:

[0011] Acquire a sample image, the label information of the sample image, and an initial image processing model, wherein the initial image processing model includes an initial encoding model and an initial decoding model, and the initial encoding model includes an initial attention model and an initial convolution model;

[0012] The initial attention model and the initial convolution model are invoked to encode the sample image based on global and local information to obtain sample encoding features;

[0013] The initial decoding model is invoked to decode the sample encoded features to obtain the sample image features;

[0014] Based on the features of the sample image, the segmentation result of the sample image is obtained;

[0015] Based on the segmentation results and label information of the sample images, the initial image processing model is trained to obtain the image processing model.

[0016] On the other hand, an image processing apparatus is provided, the apparatus comprising:

[0017] The first acquisition unit is used to acquire the image to be processed and the image processing model. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information.

[0018] The second acquisition unit is used to call the attention model and the convolution model to encode the image to be processed based on global and local information to obtain target encoding features;

[0019] The third acquisition unit is used to call the decoding model to decode the target encoded features to obtain the target image features;

[0020] The fourth acquisition unit is used to acquire the segmentation result of the image to be processed based on the features of the target image.

[0021] In one possible implementation, the number of encoding models is at least one. The second acquisition unit is used to call the attention model and convolution model in the first encoding model to encode the image to be processed based on global and local information, obtaining the basic features and connection features output by the first encoding model; starting from the second encoding model, the attention model and convolution model in the next encoding model are called to encode the basic features output by the previous encoding model based on global and local information, obtaining the basic features and connection features output by the next encoding model, until the basic features and connection features output by the penultimate encoding model are obtained. The connection features output by each encoding model from the first encoding model to the penultimate encoding model are used to provide data support for the step of calling the decoding model to decode the target encoded features to obtain the target image features; the attention model and convolution model in the last encoding model are called to encode the basic features output by the penultimate encoding model based on global and local information, obtaining the connection features output by the last encoding model, and the connection features output by the last encoding model are used as the target encoded features.

[0022] In one possible implementation, the attention model in the first encoding model includes a first attention model. The second acquisition unit is further configured to call the convolution model and the first attention model in the first encoding model to encode the image to be processed based on local and global information to obtain the basic features output by the first encoding model; and obtain the connection features output by the first encoding model based on the basic features output by the first encoding model.

[0023] In one possible implementation, the second acquisition unit is further configured to call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain a first encoding feature; call the first attention model to encode the image to be processed based on global information to obtain a second encoding feature; fuse the first encoding feature and the second encoding feature to obtain a fused feature; and acquire the basic feature output by the first encoding model based on the fused feature.

[0024] In one possible implementation, the second acquisition unit is further configured to acquire block features of each image block of the image to be processed, map the block features of each image block to obtain the mapped features of each image block; acquire image block position features; acquire reference features of the image to be processed based on the mapped features of each image block and the image block position features; and call the first attention model to encode the reference features of the image to be processed based on global information to obtain the second encoded features.

[0025] In one possible implementation, the first attention model includes an attention module and a nonlinear processing module. The second acquisition unit is further configured to call the attention module to process the reference feature to obtain a first intermediate feature; concatenate the first intermediate feature and the reference feature to obtain a feature to be processed; call the nonlinear processing module to process the feature to be processed to obtain a second intermediate feature; and concatenate the second intermediate feature and the feature to be processed to obtain a second encoded feature.

[0026] In one possible implementation, the second acquisition unit is further configured to call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain a first encoding feature; call the first attention model to encode the first encoding feature based on global information to obtain a third encoding feature; and acquire the basic feature output by the first encoding model based on the third encoding feature.

[0027] In one possible implementation, the second acquisition unit is further configured to call the first attention model to encode the image to be processed based on global information to obtain a second encoded feature; call the convolution model in the first encoding model to encode the second encoded feature based on local information to obtain a fourth encoded feature; and acquire the basic feature output by the first encoding model based on the fourth encoded feature.

[0028] In one possible implementation, the attention model in the first encoding model includes a second attention model. The second acquisition unit is further configured to call the convolution model in the first encoding model to encode the image to be processed based on local information to obtain the basic features output by the first encoding model; and call the second attention model to encode the basic features output by the first encoding model based on global information to obtain the connection features output by the first encoding model.

[0029] A training apparatus for an image processing model is also provided, the apparatus comprising:

[0030] The first acquisition unit is used to acquire a sample image, the label information of the sample image, and an initial image processing model. The initial image processing model includes an initial encoding model and an initial decoding model. The initial encoding model includes an initial attention model and an initial convolution model.

[0031] The second acquisition unit is used to call the initial attention model and the initial convolution model to encode the sample image based on global and local information to obtain sample encoding features;

[0032] The third acquisition unit is used to call the initial decoding model to decode the sample encoding features to obtain sample image features;

[0033] The fourth acquisition unit is used to acquire the segmentation result of the sample image based on the features of the sample image;

[0034] The training unit is used to train the initial image processing model based on the segmentation results and label information of the sample images to obtain the image processing model.

[0035] In one possible implementation, the sample image includes a sub-image of a reference object, and the label information of the sample image includes at least one of a point label, a first auxiliary label, or a second auxiliary label, wherein both the first auxiliary label and the second auxiliary label are obtained based on the point label, and the point label is determined based on a reference point within the region where the sub-image is located in the sample image.

[0036] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the computer device to implement any of the image processing methods or image processing model training methods described above.

[0037] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to enable a computer to implement any of the above-described image processing methods or image processing model training methods.

[0038] On the other hand, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the image processing methods or image processing model training methods described above.

[0039] The technical solution provided in this application has at least the following beneficial effects:

[0040] The technical solution provided in this application first calls an attention model and a convolution model to obtain target encoding features based on global and local information, and then obtains target image features based on the target encoding features. Since the target encoding features are obtained based on both global and local information, it ensures that the target image features obtained from the target encoding features are features obtained by comprehensively considering both global and local information. This results in richer information focus, leading to higher reliability of the target image features and thus improving the accuracy of the obtained segmentation results. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0043] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0044] Figure 3 This is a flowchart of a process provided in this application embodiment of calling the attention model and convolution model in the first encoding model to encode the image to be processed based on global and local information, and obtaining the basic features and connection features output by the first encoding model;

[0045] Figure 4 This is a schematic diagram illustrating a process provided in this application embodiment of calling the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoded feature;

[0046] Figure 5 This is a schematic diagram illustrating a process provided in this application embodiment of calling a first attention model to encode reference features of the image to be processed based on global information to obtain second encoded features;

[0047] Figure 6 This is a schematic diagram illustrating a process for acquiring target image features according to an embodiment of this application;

[0048] Figure 7 This is a flowchart of a training method for an image processing model provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of a reference object label and a dot label provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of a first auxiliary label provided in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram of a second auxiliary label provided in an embodiment of this application;

[0052] Figure 11 This is a flowchart of a method for processing histopathological images provided in an embodiment of this application;

[0053] Figure 12 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0054] Figure 13 This is a schematic diagram of a training device for an image processing model provided in an embodiment of this application;

[0055] Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0056] Figure 15 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0057] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0058] In exemplary embodiments, the image processing method and image processing model training method provided in this application can be applied to the field of artificial intelligence technology. The artificial intelligence technology will now be described.

[0059] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science. AI attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0060] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. The image processing methods and image processing model training methods provided in this application involve computer vision and machine learning technologies.

[0061] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (Three Dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0062] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0063] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0064] In an exemplary embodiment, the image processing method and the image processing model training method provided in this application embodiment are implemented in a blockchain system. The image to be processed, the image processing model, the segmentation result of the image to be processed, etc. involved in the image processing method provided in this application embodiment, as well as the sample image, the label information of the sample image, and the initial image processing model involved in the image processing model training method, are all stored on the blockchain in the blockchain system for use by various node devices in the blockchain system to ensure the security and reliability of the data.

[0065] Figure 1 A schematic diagram of an implementation environment provided in an embodiment of this application is shown. This implementation environment includes: a terminal 11 and a server 12.

[0066] The image processing method provided in this application embodiment can be executed by terminal 11, server 12, or jointly by terminal 11 and server 12; this application embodiment does not limit this. In the case where the image processing method provided in this application embodiment is jointly executed by terminal 11 and server 12, server 12 undertakes the main computational work, and terminal 11 undertakes the secondary computational work; or, server 12 undertakes the secondary computational work, and terminal 11 undertakes the main computational work; or, server 12 and terminal 11 use a distributed computing architecture for collaborative computation.

[0067] The image processing model training method provided in this application embodiment can be executed by terminal 11, server 12, or jointly by terminal 11 and server 12; this application embodiment does not limit this. In the case where the image processing model training method provided in this application embodiment is jointly executed by terminal 11 and server 12, server 12 undertakes the main computational work, and terminal 11 undertakes the secondary computational work; or, server 12 undertakes the secondary computational work, and terminal 11 undertakes the main computational work; or, server 12 and terminal 11 use a distributed computing architecture for collaborative computation.

[0068] The image processing method and image processing model training method provided in this application embodiment can be executed by the same device or by different devices, and this application embodiment does not limit this.

[0069] In one possible implementation, terminal 11 can be any electronic product capable of human-computer interaction with a user through one or more methods such as a keyboard, touchpad, touchscreen, remote control, voice interaction, or handwriting device, such as a PC (Personal Computer), mobile phone, smartphone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), tablet computer, smart car system, smart TV, smart speaker, etc. Server 12 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Terminal 11 and server 12 establish a communication connection through a wired or wireless network.

[0070] Those skilled in the art should understand that the above-described terminal 11 and server 12 are merely examples. Other existing or future terminals or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0071] Based on the above Figure 1 The implementation environment shown in this application embodiment provides an image processing method, which is executed by a computer device. This computer device can be a server 12 or a terminal 11; this application embodiment does not limit the specific type of computer device. Figure 2 As shown, the image processing method provided in this application embodiment includes the following steps 201 to 204.

[0072] In step 201, the image to be processed and the image processing model are obtained. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information.

[0073] The image to be processed is an image that needs to be processed to obtain segmentation results. For example, the image to be processed includes a sub-image of a reference object, which refers to the image region of interest included in the image to be processed. The process of processing the image to be processed refers to the process of segmenting the sub-image of the reference object from the image to be processed. This application does not limit the type of image to be processed; the type of reference object varies depending on the type of image to be processed.

[0074] For example, the image to be processed is a histopathological image obtained by acquiring images of diseased tissue in a pathological slide. The reference object is a small, numerous, and densely packed tissue in the histopathological image, such as cell nuclei, cells, or blood vessels. The pathological slide may have specific characteristics; for example, the pathological slide may be a pathological slide of biological tissue with a certain disease, such as a pathological slide of animal or plant tissue with a specific disease, or a pathological slide of tumor tissue in a certain part of the human body. For example, the pathological slide is a stained slide, specifically a slide stained with HE (Hematoxylin Eosin). By acquiring an image of a certain area of ​​the field of view in the HE-stained pathological slide, a histopathological image can be obtained, which is then used as the image to be processed.

[0075] For example, the image to be processed is a street scene image obtained by image acquisition of a street scene, and the reference object is certain specific elements in the street, such as vehicles, pedestrians, etc. For example, the image to be processed is an indoor image obtained by image acquisition of an indoor scene, and the reference object is certain specific facilities in the indoor space, such as tables, chairs, beds, etc.

[0076] This application does not limit the method of acquiring the image to be processed in the embodiments. In exemplary embodiments, the methods by which a computer device acquires the image to be processed include, but are not limited to: the computer device extracting the image to be processed from an image library; an image acquisition device having established a communication connection with the computer device sending the acquired image to be processed to the computer device; or the computer device acquiring an image to be processed uploaded by a user. In exemplary embodiments, the image to be processed may refer to the original image acquired by the image acquisition device, or it may be an image obtained after preprocessing the original image acquired by the image acquisition device. This application does not limit this. Exemplarily, the methods of preprocessing the original image include, but are not limited to, at least one of cropping, rotating, flipping, and data enhancement.

[0077] It should be noted that there may be one or more images to be processed. If there are multiple images to be processed, each image will obtain a segmentation result according to steps 202 to 204. This embodiment of the application will be described using a single image as an example.

[0078] Image processing models are used to process images. These models are obtained by training an initial image processing model. The method for training an image processing model can be found in [link to training documentation]. Figure 7The embodiments shown are not described in detail here. The image acquisition model in the embodiments of this application may refer to an image processing model trained in real time, or it may refer to extracting a pre-trained and stored image processing model. The embodiments of this application do not limit this.

[0079] Image processing models include encoding and decoding models. The encoding model acquires encoded features, and the decoding model decodes these encoded features to obtain image features. In other words, the encoding and decoding models are used to jointly extract image features. In this embodiment, the encoding model includes an attention model and a convolutional model. The attention model encodes based on global information, while the convolutional model encodes based on local information. For example, the global information used by the attention model refers to the global information corresponding to the information input to the attention model (e.g., features, images, etc.), and the local information used by the convolutional model refers to the local information corresponding to the information input to the convolutional model (e.g., features, images, etc.). That is, the encoding model provided in this embodiment can focus on both global and local information during the encoding process, thereby obtaining highly reliable encoded features, and consequently, highly reliable image features. For example, global information is obtained by focusing on long-distance dependencies.

[0080] Attention models are based on attention mechanisms and can acquire global contextual information (referred to as global information). Convolutional models are a type of CNN (Convolutional Neural Networks) model, which is a deep feedforward neural network that includes convolutional operations. CNN models can acquire fine local features through powerful convolutional operations. The convolutional operations used in CNNs employ weight sharing, and the receptive field is limited by the kernel size and network depth. Therefore, CNN-based network structures often suffer from insufficient receptive fields, especially when the reference object is large. Attention models based on attention mechanisms can effectively overcome this limitation. In other words, by designing an encoding model that includes both attention and convolutional models, the advantages of both models can be integrated, preserving global information while acquiring local features.

[0081] In an exemplary embodiment, the image processing model includes one or more encoding models, each of which includes at least one attention model and one convolutional model. When the image processing model includes multiple encoding models, the different encoding models have different arrangements; the encoding models placed earlier in the arrangement perform the encoding operation first, and the encoding models placed later in the arrangement perform the encoding operation last. Exemplarily, the specific details of the attention models and convolutional models included in different encoding models may be the same or different, and this embodiment does not limit this.

[0082] For example, the number of decoding models in the image processing model is the same as the number of encoding models. If the number of encoding models is N (N is an integer not less than 1), the number of decoding models is also N. For example, if there are multiple decoding models, the different decoding models have different arrangement positions; the decoding models with earlier positions are decoded first, and the decoding models with later positions are decoded last. For example, if there are multiple encoding and decoding models, the multiple encoding models and multiple decoding models form a U-shaped structure.

[0083] After obtaining the image to be processed and the image processing model, subsequent steps 202 to 204 are performed to obtain the segmentation result of the image to be processed.

[0084] In step 202, the attention model and convolution model are invoked to encode the image to be processed based on global and local information to obtain the target encoded features.

[0085] In the image processing model, the encoding model is used to obtain encoded features. After acquiring the image to be processed and the image processing model, the attention model and convolution model in the encoding model are called to encode the image to be processed based on global and local information, obtaining the target encoded features. The target encoded features are the features corresponding to the image to be processed that need to be decoded.

[0086] In one possible implementation, the number of encoding models is at least one. For the case where there is a first, a penultimate, and a last encoding model among the at least one encoding models, the process of calling the attention model and the convolution model to encode the image to be processed based on global and local information to obtain the target encoded features includes the following steps 2021 to 2023:

[0087] Step 2021: Call the attention model and convolution model in the first encoding model to encode the image to be processed based on global and local information, and obtain the basic features and connection features output by the first encoding model.

[0088] The connection features output by the first encoding model are obtained based on the basic features output by the first encoding model. Since the encoding process of the second encoding model needs to utilize the basic features output by the first encoding model, and the decoding process of the decoding model needs to utilize the connection features output by the first encoding model, after calling the attention model and convolution model in the first encoding model to encode the image to be processed based on global and local information, it is necessary to obtain the basic features and connection features output by the first encoding model.

[0089] As indicated in step 201, there are one or more encoding models, and each encoding model includes one or more attention models and one convolutional model. That is, the first encoding model includes one convolutional model and one or more attention models. The implementation of step 2021 varies depending on the specific attention models within the first encoding model.

[0090] In an exemplary embodiment, the attention model in the first encoding model includes at least one of a first attention model and a second attention model. That is, the first encoding model may include a first attention model, a second attention model, or both. The first and second attention models have different functions. For example, the first and second attention models have the same model structure, but their model parameters differ due to their different functions. In the exemplary embodiment, the function of the first attention model is to obtain basic features by working together with the convolutional model in the first encoding model, and the function of the second attention model is to obtain connection features by encoding the basic features.

[0091] In one possible implementation, see [link to relevant documentation]. Figure 3 In the case where the attention model in the first encoding model includes the first attention model, the implementation process of step 2021 includes the following steps 2021A and 2021B.

[0092] Step 2021A: Call the convolutional model and the first attention model in the first encoding model to encode the image to be processed based on local and global information, and obtain the basic features output by the first encoding model.

[0093] There are three ways to implement this step 2021A, including but not limited to three. The following sections will introduce each of the three implementation methods.

[0094] The first implementation method is based on steps (1) to (4):

[0095] Step (1): Call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoding feature.

[0096] Convolutional models are used to extract features through convolution operations, which focus on local information. This application does not limit the model structure of the convolutional model in the first encoding model. The process of encoding the image to be processed based on local information is the internal processing of the convolutional model. The specific process of encoding the image to be processed based on local information using convolutional models with different model structures may differ. In exemplary embodiments, each encoding model includes a convolutional model, and the model structures of the convolutional models in different encoding models may be the same or different. For example, the model structures of the convolutional models in different encoding models are the same to improve the convergence speed during model training.

[0097] In one possible implementation, the convolutional model in the first encoding model includes a convolutional module and a pooling module. The method for calling the convolutional model in the first encoding model to encode the image to be processed based on local information and obtain the first encoded feature is as follows: the convolutional module is called to process the image to be processed to obtain convolutional features; the pooling module is called to process the convolutional features to obtain the first encoded feature. Exemplarily, the convolutional model in the first encoding model includes one or more convolutional modules and one pooling module. Exemplarily, in the case where the convolutional model includes multiple convolutional modules, the structures of the different convolutional modules may be the same or different, and this application embodiment does not limit this.

[0098] For example, when there are multiple convolutional modules, the convolutional modules are connected in series. The process of calling the convolutional modules to process the image to be processed and obtaining convolutional features is as follows: the image to be processed is input into the first convolutional module to obtain the features output by the first convolutional module; starting from the second convolutional module, the features output by the previous convolutional module are input into the next convolutional module to obtain the features output by the next convolutional module, until the features output by the last convolutional module are obtained, and the features output by the last convolutional module are used as the convolutional features.

[0099] For example, each convolutional module consists of a convolutional layer and an activation layer. The size of the convolutional kernel of the convolutional layer is set empirically or flexibly adjusted according to the actual application scenario. This application embodiment does not limit this. For example, the size of the convolutional kernel of the convolutional layer is 3×3. The activation function used by the activation layer is set empirically or flexibly adjusted according to the actual application scenario. This application embodiment does not limit this. For example, the activation function used by the activation layer is a ReLU (Rectified Linear Unit) function or a Sigmoid function. In the process of calling a convolutional module to process features (or images), the convolutional layer is called first for processing, and then the activation layer is called to process the features output by the convolutional layer.

[0100] Exemplarily, the pooling module consists of a pooling layer. This application embodiment does not limit the type of pooling layer; for example, the pooling layer in the pooling module can be a max pooling layer or an average pooling layer. This application embodiment does not limit the size of the sampling kernel of the pooling layer; for example, the size of the sampling kernel of the pooling layer is 2×2. When the sampling kernel size of the pooling layer is 2×2, the pooling layer can sample the convolutional features to 1 / 4 of their original size. Exemplarily, if the size of the convolutional features obtained by calling the convolution module to process the image to be processed is the same as the size of the image to be processed, then the size of the first encoded feature obtained by calling the pooling module to process the convolutional features is 1 / 4 of the size of the image to be processed.

[0101] For example, the convolutional model in the first encoding model includes two concatenated convolutional modules and a pooling module. Each convolutional module includes a convolutional layer and an activation layer, and the pooling module includes a pooling layer. The convolutional layer and activation layer in the first convolutional module are convolutional layer 1 and activation layer 1, respectively, and the convolutional layer and activation layer in the second convolutional module are convolutional layer 2 and activation layer 2, respectively. The process of calling the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoded feature is as follows: Figure 4 As shown, the image to be processed is input into the convolutional model in the first encoding model, and then processed sequentially through convolutional layer 1, activation layer 1, convolutional layer 2, activation layer 2, and pooling layer to output the first encoded feature.

[0102] Step (2): Call the first attention model to encode the image to be processed based on global information to obtain the second encoded features.

[0103] In the first implementation, the processing branches of the first attention model and the convolutional model are parallel. The process of calling the first attention model to encode the image to be processed based on global information is the internal processing of the first attention model, and the specific processing method is related to the model structure of the first attention model. This application does not limit the model structure of the first attention model, as long as it utilizes an attention mechanism for processing. For example, the first attention model is a Transformer model.

[0104] In an exemplary embodiment, the computational complexity of the first attention model is equivalent to the square of the number of tokens (i.e., the sequence length). Calling the first attention model to encode the image to be processed based on global information means calling the first attention model to encode the reference features of the image to be processed based on global information. The reference features of the image to be processed are obtained based on each image patch of the image to be processed. Compared to directly using each pixel in the image to be processed as a token, using each image patch as a token reduces the number of tokens, thereby reducing computational complexity. Exemplarily, the process of obtaining the reference features of the image to be processed can be regarded as the process of serializing the image to be processed.

[0105] Before invoking the first attention model to encode the reference features of the image to be processed based on global information, it is necessary to obtain the reference features of the image to be processed. The reference features of the image to be processed are the features applicable to the attention model corresponding to the image to be processed. In one possible implementation, the process of obtaining the reference features of the image to be processed includes the following steps 1-1 to 1-3:

[0106] Step 1-1: Obtain the block features of each image block of the image to be processed, and map the block features of each image block to obtain the mapped features of each image block.

[0107] Image blocks of the image to be processed are obtained by segmenting the image. For example, the ViT (Vision Transformer) model is called to segment the image to be processed into individual image blocks. For example, all image blocks of the image to be processed have the same size, denoted as P×P (pixels). The size of the image blocks is set empirically or flexibly adjusted according to the size of the image to be processed and the application scenario. This application embodiment does not limit this; for example, the size of the image block is 2×2 (i.e., P=2). The total number of image blocks of the image to be processed is related to the size of the image to be processed and the size of the image blocks. For example, assuming the size of the image to be processed is H×W and the size of the image blocks is P×P, the total number M of all image blocks is: M=HW / P 2 .

[0108] After obtaining the individual image patches in the image to be processed, reference features of the image are acquired based on these patches to reduce computational complexity. For example, if the size of an image patch is 2×2, the computational complexity can be reduced to 1 / 4 of the computational complexity of directly using each pixel in the image as a token. For example, the process of acquiring reference features of the image to be processed based on its individual image patches can be viewed as a process of reassembling the individual image patches into tokens, or as an image serialization process.

[0109] In the process of obtaining reference features of the image to be processed based on each image block in the image to be processed, it is necessary to first obtain the mapping features of each image block. The mapping features of the image block are obtained by mapping the block features of the image block, which is used to intuitively represent the features of the pixels in the image block.

[0110] In one possible implementation, the block features of any image block in the image to be processed are obtained by concatenating the features of each pixel in the image block sequentially. For example, assuming the number of channels of any image block is C (C is an integer not less than 1), the dimension of the block features of any image block is 1×P. 2 C.

[0111] After obtaining the block features of any image patch, the block features of any image patch are mapped to obtain the mapped features of that image patch. The purpose of mapping the block features of any image patch is to map the block features of any image patch to a specified ensemble space for computation. The specified ensemble space is set according to the application scenario. For example, the specified ensemble space refers to a D-dimensional ensemble space (D is an integer not less than 1).

[0112] In an exemplary embodiment, the block features of any image patch are mapped as follows: the block features of any image patch are multiplied by a transformation feature, and the product is used as the mapping feature of that image patch. The transformation feature is a feature set for multiplying the block features to map them to a specified integration space. Exemplarily, the specified integration space refers to a D-dimensional integration space, where the dimension of the block features of any image patch is 1×(P). 2 Taking C) as an example, the feature is transformed into a dimension (P) 2 The features of C)×D are such that multiplying the block features of any image patch by the transformation features yields a mapping feature of dimension 1×D. Exemplarily, the features mentioned in the embodiments of this application can be in the form of a vector or a matrix, and this embodiment of the application does not limit this.

[0113] Based on the method of obtaining the block features of any image patch, the block features of each image patch can be obtained; based on the method of obtaining the mapping features of any image patch, the mapping features of each image patch can be obtained.

[0114] Step 1-2: Obtain image patch location features.

[0115] In the process of obtaining reference features of the image to be processed based on each image patch in the image to be processed, in addition to obtaining the mapping features of each image patch, it is also necessary to obtain the image patch position features. The image patch position features are determined based on the spatial position of each image patch in the image to be processed and are used to reflect spatial information. For example, the spatial position of any image patch in the image to be processed is represented by the number of that image patch. For example, if any image patch is the h-th image patch (h is an integer not less than 1), then the spatial position of that image patch in the image to be processed is denoted as h.

[0116] In an exemplary embodiment, the image patch location feature E pos Based on the calculation using Formula 1:

[0117]

[0118] Where pos represents the spatial location of the image patch in the image to be processed, pos = 1, 2, 3, ..., M (M is the total number of image patches); i represents a certain dimension of the position feature, i = 1, 2, 3, ..., D (M is the dimension of the specified integration space).

[0119] Steps 1-3: Obtain reference features of the image to be processed based on the mapping features and position features of each image patch.

[0120] The reference features of the image to be processed integrate image patch location information, which is beneficial for preserving the spatial information of the image patches. In an exemplary embodiment, the reference features of the image to be processed are obtained based on the mapping features and image patch location features of each image patch as follows: the concatenated features of the mapping features of each image patch are obtained, and element-wise addition is performed on the concatenated features of the mapping features of each image patch and the image patch location features to obtain the reference features of the image to be processed. For example, the dimension of the concatenated features of the mapping features of each image patch is the same as the dimension of the image patch location features to facilitate element-wise addition. For example, the reference features of the image to be processed are calculated based on Formula 2:

[0121]

[0122] Where z represents the reference feature of the image to be processed; E represents the transformed feature. (j = 1, 2, 3, ..., M) represents the block feature of the j-th image patch; Represents the mapping features of the j-th image patch; E represents the concatenation feature of the mapping features of each image patch; pos This indicates the location features of an image patch.

[0123] After obtaining the reference features of the image to be processed, a first attention model is invoked to encode the reference features of the image to be processed based on global information, resulting in second encoded features. The implementation method of invoking the first attention model to encode the reference features of the image to be processed based on global information is related to the model structure of the first attention model, and this embodiment does not limit it.

[0124] In an exemplary embodiment, the first attention model includes an attention module and a nonlinear processing module. The process of calling the first attention model to encode reference features of the image to be processed based on global information to obtain second encoded features is as follows: the attention module is called to process the reference features to obtain a first intermediate feature; the first intermediate feature and the reference feature are concatenated to obtain the feature to be processed; the nonlinear processing module is called to process the feature to be processed to obtain a second intermediate feature; and the second intermediate feature and the feature to be processed are concatenated to obtain the second encoded feature.

[0125] The attention module is used to process reference features based on an attention mechanism. Exemplarily, the attention module includes a normalization layer and a self-attention layer, such as a multi-head self-attention (MSA) layer. The nonlinear processing module is used to perform nonlinear processing on the features to be processed, increasing the nonlinearity of the processing result. Exemplarily, the nonlinear processing module includes a normalization layer and a nonlinear processing layer, such as a multi-layer perceptron (MLP) layer. The normalization layer is used to standardize the features. This application embodiment does not limit the type of normalization layer; for example, it can be an LN (Layer Normalization) layer or a BN (Batch Normalization) layer. The normalization layer included in the attention module and the normalization layer included in the nonlinear processing module may be the same or different; this application embodiment does not limit this.

[0126] For example, the embodiments of this application do not limit the way two features are joined. For example, two features can be joined horizontally; or two features can be joined vertically; or two features can be added element by element.

[0127] For example, assume that the first attention model includes an attention module and a nonlinear processing module. The attention module includes a normalization layer and an MSA layer, and the nonlinear processing module includes a normalization layer and an MLP layer. The normalization layer in the attention module is denoted as Normalization Layer 1, and the normalization layer in the nonlinear processing module is denoted as Normalization Layer 2. The process of calling the first attention model to encode the reference features of the image to be processed based on global information to obtain the second encoded features is as follows: Figure 5 As shown. The reference features are input into the first attention model, and the reference features are processed sequentially through the normalization layer 1 and the MSA layer to obtain the first intermediate features; the first intermediate features and the reference features are concatenated to obtain the features to be processed; the features to be processed are processed sequentially through the normalization layer 2 and the MLP layer to obtain the second intermediate features; the second intermediate features and the features to be processed are concatenated to obtain the second encoded features.

[0128] In an exemplary embodiment, when the first attention model includes an attention module and a nonlinear processing module, and the attention module includes an LN layer and an MSA layer, and the nonlinear processing module includes an LN layer and an MLP layer, the process of calling the first attention model to encode the reference features of the image to be processed based on global information to obtain the second encoded features is implemented based on Equation 3:

[0129]

[0130] Among them, z l-1 LN(z) represents the features input to the first attention model, i.e., the reference features of the image to be processed; l-1 The reference features are processed by the LN layer in the attention module to obtain the features; MSA(LN(z) l-1 )) represents the first intermediate feature; z ′ l Represents the feature to be processed; LN(z) ′ l The ) represents the feature obtained after the feature to be processed has been processed by the LN layer in the nonlinear processing module; MLP(LN(z) ′ l )) indicates the second intermediate feature; z l This represents the second coding feature.

[0131] It should be noted that the first attention model described above, including an attention module and a nonlinear processing module, is merely an exemplary structure of the first attention model, and the embodiments of this application are not limited thereto. Exemplarily, the first attention model may also include only an attention module. Similarly, the attention module, including a normalization layer and a self-attention layer, and the nonlinear processing layer, including a normalization layer and a nonlinear processing layer, are also exemplary structures of the attention module and the nonlinear processing module. Exemplarily, the attention module may also include only a self-attention layer, and the nonlinear processing module may also include only a nonlinear processing layer.

[0132] It should be noted that the above-described method of calling the first attention model to encode the image to be processed based on global information refers to calling the first attention model to encode the reference features of the image to be processed based on global information, which is merely an exemplary description, and the embodiments of this application are not limited thereto. In an exemplary embodiment, calling the first attention model to encode the image to be processed based on global information can also refer to calling the first attention model to encode the pixel-level features of the image to be processed based on global information. The pixel-level features of the image to be processed are obtained based on the features of each pixel in the image to be processed. In this case, a pixel in the image to be processed is used as a token. Exemplarily, the process of calling the first attention model to encode the pixel-level features of the image to be processed based on global information to obtain the second encoded feature is the same as the process of calling the first attention model to encode the reference features of the image to be processed based on global information to obtain the second encoded feature, and will not be repeated here.

[0133] Step (3): Fuse the first coding feature and the second coding feature to obtain the fused feature.

[0134] The first encoding feature is obtained by calling a convolutional model to encode the image to be processed based on local information, which can focus on local information. The second encoding feature is obtained by calling a first attention model to encode the image to be processed based on global information, which can focus on global information. By fusing the first encoding feature and the second encoding feature, a fused feature can be obtained by comprehensively focusing on global and local information.

[0135] In one possible implementation, the process of fusing the first and second coding features to obtain the fused features is as follows: the second coding feature is converted into a feature to be paralleled with the first coding feature of the same size; the feature to be paralleled is then paralleled with the first coding feature in the channel dimension to obtain the fused features.

[0136] For example, the feature to be paralleled can refer to a feature obtained by converting only the size of the second coded feature; or it can refer to a feature obtained by converting both the size and the number of channels of the second coded feature. This application embodiment does not limit this. This application embodiment uses the example of a feature to be paralleled that is obtained by converting both the size and the number of channels of the second coded feature for illustration.

[0137] For example, the shape of the first encoded feature is [H l W l If the shape of the second encoded feature is [N,D], then first convert the shape of the second encoded feature to [H,C]. l W l [,D], then a 1×1 convolution kernel is used to transform the number of channels, finally obtaining a shape of [H] l W l The first encoded feature and the feature to be parallelized are then concatenated in parallel along the channel dimension to obtain a feature of shape [H]. l W l The fusion features of ,2C].

[0138] It should be noted that the above-described method of fusing the first and second coding features to obtain the fused feature is merely an exemplary description, and the embodiments of this application are not limited thereto. In an exemplary embodiment, the method of fusing the first and second coding features to obtain the fused feature is as follows: the second coding feature is converted into a feature to be paralleled with the same size and number of channels as the first coding feature; the average feature or the sum of the feature to be paralleled and the first coding feature is used as the fused feature.

[0139] Step (4): Based on the fusion features, obtain the basic features output by the first encoding model.

[0140] The fusion feature is a feature obtained by comprehensively considering both global and local information. The basic feature output by the first encoding model based on the fusion feature has high reliability. The basic feature output by the first encoding model is used as the input to the next encoding model. The implementation method for obtaining the basic feature output by the first encoding model based on the fusion feature can be flexibly set as needed, and this application embodiment does not limit it in this way.

[0141] In one possible implementation, the fused features are directly used as the base features of the first encoding model output.

[0142] In another possible implementation, the first encoding model also includes a third attention model. This third attention model is invoked to encode the fused features based on global information, yielding the base features output by the first encoding model. The third attention model refers to the attention model used to encode the features directly upon which the base features are obtained. The principle of invoking the third attention model to encode the fused features based on global information is the same as the principle of invoking the first attention model to encode the image to be processed based on global information, and will not be elaborated further here.

[0143] The second implementation method is based on steps (A) to (C):

[0144] Step (A): Call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoded feature.

[0145] The implementation method of step (A) is the same as step (1) in the first implementation method, and will not be repeated here.

[0146] Step (B): Call the first attention model to encode the first encoding feature based on global information to obtain the third encoding feature.

[0147] In the second implementation, the processing branch of the first attention model is concatenated with the processing branch of the convolutional model, and the processing branch of the first attention model follows the processing branch of the convolutional model. In other words, the processing branch of the first attention model is used to encode the first encoded feature output by the processing branch of the convolutional model.

[0148] For example, the first encoded feature has the same representation as the image to be processed. For instance, both the first encoded feature and the image to be processed are represented using matrices, or both are represented using vectors; or both are represented using images. The principle of calling the first attention model to encode the first encoded feature based on global information to obtain the third encoded feature is the same as the principle of calling the first attention model to encode the image to be processed based on global information to obtain the second encoded feature in step (2) of the first implementation method, and will not be repeated here. It should be noted that calling the first attention model to encode the image to be processed based on global information in step (2) refers to calling the first attention model to encode the reference feature of the image to be processed based on global information. The calling the first attention model to encode the first encoded feature based on global information in step (B) refers to calling the first attention model to encode the reference feature of the first encoded feature based on global information. The principle of obtaining the reference feature of the first encoded feature is the same as the principle of obtaining the reference feature of the image to be processed. It is only necessary to represent the first encoded feature in the form of an image.

[0149] Step (C): Based on the third coding feature, obtain the basic features output by the first coding model.

[0150] The implementation principle of step (C) is the same as that of step (4) in the first implementation method, and will not be repeated here.

[0151] The third implementation method is based on steps (Ⅰ) to (Ⅲ):

[0152] Step (I): Call the first attention model to encode the image to be processed based on global information to obtain the second encoded features.

[0153] The implementation method of step (Ⅰ) is the same as step (2) in the first implementation method, and will not be repeated here.

[0154] Step (II): Call the convolutional model in the first encoding model to encode the second encoding feature based on local information to obtain the fourth encoding feature.

[0155] In the third implementation, the processing branches of the first attention model and the convolutional model are connected in series, with the convolutional model's processing branch following the first attention model's processing branch. In other words, the convolutional model's processing branch is used to encode the second encoded features output by the first attention model's processing branch.

[0156] For example, the second encoded feature has the same representation as the image to be processed. For instance, both the second encoded feature and the image to be processed are represented using matrices, or both are represented using vectors; or both are represented using images. The principle of calling the convolutional model in the first encoding model to encode the second encoded feature based on local information to obtain the fourth encoded feature is the same as the principle of calling the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoded feature in step (1) of the first implementation method, and will not be repeated here.

[0157] Step (III): Based on the fourth coding feature, obtain the basic features output by the first coding model.

[0158] The implementation principle of step (Ⅲ) is the same as that of step (4) in the first implementation method, and will not be repeated here.

[0159] Step 2021B: Based on the basic features output by the first encoding model, obtain the connection features output by the first encoding model.

[0160] The connection features output by the first encoding model are used to provide data support for executing step 203. That is, the connection features output by the first encoding model will be utilized during the execution of step 203. When the attention model in the first encoding model includes a first attention model, the method of obtaining the connection features output by the first encoding model based on the basic features output by the first encoding model is set according to experience or flexibly adjusted according to the application scenario. This application embodiment does not limit this.

[0161] In an exemplary embodiment, the method for obtaining the connection features output by the first encoding model based on the basic features output by the first encoding model is to directly use the basic features output by the first encoding model as the connection features output by the first encoding model.

[0162] In an exemplary embodiment, the attention model in the first encoding model further includes a second attention model. The function of the second attention model is to obtain connection features by encoding basic features. In this case, the connection features output by the first encoding model are obtained based on the basic features output by the first encoding model by calling the second attention model to encode the basic features output by the first encoding model based on global information, thereby obtaining the connection features output by the first encoding model.

[0163] For example, the model structure of the second attention model is the same as that of the first attention model. For example, the basic features output by the first encoding model are in the same form as the representation of the image to be processed. For example, both the basic features output by the first encoding model and the image to be processed are represented using matrices, or both the basic features output by the first encoding model and the image to be processed are represented using vectors; or both the basic features output by the first encoding model and the image to be processed are represented using images. The principle of calling the second attention model to encode the basic features output by the first encoding model based on global information to obtain the connection features output by the first encoding model is the same as the principle of calling the first attention model to encode the image to be processed based on global information to obtain the second encoded features in step (2) of the first implementation method, and will not be repeated here.

[0164] In one possible implementation, see [link to relevant documentation]. Figure 3 In the case where the attention model in the first encoding model includes the second attention model, the implementation process of step 2021 includes the following steps 2021a and 2021b.

[0165] Step 2021a: Call the convolutional model in the first encoding model to encode the image to be processed based on local information, and obtain the basic features output by the first encoding model.

[0166] In cases where the attention model in the first encoding model includes a second attention model, the convolutional model in the first encoding model is called to encode the image to be processed based on local information. The method of obtaining the basic features output by the first encoding model is set according to experience or flexibly adjusted according to the application scenario. This application embodiment does not limit this.

[0167] In an exemplary embodiment, the method of calling the convolutional model in the first encoding model to encode the image to be processed based on local information and obtaining the basic features output by the first encoding model is as follows: the features obtained by calling the convolutional model in the first encoding model to encode the image to be processed based on local information are directly used as the basic features output by the first encoding model.

[0168] In an exemplary embodiment, the attention model in the first encoding model further includes a first attention model. In this case, the method of calling the convolutional model in the first encoding model to encode the image to be processed based on local information and obtaining the basic features output by the first encoding model is as follows: the convolutional model and the first attention model in the first encoding model are called to encode the image to be processed based on local and global information, thereby obtaining the basic features output by the first encoding model. The implementation of this process is described in step 2021A, and will not be repeated here.

[0169] Step 2021b: Call the second attention model to encode the basic features output by the first encoding model based on global information, and obtain the connection features output by the first encoding model.

[0170] Since the function of the second attention model is to obtain connection features by encoding basic features, when the attention model in the first encoding model includes the second attention model, the connection features output by the first encoding model are obtained by calling the second attention model to encode the basic features output by the first encoding model based on global information.

[0171] For example, the model structure of the second attention model is the same as that of the first attention model. For example, the basic features output by the first encoding model are in the same form as the representation of the image to be processed. For example, both the basic features output by the first encoding model and the image to be processed are represented using matrices, or both the basic features output by the first encoding model and the image to be processed are represented using vectors; or both the basic features output by the first encoding model and the image to be processed are represented using images. The principle of calling the second attention model to encode the basic features output by the first encoding model based on global information to obtain the connection features output by the first encoding model is the same as the principle of calling the first attention model to encode the image to be processed based on global information to obtain the second encoded features in step (2) of the first implementation method, and will not be repeated here.

[0172] In one possible implementation, see [link to relevant documentation]. Figure 3 In the case where the attention model in the first encoding model includes a first attention model and a second attention model, the implementation process of step 2021 includes the following steps 20211 and 20212.

[0173] Step 20211: Call the convolutional model and the first attention model in the first encoding model to encode the image to be processed based on local and global information, and obtain the basic features output by the first encoding model.

[0174] The implementation method for step 20211 is the same as that for step 2021A, and will not be repeated here.

[0175] Step 20212: Call the second attention model to encode the basic features output by the first encoding model based on global information, and obtain the connection features output by the first encoding model.

[0176] The implementation method for step 20212 is described in step 2021b, and will not be repeated here.

[0177] Step 2022: Starting from the second encoding model, call the attention model and convolution model in the next encoding model to encode the basic features output by the previous encoding model based on global and local information, so as to obtain the basic features and connection features output by the next encoding model, until the basic features and connection features output by the penultimate encoding model are obtained.

[0178] The connection features output by each encoding model from the first to the penultimate encoding model are used to provide data support for the step of calling the decoding model to decode the target encoded features and obtain the target image features (i.e., step 203). In other words, the connection features output by each encoding model from the first to the penultimate encoding model are utilized during the execution of step 203.

[0179] The basic features output by the previous encoding model have the same representation as the image to be processed. The attention and convolutional models in the next encoding model are used to encode the basic features output by the previous encoding model based on global and local information, obtaining the basic features and connection features output by the next encoding model. This principle is the same as that in step 2021, where the attention and convolutional models in the first encoding model are used to encode the image to be processed based on global and local information, obtaining the basic features and connection features output by the first encoding model. Therefore, it will not be repeated here. It should be noted that each encoding model includes one convolutional model and at least one attention model. The attention models in different encoding models may be the same or different, and this embodiment does not limit this.

[0180] By continuously executing the operation of calling the attention model and convolution model in the next encoding model to encode the basic features output by the previous encoding model based on global and local information, the basic features and connection features output by the next encoding model can be obtained. The basic features and connection features output by the penultimate encoding model are used to provide data support for the execution of step 2023.

[0181] For example, the size of the basic feature output by the next encoding model is smaller than the size of the basic feature output by the previous encoding model. That is, during the encoding process, basic features with decreasing resolution are gradually acquired. For example, the size of the basic feature output by any encoding model is the same as the size of the connection feature output by that encoding model.

[0182] Step 2023: Call the attention model and convolution model in the last encoding model to encode the basic features output by the penultimate encoding model based on global and local information, and obtain the connection features output by the last encoding model. Use the connection features output by the last encoding model as the target encoding features.

[0183] After obtaining the basic features output by the penultimate encoding model based on step 2022, the attention model and convolutional model in the last encoding model are invoked to encode the basic features output by the penultimate encoding model based on global and local information, obtaining the connection features output by the last encoding model. These connection features are then used as the target encoding features. The implementation principle of step 2023 is the same as that of step 2021. It should be noted that since it is not necessary to use the basic features output by the last encoding model as the input to the next encoding model, only the connection features output by the last encoding model need to be obtained during the execution of step 2023.

[0184] It should be noted that steps 2021 to 2023 above describe the operations required to obtain the target encoded features when at least one encoded model contains a first encoded model, a penultimate encoded model, and a last encoded model (i.e., the number of at least one encoded model is three or more). The embodiments of this application are not limited to this; the number of at least one encoded model may also be one or two.

[0185] In an exemplary embodiment, for the case where the number of at least one encoding model is one, the method of calling the attention model and the convolution model to encode the image to be processed based on global and local information to obtain the target encoding features is as follows: the attention model and the convolution model in the one encoding model are called to encode the image to be processed based on global and local information to obtain the connection features output by the encoding model, and the connection features output by the encoding model are used as the target encoding features.

[0186] In an exemplary embodiment, for cases where there are at least two encoding models, the attention model and convolutional model are invoked to encode the image to be processed based on global and local information to obtain the target encoded features. This is done as follows: the attention model and convolutional model in the first encoding model are invoked to encode the image to be processed based on global and local information, obtaining the basic features and connectivity features output by the first encoding model; the attention model and convolutional model in the second encoding model are invoked to encode the basic features output by the first encoding model based on global and local information, obtaining the connectivity features output by the second encoding model, and these connectivity features are used as the target encoded features. The connectivity features output by the first encoding model are used to provide data support for executing step 203.

[0187] In step 203, the decoding model is invoked to decode the target encoded features to obtain the target image features.

[0188] Target image features are the features directly used to obtain the segmentation results of the image to be processed. They are the features that best represent the essence of the image to be processed, obtained during the process of calling the image processing model to process the image. Target image features are obtained based on target decoding features, and are features obtained by comprehensively considering both global and local information. Target image features can more comprehensively represent the image to be processed.

[0189] In an exemplary embodiment, the number of decoding models is the same as the number of encoding models, both being at least one. The at least one decoding model is cascaded, meaning that the decoding of the target encoded features is achieved using at least one cascaded decoding model. The at least one decoding model and at least one encoding model form a U-shaped structure. Through skip links between corresponding layers (i.e., the decoding process of the decoding model utilizes the connection features output by the encoding model), feature aggregation can be achieved at different resolution levels.

[0190] For example, if at least one encoding model contains a first encoding model, a penultimate encoding model, and a last encoding model, then at least one decoding model also contains a first decoding model, a penultimate decoding model, and a last decoding model. In this case, the process of calling the decoding model to decode the target encoded features and obtain the target image features includes the following steps 2031 to 2033:

[0191] Step 2031: Call the first decoding model to decode the target encoded features and obtain the decoded features output by the first decoding model.

[0192] The process of calling the first decoding model to decode the target encoded features is an internal processing procedure of the first decoding model, which is related to the model structure of the first decoding model. Under different structures, the way the first decoding model is called to decode the target encoded features is different.

[0193] In an exemplary embodiment, the first decoding model includes an upsampling layer, a convolutional layer, and an activation layer. This application embodiment does not limit the type of the upsampling layer, the size of the convolutional kernel of the convolutional layer, or the type of activation function used in the activation layer of the first decoding model. For example, the upsampling layer in the first decoding model is a 2×2 bilinear interpolation layer, the convolutional kernel of the first decoding model has a size of 3×3, and the activation function sampled by the activation layer in the first decoding model is the ReLU function.

[0194] The target encoded features are input into the first decoding model, and then processed sequentially through an upsampling layer, a convolutional layer, and an activation layer to obtain the decoded features output by the first decoding model. In an exemplary embodiment, the size of the decoded features output by the first decoding model is larger than the size of the target encoded features input into the first decoding model, so as to obtain features with higher resolution through decoding.

[0195] Step 2032: Starting from the second decoding model, call the next decoding model to decode the concatenation feature corresponding to the next decoding model to obtain the decoding feature output by the next decoding model, until the decoding feature output by the penultimate decoding model is obtained; the concatenation feature corresponding to the next decoding model is the concatenation feature of the decoding feature output by the previous decoding model and the connection feature output by the encoding model corresponding to the next decoding model.

[0196] After obtaining the decoded features output by the previous decoding model, the decoded features output by the previous decoding model and the connection features output by the encoding model corresponding to the next decoding model are concatenated to obtain the concatenated features corresponding to the next decoding model. The encoding model corresponding to the next decoding model is the encoding model whose position in at least one encoding model is at a distance from the position of the last encoding model to a reference position distance. The reference position distance is the position distance between the position of the next decoding model in at least one decoding model and the position of the first decoding model. That is, assuming the next decoding model is the Kth decoding model, where K is an integer not less than 2 and not greater than (N-1), and N is the total number of decoding models (also the total number of encoding models), the encoding model whose position distance from the last encoding model to a reference position distance is the (N-K+1)th encoding model. In other words, the encoding model corresponding to the Kth decoding model is the (N-K+1)th encoding model. For example, assuming K=2 and N=5, the encoding model corresponding to the 2nd decoding model is the 4th encoding model.

[0197] It should be noted that since K is an integer not less than 2 and not greater than (N-1), the decoding model corresponding to the next encoding model is one of the decoding models from the (N-1)th to the second decoding model. As described in steps 2021 to 2023 above, the connection features output by each decoding model from the second to the penultimate decoding model (i.e., the (N-1)th decoding model) have already been obtained during the encoding process. Therefore, the connection features output by the encoding model corresponding to the next decoding model can be directly obtained. Furthermore, by concatenating the decoding features output by the previous decoding model and the connection features output by the encoding model corresponding to the next decoding model, the concatenated features corresponding to the next decoding model are obtained.

[0198] In an exemplary embodiment, the decoding features output by the previous decoding model and the connection features output by the encoding model corresponding to the next decoding model have the same size. The method of concatenating the decoding features output by the previous decoding model and the connection features output by the encoding model corresponding to the next decoding model is to connect the decoding features output by the previous decoding model and the connection features output by the encoding model corresponding to the next decoding model in parallel along the channel dimension.

[0199] The principle of calling the next decoding model to decode the concatenated features corresponding to the next decoding model and obtaining the decoded features output by the next decoding model is the same as the principle of calling the first decoding model to decode the target encoded features and obtaining the decoded features output by the first decoding model in step 2031, and will not be repeated here. By continuously executing the operation of calling the next decoding model to decode the concatenated features corresponding to the next decoding model and obtaining the decoded features output by the next decoding model, the decoded features output by the penultimate decoding model can be obtained, and then step 2033 is executed.

[0200] Step 2033: Call the last decoding model to decode the decoding features output by the second-to-last decoding model to obtain the target image features.

[0201] After obtaining the decoded features output by the penultimate decoding model, the last decoding model is called to decode the decoded features output by the penultimate decoding model to obtain the target image features. The principle of this process is the same as step 2031, and will not be repeated here.

[0202] For example, the number of at least one encoding model and at least one decoding model are both three; the attention models in the first and second encoding models both include a first attention model; the attention models in the third encoding model include both a first attention model and a second attention model; and the processing branches of the first attention model in the encoding models are parallel to the processing branches of the convolutional models. In this case, the process of acquiring the target image features is as follows: Figure 6 As shown.

[0203] The encoded features obtained by encoding the image 600 to be processed based on local information by calling the convolutional model 601 in the first encoding model and the encoded features obtained by encoding the image 600 to be processed based on global information by calling the first attention model 602 in the first encoding model are fused to obtain the basic features 603 output by the first encoding model. The basic features 603 output by the first encoding model are used as the connection features 604 output by the first encoding model.

[0204] The encoded features obtained by encoding the basic features 603 output by the first encoding model based on local information by the convolutional model 605 in the second encoding model and the encoded features obtained by encoding the basic features 603 output by the first encoding model based on global information by the first attention model 606 in the second encoding model are fused together to obtain the basic features 607 output by the second encoding model. The basic features 607 output by the second encoding model are then used as the connection features 608 output by the second encoding model.

[0205] The encoded features obtained by encoding the basic features 607 output by the second encoding model based on local information using the convolutional model 609 in the third encoding model are fused with the encoded features obtained by encoding the basic features 607 output by the second encoding model based on global information using the first attention model 610 in the third encoding model to obtain the basic features 611 output by the third encoding model. The basic features 611 output by the third encoding model are then encoded based on global information using the second attention model 612 in the third encoding model to obtain the connection features 613 output by the third encoding model. The connection features 613 output by the third encoding model are then used as the target encoded features.

[0206] The first decoding model (not shown in the figure) is used to decode the connection feature 613 (i.e., the target encoded feature) output by the third encoding model, resulting in the decoded feature 614 output by the first decoding model. The second decoding model (not shown in the figure) is then used to decode the concatenated feature of the decoded feature 614 output by the first decoding model and the connection feature 608 output by the second encoding model, resulting in the decoded feature 615 output by the second decoding model. The third decoding model (not shown in the figure) is then used to decode the concatenated feature of the decoded feature 615 output by the second decoding model and the connection feature 604 output by the first encoding model, resulting in the decoded feature 616 output by the third decoding model. This decoded feature 616 is then used as the target image feature.

[0207] It should be noted that steps 2031 to 2033 above describe the operations required to decode the target encoded features and obtain the target image features when at least one encoding model contains a first encoding model, a second-to-last encoding model, and a last encoding model (i.e., at least three or more decoding models). The embodiments of this application are not limited to this; the number of at least one encoding model may also be one or two.

[0208] In an exemplary embodiment, if the number of at least one encoding model is one, the number of at least one decoding model is also one. The process of calling the decoding model to decode the target encoded features and obtain the target image features is as follows: calling the decoding model to decode the target encoded features and obtain the target image features.

[0209] In an exemplary embodiment, for cases where there are at least two encoding models, there are also at least two decoding models. The process of calling the decoding model to decode the target encoded features and obtain the target image features is as follows: calling the first decoding model to decode the target encoded features and obtaining the decoded features output by the first decoding model; calling the second decoding model to decode the decoded features output by the first decoding model and obtaining the decoded features output by the second decoding model; and using the decoded features output by the second decoding model as the target image features.

[0210] In an exemplary embodiment, the size of the target image features is the same as the size of the image to be processed, so that a segmentation result with the same size as the image to be processed can be obtained based on the target image features.

[0211] In step 204, the segmentation result of the image to be processed is obtained based on the features of the target image.

[0212] After acquiring the features of the target image, the segmentation result of the image to be processed is obtained based on these features. The image to be processed includes a sub-image of the reference object and the segmentation result of the image to be processed. The segmentation result of the image to be processed is used to indicate the region where the sub-image of the reference object is located in the image to be processed. In other words, based on the segmentation result of the image to be processed, it is possible to determine which regions in the image to be processed are the regions where the sub-image of the reference object is located.

[0213] This application does not limit the form of the image segmentation result to be processed. For example, the segmentation result of the image to be processed is in the form of a two-channel probability map. One channel probability map is used to display the probability that each pixel belongs to a sub-image of the reference object, and the other channel probability map is used to display the probability that each pixel does not belong to a sub-image of the reference object. For example, the segmentation result of the image to be processed is in the form of numerical pairs, with one numerical pair corresponding to one pixel. The numerical pair corresponding to one pixel includes the position coordinates of the pixel, the probability that the pixel belongs to a sub-image of the reference object, and the probability that the pixel does not belong to a sub-image of the reference object.

[0214] In one possible implementation, the process of obtaining the segmentation result of the image to be processed based on the target image features is as follows: A target convolutional layer is invoked to convolve the target image features to obtain the segmentation result of the image to be processed. The target convolutional layer is used to convert the target image features into a segmentation result of a specified form. In an exemplary embodiment, the target convolutional layer is a convolutional layer with a kernel size of 1×1. The target convolutional layer can refer to a convolutional layer in an image processing model, or it can refer to a single convolutional layer; this embodiment does not limit this.

[0215] In one possible implementation, after obtaining the segmentation result of the image to be processed, the method further includes: transforming the segmentation result of the image to be processed to obtain an image processing result. The segmentation result of the image to be processed is used to indicate the probability that each pixel belongs to a sub-image of a reference object and the probability that it does not belong to a sub-image of the reference object. The process of transforming the segmentation result of the image to be processed is the process of determining whether each pixel belongs to a sub-image of the reference object based on the probability that each pixel belongs to a sub-image of the reference object and the probability that it does not belong to a sub-image of the reference object indicated by the segmentation result of the image to be processed. The result used to indicate whether each pixel belongs to a sub-image of the reference object is used as the image processing result. Exemplarily, the image processing result can be in the form of an image or a pair of numerical values. Exemplarily, if the image processing result is in the form of a pair of numerical values, the numerical values ​​can be visualized as an image for easy visual observation.

[0216] The technical solution provided in this application first calls an attention model and a convolution model to obtain target encoding features based on global and local information, and then obtains target image features based on the target encoding features. Since the target encoding features are obtained based on both global and local information, it ensures that the target image features obtained from the target encoding features are features obtained by comprehensively considering both global and local information. This results in richer information focus, leading to higher reliability of the target image features and thus improving the accuracy of the obtained segmentation results.

[0217] Based on the above Figure 1 The implementation environment shown in this application embodiment provides a training method for an image processing model. This training method is executed by a computer device, which can be a server 12 or a terminal 11; this application embodiment does not limit the specific computer device used. Figure 7 As shown, the training method for the image processing model provided in this application embodiment includes the following steps 701 to 705.

[0218] In step 701, sample images, label information of sample images, and an initial image processing model are obtained. The initial image processing model includes an initial encoding model and an initial decoding model. The initial encoding model includes an initial attention model and an initial convolution model.

[0219] Sample images refer to the images required to train the initial image processing model. For example, sample images include sub-images of the reference object. For example, sample images and... Figure 2 In the illustrated embodiments, the images to be processed are of the same type to ensure the processing effect of the trained image processing model on the images to be processed. It should be noted that the sample images mentioned in the embodiments of this application refer to the sample images used to train the initial image processing model once. The number of sample images can be one or more, and this embodiment of the application does not limit this. For example, the number of sample images is multiple to ensure the model training effect.

[0220] In an exemplary embodiment, the computer device acquires sample images by extracting sample images from an image library.

[0221] In an exemplary embodiment, the computer device acquires sample images by using training images from a publicly available dataset as sample images. For example, the computer device uses training images from the MoNuSeg (Multi-organ Nucleus Segmentation) dataset as sample images. The MoNuSeg dataset was obtained by accurately annotating histopathological images of different tumor organs from multiple patients in multiple hospitals, and consists of HE-stained images magnified 40 times from the TCGA (Cancer Genome Atlas) archive. The MoNuSeg dataset contains 30 training images and 14 test images, each image being 1000×1000 pixels in size. The training data covers seven different organs, including the breast, liver, kidney, prostate, bladder, colon, and stomach, and contains approximately 22,000 complete nuclear boundary annotations. The test data covers seven different organs, including the kidney, lung, colon, breast, bladder, prostate, and brain, and contains approximately 7,000 complete nuclear boundary annotations.

[0222] In an exemplary embodiment, the computer device acquires sample images by processing the original images acquired by an image acquisition device (e.g., a microscope, an imported scanner, a domestic scanner, etc.) to obtain sample images. In this case, the original images may be extracted from an image library or uploaded manually, etc., and this embodiment does not limit this. Methods of processing the original images include, but are not limited to, cropping, data augmentation, etc., and this embodiment does not limit this. Exemplarily, the method of processing the original images is related to the computing power of the computer device and the input size required by the segmentation model.

[0223] The label information of the sample images is used to provide supervision information for the model training process. The label information of the sample images provides information on whether each pixel in the sample image belongs to a sub-image of the reference object. In an exemplary embodiment, the information on whether each pixel in the sample image belongs to a sub-image of the reference object can be obtained through manual annotation; the label information of the sample images refers to the pixel-level labels of the sample images.

[0224] In an exemplary embodiment, the reference objects in the sample image are small in size, numerous, and densely arranged, making it difficult to directly obtain pixel-level labels manually. In this case, the labeling information of the sample image includes at least one of point labels, a first auxiliary label, and a second auxiliary label. The training process in this case is a weakly supervised training process. Both the first and second auxiliary labels are obtained based on the point labels. The point labels are determined based on reference points within the region where the sub-image of the reference object is located in the sample image. The first and second auxiliary labels can provide more supervision information than the point labels. Next, the processes for obtaining the point labels, the first auxiliary labels, and the second auxiliary labels will be described respectively.

[0225] 1. The process of obtaining tags

[0226] The point label is determined based on a reference point within the region where the sub-image of a reference object is located in the sample image. Since the reference point is located within the region where the sub-image is located in the sample image, it can provide partial location information of the sub-image. For example, a sample image includes one or more sub-images of reference objects, and each sub-image of a reference object has a reference point within the region where it is located in the sample image, which is used to roughly represent the location of the sub-image of the reference object. Which point within the region where a sub-image of a reference object is located in the sample image is chosen as the reference point is set empirically, manually specified, or flexibly adjusted according to the actual application scenario; this application embodiment does not limit this. For example, this application embodiment does not limit the size and shape of the reference point; for example, the reference point is a 1×1 (pixel) square point; or, the reference point is a 2×1 (pixel) rectangular point.

[0227] The point labels include sub-labels indicating that pixels located at a reference point belong to a sub-image of the reference object, and sub-labels indicating that pixels not located at a reference point do not belong to a sub-image of the reference object. This application does not limit the form of the point labels corresponding to the sample image. For example, the point labels corresponding to the sample image are in the form of numerical pairs, consisting of pixel position coordinates and the corresponding label value. For example, the point labels corresponding to the sample image are in the form of an image, which has the same size as the sample image. In this image, pixels located at the reference point (i.e., pixels belonging to a sub-image of the reference object) and pixels not located at the reference point (i.e., pixels not belonging to a sub-image of the reference object) are presented using different presentation methods.

[0228] In one possible implementation, the point labels corresponding to the sample images are generated by a computer device based on reference points manually marked in the sample images. In another possible implementation, the point labels corresponding to the sample images are stored in correspondence with the sample images, so that the point labels corresponding to the sample images can be extracted simultaneously with the extraction of the sample images.

[0229] In one possible implementation, the sample image has a reference object label. That is, the sample image and the reference object label are stored correspondingly, and the reference object label can be extracted simultaneously with the sample image. The reference object label indicates the region where a sub-image of a reference object is located in the sample image. Exemplarily, the reference object label is obtained based on the boundaries of the sub-images of the reference object manually marked in the sample image. The reference object label can be in the form of a numerical pair or an image; this embodiment does not limit the specific form. Exemplarily, when the sample image has a reference object label, the process of obtaining the point label corresponding to the sample image includes the following steps 7011 to 7014:

[0230] Step 7011: Based on the reference object label, determine the region where the sub-image is located in the sample image.

[0231] Since the reference object label is used to indicate the region where the reference object's sub-image is located in the sample image, the region where the reference object's sub-image is located in the sample image can be determined based on the reference object label.

[0232] Step 7012: Determine the region center of the area where the sub-image is located in the sample image.

[0233] After determining the region where the sub-image of the reference object is located in the sample image, the region center of that region is determined. Exemplarily, the region center of the region where the sub-image of the reference object is located in the sample image refers to the centroid of that region. In an exemplary embodiment, when the sample image includes multiple sub-images of reference objects, the regions where the sub-images of different reference objects are located in the sample image are different, and the region centers of the different regions are also different.

[0234] Step 7013: Determine the reference point based on the region center.

[0235] The location of the region center can represent the center location of the sub-image of the reference object. After determining the region center, a reference point is determined based on the region center. In the exemplary embodiment, the region center is directly used as the reference point within the region where the sub-image of the reference object is located in the sample image. This method of determining the reference point is highly efficient.

[0236] In an exemplary embodiment, the center of the region is expanded to obtain a reference point. For example, the region center is expanded by 3 pixels; or, the region center is expanded by 5 pixels, etc. The reference point obtained by expanding the region center is clearly visible.

[0237] Step 7014: Determine the point labels corresponding to the sample image based on the reference point.

[0238] For example, when the point labels corresponding to the sample image are in the form of an image, the method for determining the point labels corresponding to the sample image based on the reference point is as follows: different presentation methods are used to present the pixels located at the reference point and other pixels to obtain the point labels in image form corresponding to the sample image. The presentation method is set based on experience or flexibly adjusted according to the application scenario; this embodiment does not limit this. For example, the presentation method is to present color, such as using white to present the pixels located at the reference point and using black to present other pixels. For example, the presentation method is to present stripes, such as using horizontal stripes to present the pixels located at the reference point and using vertical stripes to present other pixels.

[0239] For example, when the point labels corresponding to the sample image are numerical pairs, the method for determining the point labels corresponding to the sample image based on the reference point is as follows: assign a first label value to pixels whose position coordinates are located on the reference point, assign a second label value to pixels whose position coordinates are not located on the reference point, and use all pixel position coordinate-label value pairs as the point labels in the form of numerical pairs corresponding to the sample image. The first label value and the second label value are set based on experience or flexibly adjusted according to the application scenario, and this application embodiment does not limit this. For example, the first label value is 1 and the second label value is 0; or, the first label value is 0 and the second label value is 1.

[0240] The point labels corresponding to a sample image can provide partial location information of a sub-image of a reference object. For example, the point labels corresponding to a sample image are called weak labels of the sample image, and the process of training a model based on point labels refers to the process of training a model based on weakly supervised learning.

[0241] For example, the reference object label in image form corresponding to the sample image is as follows: Figure 8 As shown in (a) in the figure, in Figure 8 In (a), white is used to represent pixels within the region of the sub-image of the reference object, and black is used to represent pixels outside the region of the sub-image of the reference object. Based on Figure 8 The reference object label shown in (a) can be used to obtain, for example... Figure 8 The dot labels are shown in image form (b) in the diagram. Figure 8 In (b), white is used to represent pixels located on the reference point, and black is used to represent pixels not located on the reference point.

[0242] 2. The process of obtaining the first auxiliary tag

[0243] The first auxiliary label is obtained based on the dot label. In one possible implementation, the process of obtaining the first auxiliary label based on the dot label includes the following steps 701A to 701C:

[0244] Step 701A: Based on the point labels, determine the reference points within the region where the sub-image is located in the sample image.

[0245] Since the point labels corresponding to the sample image are determined based on reference points within the region where the sub-image of the reference object is located in the sample image, reference points within the region where the sub-image is located in the sample image can be determined based on the point labels. For example, when the point labels are in the form of an image, the image uses white to represent pixels located at the reference points and black to represent other pixels. In this case, the reference points can be determined based on the white areas presented in the image.

[0246] Step 701B: Generate the Thiessen polygon corresponding to the reference point in the sample image.

[0247] Thiessen polygons are continuous polygons formed by the perpendicular bisectors of the lines connecting two adjacent points. After determining a reference point, adjacent reference points can be connected, and then the Thiessen polygon corresponding to the reference point can be obtained based on the perpendicular bisectors of the lines connecting adjacent reference points. Thiessen polygons divide the plane containing the sample image into ideal polygonal blocks. In this embodiment, pixels located on the Thiessen polygons are considered pixels of sub-images that do not belong to the reference object. That is, Thiessen polygons can provide reliable negative samples, which can help prevent overlap in highly clustered sub-images of the reference object during segmentation.

[0248] Step 701C: Based on the reference point and the Thiessen polygon, obtain a first auxiliary label. The first auxiliary label includes a sub-label for indicating that a pixel located on the reference point belongs to the sub-image, a sub-label for indicating that a pixel located on the Thiessen polygon does not belong to the sub-image, and a sub-label for indicating that a pixel located outside the reference point and the Thiessen polygon belongs to an uncertain pixel.

[0249] After determining the reference point and generating the corresponding Thiessen polygon, a first auxiliary label is obtained based on the reference point and the Thiessen polygon. The first auxiliary label determines which pixels in the sample image belong to the sub-image of the reference object, which pixels do not belong to the sub-image of the reference object, and which pixels are of uncertain significance, thus providing strong supervision information for the model training process.

[0250] The method for obtaining the first auxiliary label based on the reference point and the Thiessen polygon is related to the form of the first auxiliary label. For example, if the first auxiliary label is an image, then an image of the same size as the sample image, showing the pixels at the reference point, the pixels at the Thiessen polygon, and other pixels using different rendering methods, will be used as the first auxiliary label. For example, an image-based first auxiliary label might look like this: Figure 9 As shown, in Figure 9 In the image shown, pixels located at the reference point are represented in white, pixels located on the Thiessen polygon are represented in gray, and pixels located outside the reference point and the Thiessen polygon are represented in black. In this case, the sub-label used to indicate that a pixel located at the reference point belongs to a sub-image is in the form of an image presented in white; the sub-label used to indicate that a pixel located on the Thiessen polygon does not belong to a sub-image is in the form of an image presented in gray; and the sub-label used to indicate that a pixel located at the reference point and outside the Thiessen polygon belongs to an uncertain pixel is in the form of an image presented in black.

[0251] For example, if the first auxiliary label is in the form of a numerical pair, then based on the reference point and the Thiessen polygon, the first auxiliary label is obtained as follows: a first value is assigned to the pixel located at the reference point, a second value is assigned to the pixel located at the Thiessen polygon, and a third value is assigned to the pixel located outside the reference point and the Thiessen polygon. The position coordinate-label value pair of each pixel is used as the first auxiliary label in the form of a numerical pair. In this case, the sub-label used to indicate that the pixel located at the reference point belongs to the sub-image is in the form of a numerical pair including the label value of the first value; the sub-label used to indicate that the pixel located at the Thiessen polygon does not belong to the sub-image is in the form of a numerical pair including the label value of the second value; and the sub-label used to indicate that the pixel located at the reference point and outside the Thiessen polygon belongs to an uncertain pixel is in the form of a numerical pair including the label value of the third value. The first value, the second value, and the third value are set based on experience or flexibly adjusted according to the application scenario, and this application embodiment does not limit this. For example, the first value is 1, the second value is 0, and the third value is -1.

[0252] In an exemplary embodiment, the first auxiliary label obtained based on the reference point and the Thiessen polygon can also be referred to as a point-edge label.

[0253] 3. The process of obtaining the second auxiliary tag

[0254] The second auxiliary label is obtained based on the dot label. In one possible implementation, the process of obtaining the second auxiliary label based on the dot label includes the following steps 701a to 701d:

[0255] Step 701a: Based on the point labels, determine the reference points within the region where the sub-image is located in the sample image.

[0256] The implementation method of step 701a is the same as that of step 701A above, and will not be repeated here.

[0257] Step 701b: Based on the reference point, obtain the reference features corresponding to each pixel in the sample image.

[0258] The reference features corresponding to a pixel are the features used for clustering pixels. These reference features are determined based on and related to a reference point. The principle for obtaining the reference features corresponding to each pixel in a sample image is the same. Taking any pixel (called the first pixel) as an example, we will introduce how to obtain the reference features corresponding to the first pixel based on the reference point.

[0259] In one possible implementation, the reference feature corresponding to the first pixel is obtained based on the reference point as follows: the distance between the first pixel and the target reference point is used as the distance feature corresponding to the first pixel, and the target reference point is the reference point closest to the first pixel; the reference feature corresponding to the first pixel is obtained based on the distance feature corresponding to the first pixel and the color feature of the first pixel.

[0260] By calculating the distances between the first pixel and each reference point, the target reference point closest to the first pixel can be determined. This application does not limit the method of calculating the distance between two points; exemplarily, the Euclidean distance between the two points is calculated. After determining the target reference point, the distance between the first pixel and the target reference point is used as the distance feature corresponding to the first pixel.

[0261] For example, the sample image is an image stained with a dye. Different pixels in the sample image may be stained with different colors. For instance, in the case where the reference object is a cell nucleus and the dye is HE dye, pixels belonging to the cell nucleus sub-image are stained blue, while pixels belonging to the extracellular matrix and cytoplasm sub-images are stained pink. Therefore, in obtaining the reference features corresponding to the first pixel, in addition to considering distance features, the color features of the first pixel are also considered. For example, the color features of the first pixel refer to the color values ​​that the first pixel has under each color component of the first color space. For example, if the first color space is the RGB color space, the color features of the first pixel can be represented as (r i ,g i ,b i ).

[0262] The reference feature corresponding to the first pixel is obtained based on the distance feature corresponding to the first pixel and the color feature of the first pixel. For example, the first pixel x i The corresponding distance feature is denoted as d. i , set the first pixel x i The color characteristics are denoted as (r) i ,g i ,b i If the first pixel is x, then... i The corresponding reference feature can be represented as f xi =(d i ,r i ,g i ,b i ).

[0263] The color difference between pixels in a sub-image belonging to the reference object and pixels in sub-images not belonging to the reference object is significant. Therefore, when obtaining the reference features corresponding to pixels, the color features of the pixels are considered to facilitate the classification of pixels of different categories into different clusters to a certain extent. In addition to considering color features, distance features are also considered to avoid the adverse effects of uneven coloring on the clustering results. Given similar colors, pixels in sub-images belonging to the same reference object should be sufficiently close to the reference point corresponding to that reference object's sub-image. Pixels in sub-images not belonging to the reference object not only have a large color difference from pixels in sub-images belonging to the reference object, but are also sufficiently distant from the reference point. Therefore, using the distance between a pixel and the nearest reference point as a distance feature allows for the classification of pixels of different categories into different clusters to a certain extent.

[0264] By referring to the method of obtaining the reference features corresponding to the first pixel, it is possible to obtain the reference features corresponding to each pixel in the sample image.

[0265] Step 701c: Based on the reference features corresponding to each pixel, cluster each pixel to obtain clustering results, which include a first cluster and a second cluster.

[0266] After obtaining the reference features corresponding to each pixel, the pixels are clustered based on the reference features corresponding to each pixel.

[0267] In an exemplary embodiment, before clustering each pixel based on the reference features corresponding to each pixel, the number of clusters K (K is an integer not less than 1) to be obtained is specified first. Then, the K-Means clustering method is used to cluster each pixel. In an exemplary embodiment, the process of clustering each pixel based on the K-Means clustering method is as follows: for a given set of N (N is an integer not less than 1) pixels (x1, x2, ..., x...), ... N Given a sample image x, based on the reference features f corresponding to each pixel... xi (i = 1, 2, ..., N) Divide the N pixels into K clusters (C = C1, C2, ..., Cn). K The goal of the K-Means clustering method is to minimize the difference in reference features between pixels of the same class and maximize the difference in reference features between pixels of different classes. The K-Means clustering method aims to minimize the squared error E, which is calculated based on Equation 4:

[0268]

[0269] Where, μj Represents cluster C j The average reference feature of the pixels in, μ j The calculation formula is:

[0270] In this embodiment, K is an integer not less than 2, so that the pixels are divided into at least two categories: pixels belonging to the sub-image of the reference object and pixels not belonging to the sub-image of the reference object, thereby obtaining a clustering result including a first cluster and a second cluster. The first cluster is the cluster in the clustering result in which the pixels are closest to the reference point, and the second cluster is the cluster in the clustering result in which the pixels are furthest from the reference point. Based on this approach, the first cluster is considered to be a cluster composed of pixels belonging to the sub-image of the reference object, and the second cluster is considered to be a cluster composed of pixels not belonging to the sub-image of the reference object. For example, the distance between the pixels in the cluster and the reference point is calculated by taking the average of the distance features corresponding to each pixel in the cluster as the distance between the pixels in the cluster and the reference point.

[0271] Step 701d: Based on the clustering results, obtain the second auxiliary label. The second auxiliary label includes a sub-label for indicating that the pixels in the first cluster belong to the sub-image, a sub-label for indicating that the pixels in the second cluster do not belong to the sub-image, and a sub-label for indicating that the pixels other than those in the first and second clusters belong to uncertain pixels.

[0272] The second auxiliary label is obtained based on the clustering results. Based on this label, it can be determined which pixels in the sample image belong to the sub-image of the reference object, which pixels do not belong to the sub-image of the reference object, and which pixels are uncertain pixels. This provides strong supervisory information for the model training process. For example, the region where pixels that do not belong to the sub-image of the reference object are located is called the background, and the region where pixels that belong to uncertain pixels are located is called the uncertain region at the boundary between the region where the sub-image of the reference object is located and the background.

[0273] Based on the clustering results, the method for obtaining the second auxiliary label is related to the form of the second auxiliary label. For example, if the form of the second auxiliary label is an image, then an image with the same size as the sample image, which uses different presentation methods to present the pixels in the first cluster, the pixels in the second cluster, and the pixels other than those in the first and second clusters, will be used as the second auxiliary label.

[0274] In an exemplary embodiment, when presenting pixels in the first cluster, pixels in the second cluster, and pixels other than those in the first and second clusters using different presentation methods, the pixels in the first cluster, pixels in the second cluster, and pixels other than those in the first and second clusters can be presented directly using different presentation methods. Alternatively, the pixels obtained by performing a morphological opening operation on the pixels in the first cluster, the pixels obtained by performing a morphological opening operation on the pixels in the second cluster, and the pixels obtained by performing a morphological opening operation on the pixels other than those in the first and second clusters can be presented using different presentation methods.

[0275] For example, second auxiliary labels in image form, such as Figure 10 As shown. In Figure 10 In the diagram, white is used to represent pixels in the first cluster, gray is used to represent pixels in the second cluster, and black is used to represent pixels other than those in the first and second clusters. Based on this, it is believed that... Figure 10 Pixels within the white area belong to the sub-image of the reference object, pixels within the gray area do not belong to the sub-image of the reference object, and pixels within the black area are undefined pixels. In this case, the sub-label used to indicate that pixels in the first cluster belong to the sub-image is presented in white; the sub-label used to indicate that pixels in the second cluster do not belong to the sub-image is presented in gray; and the sub-label used to indicate that pixels other than those in the first and second clusters belong to undefined pixels is presented in black.

[0276] For example, if the second auxiliary label is in the form of a numerical pair, then based on the clustering result, the second auxiliary label is obtained as follows: a fourth value is assigned to pixels in the first cluster, a fifth value is assigned to pixels in the second cluster, and a sixth value is assigned to pixels other than those in the first and second clusters. The position coordinates-label value pairs of each pixel are used as the second auxiliary label in the form of numerical pairs. In this case, the sub-label used to indicate that a pixel in the first cluster belongs to a sub-image is a numerical pair including the fourth value; the sub-label used to indicate that a pixel in the second cluster does not belong to a sub-image is a numerical pair including the fifth value; and the sub-label used to indicate that a pixel other than those in the first and second clusters belongs to an uncertain pixel is a numerical pair including the sixth value.

[0277] The fourth, fifth, and sixth values ​​are set based on experience or can be flexibly adjusted according to the application scenario; this application embodiment does not limit this. For example, the fourth value is the same as the first value (e.g., 1), the fifth value is the same as the second value (e.g., 0), and the sixth value is the same as the third value (e.g., -1).

[0278] Clustering based on the K-Means clustering method is unsupervised clustering. In this embodiment, the second auxiliary label is obtained by relying on the color feature differences and distance feature differences between pixels in the sub-image belonging to the reference object and pixels in the sub-image not belonging to the reference object. For example, the second auxiliary label can also be called a clustering label.

[0279] The label information of the sample image includes at least one of point labels, a first auxiliary label, or a second auxiliary label. Based on the above-described process of obtaining point labels, obtaining the first auxiliary label, and obtaining the second auxiliary label, the label information of the sample image can be obtained.

[0280] The initial image processing model is the image processing model to be trained, used to process the image to segment sub-images of reference objects, obtaining the image segmentation result. The initial image processing model includes an initial encoding model and an initial decoding model. The initial encoding model includes an initial attention model and an initial convolutional model. For a detailed introduction to the initial image processing model, please refer to [link to relevant documentation]. Figure 2 The descriptions of the image processing models in the illustrated embodiments will not be repeated here.

[0281] In step 702, the initial attention model and the initial convolution model are invoked to encode the sample image based on global and local information to obtain sample encoding features.

[0282] For the implementation of step 702, please refer to [link / reference]. Figure 2 Step 202 in the illustrated embodiment will not be repeated here.

[0283] In step 703, the initial decoding model is called to decode the sample encoded features to obtain the sample image features.

[0284] For the implementation of step 703, please refer to [link / reference]. Figure 2 Step 203 in the illustrated embodiment will not be repeated here.

[0285] In step 704, the segmentation result of the sample image is obtained based on the features of the sample image.

[0286] For the implementation of step 704, please refer to [link / reference]. Figure 2 Step 204 in the illustrated embodiment will not be repeated here.

[0287] In step 705, the initial image processing model is trained based on the segmentation results and label information of the sample images to obtain the image processing model.

[0288] After obtaining the segmentation results of the sample images, the initial image processing model is trained based on the segmentation results and label information of the sample images to obtain a trained image processing model. In one possible implementation, the process of training the initial image processing model based on the segmentation results and label information of the sample images is as follows: obtain the target loss function based on the segmentation results and label information of the sample images; and train the initial image processing model using the target loss function.

[0289] The number of sample images can be one or more. If there is only one sample image, the number of segmentation results is also only one. In this case, the target loss function is obtained directly based on the segmentation result and the label information of the sample image. If there are multiple sample images, the number of segmentation results is also multiple. In this case, the target loss function is obtained as follows: based on the segmentation result and label information of each sample image, a sub-loss function is obtained; the average of all obtained sub-loss functions is used as the target loss function. The method for obtaining a sub-loss function when there are multiple sample images is the same as the method for obtaining the target loss function when there is only one sample image. This embodiment uses a single sample image as an example for illustration.

[0290] The target loss function reflects the difference between the segmentation result and the label information of the sample image. For cases where the sample image's label information includes only one label (e.g., pixel-level label, point label, first auxiliary label, second auxiliary label), the target loss function is directly obtained based on the segmentation result and that single label. For example, the cross-entropy loss function between the segmentation result and the single label is used as the target loss function. For example, the cross-entropy loss function between the segmentation result and the single label is calculated based on Equation 5:

[0291]

[0292] Among them, L ce The cross-entropy loss function represents the segmentation result of the sample image and the label; y represents the label. This indicates the segmentation result of the sample image.

[0293] In an exemplary embodiment, the label information of the sample image includes multiple labels. In this case, the target loss function is obtained based on the segmentation result of the sample image and the label information of the sample image by: obtaining the cross-entropy loss function of the segmentation result of the sample image and each label respectively; and obtaining the target loss function based on the cross-entropy loss function of the segmentation result of the sample image and each label.

[0294] For example, if the label information of the sample image includes point labels, a first auxiliary label, and a second auxiliary label, then it is necessary to obtain the cross-entropy loss function between the segmentation result of the sample image and the point labels, the cross-entropy loss function between the segmentation result of the sample image and the first auxiliary label, and the cross-entropy loss function between the segmentation result of the sample image and the second auxiliary label, respectively. Then, based on the three cross-entropy loss functions, a target loss function is obtained, and the target loss function is used to constrain the training process of the model.

[0295] In an exemplary embodiment, during the process of obtaining the cross-entropy loss function between the segmentation result of the sample image and a certain label, the cross-entropy loss function is calculated based on the valid pixels in the sample image, while invalid pixels in the sample image are ignored. For example, valid pixels in the sample image refer to pixels belonging to the sub-image of the reference object as indicated by the label (also referred to as pixels representing positive samples) and pixels not belonging to the sub-image of the reference object (also referred to as pixels representing negative samples). Invalid pixels in the sample image refer to pixels belonging to uncertain pixels as indicated by the label. For example, taking a certain label as... Figure 9 The first auxiliary label shown may be Figure 10 Taking the second auxiliary label as an example, the effective pixels in the sample image are: Figure 9 and Figure 10 The pixels in the white and gray areas of the sample image are invalid pixels. Figure 9 and Figure 10 The pixels within the black area.

[0296] This application does not limit the specific implementation method of obtaining the target loss function based on the segmentation result of the sample image and the cross-entropy loss function of each label. For example, the target loss function is the weighted sum of the segmentation result of the sample image and the cross-entropy loss function of each label. In calculating the weighted sum, the weights corresponding to the segmentation result of the sample image and the cross-entropy loss function of each label are set empirically or flexibly adjusted according to the application scenario; this application does not limit this. For example, if the weights corresponding to the segmentation result of the sample image and the cross-entropy loss function of each label are all 1, then the target loss function is the sum of the segmentation result of the sample image and the cross-entropy loss function of each label.

[0297] After obtaining the target loss function, the initial image processing model is trained using the target loss function to obtain the image processing model. In an exemplary embodiment, the process of training the initial image processing model using the target loss function is an iterative process: the parameters of each model in the initial image processing model are updated backward using the target loss function; after each parameter update, it is determined whether the training process meets the training termination condition; if the training process meets the training termination condition, the iterative process is stopped, and the trained model is taken as the trained image processing model.

[0298] If the training process does not meet the training termination condition, a new target loss function is obtained according to steps 701 to 705, and the parameters of the image processing model are updated in reverse using the new target loss function. This process continues until the training process meets the training termination condition, resulting in a trained image processing model. It should be noted that the sample images used to obtain the new target loss function according to steps 701 to 705 may change or remain the same; this embodiment does not limit this.

[0299] For example, Table 1 shows the comparison results of the image processing model trained based on the method provided in the embodiments of this application and the image processing model composed of convolutional model and decoding model in related technologies:

[0300] Table 1

[0301] Image processing model Accuracy F1 value Related technologies 0.8989 0.7473 This application 0.9007 0.9399

[0302] As shown in Table 1, the image processing model trained based on the method provided in the embodiments of this application can achieve higher accuracy and higher F1 score (an indicator used to measure model performance) than the image processing models in related technologies.

[0303] Based on the technical solution provided in the embodiments of this application, an image processing model including an attention model and a convolution model can be trained. This lays the foundation for calling the attention model and the convolution model to obtain target encoding features based on global and local information. By comprehensively focusing on global and local information, target image features can be obtained. The information focused on is relatively rich, which makes the reliability of target image features higher, thereby improving the accuracy of the obtained segmentation results.

[0304] Next, we will introduce an exemplary application of the embodiments of this application in a real-world application scenario.

[0305] In an exemplary embodiment, the image processing method provided in this application can be applied to a scenario where a histopathological image is processed to segment sub-images of cell nuclei within the histopathological image. In this scenario, the image to be processed is a histopathological image, which is an image acquired by capturing a specific region of a pathological slide obtained after staining with HE dye. The sub-image of the reference object included in the image to be processed refers to the sub-image of cell nuclei stained with H dye in the histopathological image.

[0306] See Figure 11 The method for processing histopathological images includes the following steps 1101 to 1104.

[0307] In step 1101, the pathological image of the tissue to be processed and the image processing model are obtained. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information.

[0308] The image processing model is trained on an initial image processing model based on the sample histopathological image and its label information. For example, the label information of the sample histopathological image includes at least one of point labels, a first auxiliary label, and a second auxiliary label.

[0309] In step 1102, the attention model and convolution model are invoked to encode the histopathological image to be processed based on global and local information to obtain the target encoded features.

[0310] In step 1103, the decoding model is invoked to decode the target encoded features to obtain the target image features.

[0311] In step 1104, the segmentation results of the pathological image of the tissue to be processed are obtained based on the features of the target image.

[0312] The segmentation results of the histopathological image to be processed are used to indicate the region where the cell nucleus sub-image is located in the image to be processed.

[0313] The implementation methods for steps 1101 to 1104 above can be found in [reference needed]. Figure 2 The embodiments shown are not described in detail here.

[0314] The method provided in this application can be applied to the automatic analysis of histopathological images. By subsequently calculating the segmentation results of cell nucleus sub-images, features such as the average size, density, and arrangement of cell nuclei can be obtained, thereby enabling clinical diagnosis and treatment of cancer, including different types of cancer grading and patient risk stratification. Furthermore, in addition to segmenting cell nucleus sub-images, the method provided in this application can also be applied to the segmentation of cell sub-images, or the segmentation of sub-images of tissues that are small in size, numerous, and densely arranged. The principles of cell sub-image segmentation and the segmentation of other tissue sub-images are the same as those of cell nucleus sub-image segmentation, and will not be elaborated further here.

[0315] See Figure 12 This application provides an image processing apparatus, which includes:

[0316] The first acquisition unit 1201 is used to acquire the image to be processed and the image processing model. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information.

[0317] The second acquisition unit 1202 is used to call the attention model and the convolution model to encode the image to be processed based on global and local information to obtain the target encoded features;

[0318] The third acquisition unit 1203 is used to call the decoding model to decode the target encoded features and obtain the target image features;

[0319] The fourth acquisition unit 1204 is used to acquire the segmentation result of the image to be processed based on the features of the target image.

[0320] In one possible implementation, there is at least one encoding model. The second acquisition unit 1202 is used to call the attention model and convolution model in the first encoding model to encode the image to be processed based on global and local information, so as to obtain the basic features and connection features output by the first encoding model. Starting from the second encoding model, the attention model and convolution model in the next encoding model are called to encode the basic features output by the previous encoding model based on global and local information, so as to obtain the basic features and connection features output by the next encoding model, until the basic features and connection features output by the penultimate encoding model are obtained. The connection features output by each encoding model from the first encoding model to the penultimate encoding model are used to provide data support for the step of calling the decoding model to decode the target encoding features and obtain the target image features. The attention model and convolution model in the last encoding model are called to encode the basic features output by the penultimate encoding model based on global and local information, so as to obtain the connection features output by the last encoding model, and the connection features output by the last encoding model are used as the target encoding features.

[0321] In one possible implementation, the attention model in the first encoding model includes a first attention model. The second acquisition unit 1202 is also used to call the convolution model and the first attention model in the first encoding model to encode the image to be processed based on local and global information to obtain the basic features output by the first encoding model; and to obtain the connection features output by the first encoding model based on the basic features output by the first encoding model.

[0322] In one possible implementation, the second acquisition unit 1202 is further configured to call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain a first encoding feature; call the first attention model to encode the image to be processed based on global information to obtain a second encoding feature; fuse the first encoding feature and the second encoding feature to obtain a fused feature; and based on the fused feature, acquire the basic feature output by the first encoding model.

[0323] In one possible implementation, the second acquisition unit 1202 is further configured to acquire block features of each image block of the image to be processed, map the block features of each image block to obtain the mapped features of each image block; acquire image block position features; acquire reference features of the image to be processed based on the mapped features and image block position features of each image block; and call the first attention model to encode the reference features of the image to be processed based on global information to obtain the second encoded features.

[0324] In one possible implementation, the first attention model includes an attention module and a nonlinear processing module. The second acquisition unit 1202 is further configured to call the attention module to process the reference features to obtain a first intermediate feature; concatenate the first intermediate feature and the reference feature to obtain the feature to be processed; call the nonlinear processing module to process the feature to be processed to obtain a second intermediate feature; and concatenate the second intermediate feature and the feature to be processed to obtain a second encoded feature.

[0325] In one possible implementation, the second acquisition unit 1202 is further configured to call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain a first encoding feature; call the first attention model to encode the first encoding feature based on global information to obtain a third encoding feature; and acquire the basic feature output by the first encoding model based on the third encoding feature.

[0326] In one possible implementation, the second acquisition unit 1202 is further configured to call the first attention model to encode the image to be processed based on global information to obtain a second encoded feature; call the convolution model in the first encoding model to encode the second encoded feature based on local information to obtain a fourth encoded feature; and acquire the basic feature output by the first encoding model based on the fourth encoded feature.

[0327] In one possible implementation, the attention model in the first encoding model includes a second attention model. The second acquisition unit 1202 is also used to call the convolution model in the first encoding model to encode the image to be processed based on local information to obtain the basic features output by the first encoding model; and to call the second attention model to encode the basic features output by the first encoding model based on global information to obtain the connection features output by the first encoding model.

[0328] The technical solution provided in this application first calls an attention model and a convolution model to obtain target encoding features based on global and local information, and then obtains target image features based on the target encoding features. Since the target encoding features are obtained based on both global and local information, it ensures that the target image features obtained from the target encoding features are features obtained by comprehensively considering both global and local information. This results in richer information focus, leading to higher reliability of the target image features and thus improving the accuracy of the obtained segmentation results.

[0329] See Figure 13 This application provides a training apparatus for an image processing model, the apparatus comprising:

[0330] The first acquisition unit 1301 is used to acquire sample images, label information of sample images, and an initial image processing model. The initial image processing model includes an initial encoding model and an initial decoding model. The initial encoding model includes an initial attention model and an initial convolution model.

[0331] The second acquisition unit 1302 is used to call the initial attention model and the initial convolution model to encode the sample image based on global information and local information to obtain sample encoding features;

[0332] The third acquisition unit 1303 is used to call the initial decoding model to decode the sample encoding features and obtain the sample image features;

[0333] The fourth acquisition unit 1304 is used to acquire the segmentation result of the sample image based on the features of the sample image;

[0334] Training unit 1305 is used to train the initial image processing model based on the segmentation results and label information of the sample images to obtain the image processing model.

[0335] In one possible implementation, the sample image includes a sub-image of a reference object, and the label information of the sample image includes at least one of a point label, a first auxiliary label, or a second auxiliary label. Both the first and second auxiliary labels are obtained based on the point label, which is determined based on a reference point within the region where the sub-image is located in the sample image.

[0336] Based on the technical solution provided in the embodiments of this application, an image processing model including an attention model and a convolution model can be trained. This lays the foundation for calling the attention model and the convolution model to obtain target encoding features based on global and local information. By comprehensively focusing on global and local information, target image features can be obtained. The information focused on is relatively rich, which makes the reliability of target image features higher, thereby improving the accuracy of the obtained segmentation results.

[0337] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0338] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory, wherein at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors to enable the computer device to implement any of the above-described image processing methods or image processing model training methods. The computer device can be a server or a terminal, and this embodiment does not limit this. The structures of the server and the terminal will be described below.

[0339] Figure 14 This is a schematic diagram of a server structure provided in an embodiment of this application. The server can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1401 and one or more memories 1402. The one or more memories 1402 store at least one computer program, which is loaded and executed by the one or more processors 1401 to enable the server to implement the image processing methods or image processing model training methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0340] Figure 15 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Exemplarily, the terminal can be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0341] Typically, a terminal includes a processor 1501 and a memory 1502.

[0342] Processor 1501 may include one or more processing cores. Processor 1501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1501 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0343] The memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1502 are used to store at least one instruction, which is executed by the processor 1501 to cause the terminal to implement the image processing method or image processing model training method provided in the method embodiments of this application.

[0344] In some embodiments, the terminal may also optionally include: a peripheral device interface 1503 and at least one peripheral device. The processor 1501, memory 1502, and peripheral device interface 1503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507, and a power supply 1509.

[0345] Peripheral interface 1503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1501 and memory 1502. Radio frequency (RF) circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. RF circuit 1504 communicates with communication networks and other communication devices via electromagnetic signals. Display screen 1505 is used to display the user interface (UI). The UI may include graphics, text, icons, video, and any combination thereof. Camera component 1506 is used to capture images or video.

[0346] Audio circuitry 1507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to processor 1501 for processing, or to radio frequency circuitry 1504 for voice communication. The speaker is used to convert electrical signals from processor 1501 or radio frequency circuitry 1504 into sound waves. Power supply 1509 is used to power the various components in the terminal. Power supply 1509 may be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0347] In some embodiments, the terminal further includes one or more sensors 1510. The one or more sensors 1510 include, but are not limited to: an acceleration sensor 1511, a gyroscope sensor 1512, a pressure sensor 1513, an optical sensor 1515, and a proximity sensor 1516.

[0348] Accelerometer 1511 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by the terminal. Gyroscope 1512 can detect the terminal's orientation and rotation angle. Gyroscope 1512 can work in conjunction with accelerometer 1511 to acquire the user's 3D movements on the terminal. Pressure sensor 1513 can be installed on the side bezel of the terminal and / or on the lower layer of display screen 1505. When pressure sensor 1513 is installed on the side bezel of the terminal, it can detect the user's grip signal on the terminal. Processor 1501 can then perform left / right hand recognition or quick operation based on the grip signal acquired by pressure sensor 1513. When pressure sensor 1513 is installed on the lower layer of display screen 1505, processor 1501 can control the operable controls on the UI interface based on the user's pressure on display screen 1505.

[0349] Optical sensor 1515 is used to collect ambient light intensity. Proximity sensor 1516, also known as a distance sensor, is typically located on the front panel of the terminal. Proximity sensor 1516 is used to collect the distance between the user and the front of the terminal.

[0350] Those skilled in the art will understand that Figure 15 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0351] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program, which is loaded and executed by a processor of a computer device to enable the computer to implement any of the above-described image processing methods or image processing model training methods.

[0352] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0353] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the above-described image processing methods or image processing model training methods.

[0354] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0355] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0356] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed and the image processing model. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information. The number of encoding models is at least one. The attention model and convolution model in the first encoding model are invoked to encode the image to be processed based on global and local information, thereby obtaining the basic features and connectivity features output by the first encoding model; Starting from the second encoding model, the attention model and convolution model in the next encoding model are called to encode the basic features output by the previous encoding model based on global and local information, so as to obtain the basic features and connection features output by the next encoding model, until the basic features and connection features output by the penultimate encoding model are obtained. The attention model and convolution model in the last encoding model are called to encode the basic features output by the penultimate encoding model based on global and local information, so as to obtain the connection features output by the last encoding model. The connection features output by the last encoding model are used as the target encoding features. Based on the connection features output by each encoding model from the first encoding model to the penultimate encoding model, the decoding model is invoked to decode the target encoding features to obtain the target image features; Based on the target image features, the segmentation result of the image to be processed is obtained.

2. The method according to claim 1, characterized in that, The attention model in the first encoding model includes a first attention model. The invocation of the attention model and convolutional model in the first encoding model encodes the image to be processed based on global and local information, obtaining the basic features and connectivity features output by the first encoding model, including: The convolutional model and the first attention model in the first encoding model are invoked to encode the image to be processed based on local and global information, so as to obtain the basic features output by the first encoding model; Based on the basic features output by the first encoding model, the connection features output by the first encoding model are obtained.

3. The method according to claim 2, characterized in that, The first encoding model calls the convolutional model and the first attention model to encode the image to be processed based on local and global information, obtaining the basic features output by the first encoding model, including: The convolutional model in the first encoding model is invoked to encode the image to be processed based on local information to obtain the first encoded feature; The first attention model is invoked to encode the image to be processed based on global information to obtain the second encoded features; The first coding feature and the second coding feature are fused to obtain the fused feature; Based on the fusion features, the basic features output by the first encoding model are obtained.

4. The method according to claim 3, characterized in that, The step of calling the first attention model to encode the image to be processed based on global information to obtain the second encoded features includes: Obtain the block features of each image block of the image to be processed, and map the block features of each image block to obtain the mapped features of each image block; Obtain image patch location features; Based on the mapping features of each image patch and the positional features of the image patch, the reference features of the image to be processed are obtained; The first attention model is invoked to encode the reference features of the image to be processed based on global information to obtain the second encoded features.

5. The method according to claim 4, characterized in that, The first attention model includes an attention module and a non-linear processing module. The step of calling the first attention model to encode reference features of the image to be processed based on global information to obtain the second encoded features includes: The attention module is invoked to process the reference features to obtain the first intermediate features; The first intermediate feature and the reference feature are concatenated to obtain the feature to be processed; The nonlinear processing module is invoked to process the feature to be processed, thereby obtaining the second intermediate feature; The second intermediate feature and the feature to be processed are concatenated to obtain the second encoded feature.

6. The method according to claim 2, characterized in that, The first encoding model calls the convolutional model and the first attention model to encode the image to be processed based on local and global information, obtaining the basic features output by the first encoding model, including: The convolutional model in the first encoding model is invoked to encode the image to be processed based on local information to obtain the first encoded feature; The first attention model is invoked to encode the first encoded feature based on global information to obtain the third encoded feature; Based on the third encoding feature, the basic features output by the first encoding model are obtained.

7. The method according to claim 2, characterized in that, The first encoding model calls the convolutional model and the first attention model to encode the image to be processed based on local and global information, obtaining the basic features output by the first encoding model, including: The first attention model is invoked to encode the image to be processed based on global information to obtain the second encoded features; The convolutional model in the first encoding model is invoked to encode the second encoding feature based on local information, thereby obtaining the fourth encoding feature; Based on the fourth encoding feature, the basic features output by the first encoding model are obtained.

8. The method according to claim 1, characterized in that, The attention model in the first encoding model includes a second attention model. The invocation of the attention model and convolutional model in the first encoding model encodes the image to be processed based on global and local information, obtaining the basic features and connectivity features output by the first encoding model, including: The convolutional model in the first encoding model is invoked to encode the image to be processed based on local information, thereby obtaining the basic features output by the first encoding model; The second attention model is invoked to encode the basic features output by the first encoding model based on global information, thereby obtaining the connection features output by the first encoding model.

9. A training method for an image processing model, characterized in that, The method includes: Acquire a sample image, the label information of the sample image, and an initial image processing model. The initial image processing model includes an initial encoding model and an initial decoding model. The initial encoding model includes an initial attention model and an initial convolution model. The number of initial encoding models is at least one. The initial attention model and initial convolution model in the first initial encoding model are invoked to encode the sample image based on global and local information, thereby obtaining the basic features and connection features output by the first initial encoding model; Starting from the second initial encoding model, the initial attention model and initial convolution model in the next initial encoding model are called to encode the basic features output by the previous initial encoding model based on global and local information, so as to obtain the basic features and connection features output by the next initial encoding model, until the basic features and connection features output by the penultimate initial encoding model are obtained. The initial attention model and initial convolution model in the last initial encoding model are called to encode the basic features output by the penultimate initial encoding model based on global and local information, so as to obtain the connection features output by the last initial encoding model. The connection features output by the last initial encoding model are used as sample encoding features. Based on the connection features output by each initial coding model from the first initial coding model to the penultimate initial coding model, the initial decoding model is invoked to decode the sample coding features to obtain sample image features; Based on the features of the sample image, the segmentation result of the sample image is obtained; Based on the segmentation results and label information of the sample images, the initial image processing model is trained to obtain the image processing model.

10. The method according to claim 9, characterized in that, The sample image includes a sub-image of a reference object. The label information of the sample image includes at least one of a point label, a first auxiliary label, or a second auxiliary label. Both the first auxiliary label and the second auxiliary label are obtained based on the point label. The point label is determined based on a reference point within the region where the sub-image is located in the sample image.

11. An image processing apparatus, characterized in that, The device includes: The first acquisition unit is used to acquire the image to be processed and the image processing model. The image processing model includes an encoding model and a decoding model. The encoding model includes an attention model and a convolution model. The attention model encodes based on global information, and the convolution model encodes based on local information. The number of encoding models is at least one. The second acquisition unit is used to call the attention model and convolution model in the first encoding model to encode the image to be processed based on global and local information, thereby obtaining the basic features and connection features output by the first encoding model; starting from the second encoding model, it calls the attention model and convolution model in the next encoding model to encode the basic features output by the previous encoding model based on global and local information, thereby obtaining the basic features and connection features output by the next encoding model, until the basic features and connection features output by the penultimate encoding model are obtained; it calls the attention model and convolution model in the last encoding model to encode the basic features output by the penultimate encoding model based on global and local information, thereby obtaining the connection features output by the last encoding model, and uses the connection features output by the last encoding model as the target encoding features; The third acquisition unit is used to call the decoding model to decode the target encoded features based on the connection features output by each encoding model from the first encoding model to the penultimate encoding model, so as to obtain the target image features; The fourth acquisition unit is used to acquire the segmentation result of the image to be processed based on the features of the target image.

12. The apparatus according to claim 11, characterized in that, The attention model in the first encoding model includes a first attention model and a second acquisition unit, which is used to call the convolution model and the first attention model in the first encoding model to encode the image to be processed based on local information and global information, so as to obtain the basic features output by the first encoding model. Based on the basic features output by the first encoding model, the connection features output by the first encoding model are obtained.

13. The apparatus according to claim 12, characterized in that, The second acquisition unit is used to call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoding feature; The first attention model is invoked to encode the image to be processed based on global information to obtain a second encoded feature; the first encoded feature and the second encoded feature are fused to obtain a fused feature; Based on the fusion features, the basic features output by the first encoding model are obtained.

14. The apparatus according to claim 13, characterized in that, The second acquisition unit is used to acquire the block features of each image block of the image to be processed, map the block features of each image block to obtain the mapping features of each image block, and acquire the image block position features; Based on the mapping features of each image patch and the positional features of the image patch, the reference features of the image to be processed are obtained; The first attention model is invoked to encode the reference features of the image to be processed based on global information to obtain the second encoded features.

15. The apparatus according to claim 14, characterized in that, The first attention model includes an attention module and a nonlinear processing module. The second acquisition unit is used to call the attention module to process the reference feature to obtain a first intermediate feature; and to concatenate the first intermediate feature and the reference feature to obtain the feature to be processed. The nonlinear processing module is invoked to process the feature to be processed to obtain a second intermediate feature; the second intermediate feature and the feature to be processed are concatenated to obtain the second encoded feature.

16. The apparatus according to claim 12, characterized in that, The second acquisition unit is used to call the convolutional model in the first encoding model to encode the image to be processed based on local information to obtain the first encoding feature; The first attention model is invoked to encode the first encoding feature based on global information to obtain the third encoding feature; based on the third encoding feature, the basic feature output by the first encoding model is obtained.

17. The apparatus according to claim 12, characterized in that, The second acquisition unit is used to call the first attention model to encode the image to be processed based on global information to obtain a second encoded feature; and to call the convolution model in the first encoding model to encode the second encoded feature based on local information to obtain a fourth encoded feature. Based on the fourth encoding feature, the basic features output by the first encoding model are obtained.

18. The apparatus according to claim 11, characterized in that, The attention model in the first encoding model includes a second attention model. The second acquisition unit is used to call the convolution model in the first encoding model to encode the image to be processed based on local information, so as to obtain the basic features output by the first encoding model. The second attention model is invoked to encode the basic features output by the first encoding model based on global information, thereby obtaining the connection features output by the first encoding model.

19. A training device for an image processing model, characterized in that, The device includes: The first acquisition unit is used to acquire a sample image, the label information of the sample image, and an initial image processing model. The initial image processing model includes an initial encoding model and an initial decoding model. The initial encoding model includes an initial attention model and an initial convolution model. The number of the initial encoding models is at least one. The second acquisition unit is used to call the initial attention model and initial convolution model in the first initial encoding model to encode the sample image based on global and local information, thereby obtaining the basic features and connection features output by the first initial encoding model; starting from the second initial encoding model, it calls the initial attention model and initial convolution model in the next initial encoding model to encode the basic features output by the previous initial encoding model based on global and local information, thereby obtaining the basic features and connection features output by the next initial encoding model, until the basic features and connection features output by the penultimate initial encoding model are obtained; it calls the initial attention model and initial convolution model in the last initial encoding model to encode the basic features output by the penultimate initial encoding model based on global and local information, thereby obtaining the connection features output by the last initial encoding model, and uses the connection features output by the last initial encoding model as the sample encoding features; The third acquisition unit is used to call the initial decoding model to decode the sample encoding features based on the connection features output by each initial encoding model from the first initial encoding model to the penultimate initial encoding model, so as to obtain sample image features; The fourth acquisition unit is used to acquire the segmentation result of the sample image based on the features of the sample image; The training unit is used to train the initial image processing model based on the segmentation results and label information of the sample images to obtain the image processing model.

20. The apparatus according to claim 19, characterized in that, The sample image includes a sub-image of a reference object. The label information of the sample image includes at least one of a point label, a first auxiliary label, or a second auxiliary label. Both the first auxiliary label and the second auxiliary label are obtained based on the point label. The point label is determined based on a reference point within the region where the sub-image is located in the sample image.

21. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the computer device to implement the image processing method as described in any one of claims 1 to 8, or the training method for the image processing model as described in any one of claims 9 to 10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement the image processing method as described in any one of claims 1 to 8, or the training method for the image processing model as described in any one of claims 9 to 10.

23. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the image processing method as described in any one of claims 1 to 8, or the image processing model training method as described in any one of claims 9 to 10.

Images