Model training methods, image processing methods, devices, equipment and storage media

Abstract

This application discloses a model training method, an image processing method, an apparatus, a device, and a storage medium. The model training method includes: acquiring a sample image and an initial image processing model; extracting image features from the sample image based on a first processing model; performing recognition processing on the image features using a second processing model to obtain a recognition result of the sample image and an uncertainty parameter corresponding to the recognition result; obtaining a first loss function based on the recognition result, the uncertainty parameter, and a first annotation; and training the first processing model and the second processing model according to the first loss function to obtain a target image processing model. The image processing method includes: acquiring a target image to be processed; processing the target image based on the trained target image processing model to obtain a target recognition result of the target image and a target uncertainty parameter corresponding to the target recognition result. Based on the above process, it is beneficial to improve the performance of the image processing model, and the image processing effect is better.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence technology, the demand for image processing using models is becoming increasingly prominent in various fields. For example, in the medical field, models are used for segmentation and classification of fundus images.

[0003] Before using a model to process images, the model needs to be trained. The usual training method is to input sample images into the model, determine the loss by comparing the processing results output by the model with the annotations of the sample images, and then train the model according to the loss function.

[0004] In the above training process, the loss function is only related to the processing result of the model and the annotation of the sample image. The performance of the model trained according to this loss function is poor, and the image processing effect of the trained model is not good. Summary of the Invention

[0005] This application provides a model training method, an image processing method, an apparatus, a device, and a storage medium, which can be used to improve the processing effect of image processing. The technical solution is as follows:

[0006] On the one hand, embodiments of this application provide a model training method, the method comprising:

[0007] Acquire a sample image and an initial image processing model, wherein the sample image has a first annotation and the initial image processing model includes a first processing model and a second processing model;

[0008] Based on the first processing model, image features of the sample image are extracted;

[0009] The image features are processed by the second processing model to obtain the recognition result of the sample image and the uncertainty parameter corresponding to the recognition result. The uncertainty parameter is used to indicate the degree of uncertainty of the recognition result.

[0010] Based on the identification result, the uncertainty parameter, and the first label, a first loss function is obtained;

[0011] The first processing model and the second processing model are trained based on the first loss function to obtain the target image processing model.

[0012] An image processing method is also provided, the method comprising:

[0013] Obtain the target image to be processed;

[0014] The target image processing model trained using the above method is used to process the target image to obtain the target recognition result and the target uncertainty parameter corresponding to the target recognition result. The target uncertainty parameter is used to indicate the degree of uncertainty of the target recognition result.

[0015] On the other hand, a model training apparatus is provided, the apparatus comprising:

[0016] An acquisition module is used to acquire a sample image and an initial image processing model, wherein the sample image has a first annotation and the initial image processing model includes a first processing model and a second processing model.

[0017] The extraction module is used to extract image features from the sample image based on the first processing model;

[0018] The processing module is used to perform recognition processing on the image features through the second processing model to obtain the recognition result of the sample image and the uncertainty parameter corresponding to the recognition result, wherein the uncertainty parameter is used to indicate the degree of uncertainty of the recognition result;

[0019] The acquisition module is further configured to acquire a first loss function based on the identification result, the uncertainty parameter, and the first annotation;

[0020] The training module is used to train the first processing model and the second processing model according to the first loss function to obtain the target image processing model.

[0021] In one possible implementation, the second processing model includes a first sub-model, which includes at least three fully connected layers.

[0022] In one possible implementation, the first sub-model includes a first fully connected layer, a second fully connected layer, and a third fully connected layer, and the recognition result includes a classification result; the processing module is configured to input the image features into the first fully connected layer to obtain fully connected features output by the first fully connected layer; input the fully connected features into the second fully connected layer to obtain the classification result of the sample image output by the second fully connected layer; and input the fully connected features into the third fully connected layer to obtain a classification uncertainty parameter output by the third fully connected layer corresponding to the classification result.

[0023] In one possible implementation, the identification result includes a classification result, the uncertainty parameter includes a classification uncertainty parameter, the first label includes a classification label, and the acquisition module is further configured to acquire a first sub-loss function and a second sub-loss function based on the classification result, the classification uncertainty parameter, and the classification label; set a first weight and a second weight for the first sub-loss function and the second sub-loss function, respectively; and use the sum of the first product and the second product as the first loss function, wherein the first product is the product of the first sub-loss function and the first weight, and the second product is the product of the second sub-loss function and the second weight.

[0024] In one possible implementation, the second processing model includes a second sub-model, which includes multiple Monte Carlo networks, and the recognition result includes a segmentation result; the processing module is used to input the image features into the multiple Monte Carlo networks to obtain multiple prediction results output by the multiple Monte Carlo networks; and based on the multiple prediction results, to determine the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result.

[0025] In one possible implementation, the second processing model includes a first sub-model and a second sub-model, and the recognition result includes a classification result and a segmentation result; the processing module is used to classify the image features using the first sub-model to obtain a classification result of the sample image and a classification uncertainty parameter corresponding to the classification result; and to segment the image features using the second sub-model to obtain a segmentation result of the sample image and a segmentation uncertainty parameter corresponding to the segmentation result.

[0026] In one possible implementation, the first label includes a classification label and a segmentation label, and the first loss function includes a second loss function and a third loss function; the acquisition module is further configured to acquire the second loss function based on the classification result, the classification uncertainty parameter, and the classification label; and acquire the third loss function based on the segmentation result, the segmentation uncertainty parameter, and the segmentation label.

[0027] In one possible implementation, the initial image processing model further includes a third processing model. The processing module is further configured to concatenate the image features and the segmentation results to obtain a concatenated result; and to use the third processing model to jointly process the concatenated result and the classification result to obtain target parameters.

[0028] The acquisition module is further configured to acquire a fourth loss function corresponding to the target parameters based on the target parameters and the classification label;

[0029] The training module is used to train the first processing model, the first sub-model, the second sub-model, and the third processing model according to the second loss function, the third loss function, and the fourth loss function to obtain the target image processing model.

[0030] In one possible implementation, the training module is further configured to update the parameters of the first processing model according to the second loss function, the third loss function, and the fourth loss function; update the parameters of the first sub-model according to the second loss function and the fourth loss function; update the parameters of the second sub-model according to the third loss function and the fourth loss function; and update the parameters of the third processing model according to the fourth loss function.

[0031] In one possible implementation, the third processing model includes a third sub-model and a fourth sub-model. The processing module is further configured to input the splicing result into the third sub-model to obtain a first sub-parameter output by the third sub-model; obtain a second sub-parameter from the classification result; and input the first sub-parameter and the second sub-parameter into the fourth sub-model to obtain a target parameter output by the fourth sub-model.

[0032] In one possible implementation, the processing module is further configured to perform image processing on the test image using the target image processing model to obtain the processing result of the test image;

[0033] The acquisition module is further configured to acquire a first evaluation index and a second evaluation index of the target image processing model based on the processing result of the test image. The first evaluation index is used to evaluate the image processing effect of the target image processing model, and the second evaluation index is used to evaluate the image processing repeatability of the target image processing model.

[0034] An image processing apparatus is also provided, the apparatus comprising:

[0035] The acquisition module is used to acquire the target image to be processed;

[0036] The processing module is used to process the target image based on the trained target image processing model to obtain the target recognition result of the target image and the target uncertainty parameter corresponding to the target recognition result. The target uncertainty parameter is used to indicate the degree of uncertainty of the target recognition result.

[0037] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to implement any of the model training methods or any of the image processing methods described above.

[0038] On the other hand, a computer-readable storage medium is also provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the at least one piece of program code being loaded and executed by a processor to implement any of the model training methods or any of the image processing methods described above.

[0039] The technical solution provided in this application has at least the following beneficial effects:

[0040] During model training, a loss function is obtained based on the recognition results of sample images, the corresponding uncertainty parameter, and the annotations of the sample images. In other words, the loss function considers not only the recognition results and annotations but also the degree of uncertainty in the recognition results. Training the image processing model using this loss function improves its performance. When using the trained image processing model to process target images, in addition to obtaining the recognition results, an uncertainty parameter indicating the degree of uncertainty in the target recognition results is also obtained, resulting in better image processing performance. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0043] Figure 2 This is a flowchart of a model training method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of an initial image processing model provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of an initial image processing model provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of an initial image processing model provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of an initial image processing model provided in an embodiment of this application;

[0048] Figure 7This is a flowchart of an image processing method provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of a model training device provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0054] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0055] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0056] The solutions provided in this application relate to computer vision technology in artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multi-dimensional data. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3-dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0057] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0058] With the continuous development of artificial intelligence technology, the demand for image processing using models is becoming increasingly prominent in various fields. For example, in the medical field, models are used for segmentation and classification of fundus images.

[0059] To address this, embodiments of this application provide a model training method and an image processing method, please refer to... Figure 1 The diagram illustrates the implementation environment of the method provided in this application embodiment. This implementation environment may include: terminal 11 and server 12.

[0060] Both terminal 11 and server 12 can train image processing models using the methods provided in this application embodiment, and this application embodiment does not limit this. Terminal 11 can acquire the target image to be processed, and then process the target image using the target image processing model trained by terminal 11 or server 12. Of course, terminal 11 can also send the acquired target image to server 12, and server 12 can process the target image according to the target image processing model trained by terminal 11 or server 12.

[0061] In one possible implementation, terminal 11 can be any electronic product capable of human-computer interaction with a user through one or more methods such as a keyboard, touchpad, touchscreen, remote control, voice interaction, or handwriting device, such as a PC (Personal Computer), mobile phone, smartphone, PDA (Personal Digital Assistant), wearable device, Pocket PC, tablet computer, smart car system, smart TV, smart speaker, etc. Server 12 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Terminal 11 and server 12 establish a communication connection through a wired or wireless network.

[0062] Those skilled in the art should understand that the above-described terminal 11 and server 12 are merely examples. Other existing or future terminals or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0063] Based on the above Figure 1 The implementation environment shown in this application provides a model training method, taking its application to a terminal as an example. Figure 2 As shown, the method provided in this application embodiment may include the following steps:

[0064] In step 201, a sample image and an initial image processing model are obtained. The sample image has a first annotation, and the initial image processing model includes a first processing model and a second processing model.

[0065] A sample image refers to an image used to train an image processing model. In this embodiment, a supervised model training method is used, so the sample images have a first annotation. The first annotation is used to annotate the true recognition result of the sample image. In this embodiment, the recognition result of the sample image includes at least one of classification results and segmentation results. Accordingly, the first annotation includes at least one of classification annotation and segmentation annotation. The classification annotation is used to annotate the true type of the sample image, and the segmentation annotation is used to annotate the true distribution of regions in the sample image.

[0066] The initial image processing model refers to an image processing model that has not been trained using sample images. The initial image processing model includes, but is not limited to, a first processing model and a second processing model. The first processing model is used to extract image features from the sample images, and the second processing model is used to perform recognition processing on the sample images. In the embodiments of this application, the recognition processing includes at least one of classification processing and segmentation processing.

[0067] Next, we will further introduce the sample images and the initial image processing model:

[0068] 1. Sample Images

[0069] Sample images can be obtained from the training image set. The training image set includes multiple images used to train the initial image processing model. The images in the training image set can be from different devices or institutions to improve the model's generalization ability. In this embodiment, the sample images are sample images from one iteration of training. The number of sample images in one iteration of training can be set empirically, and this embodiment does not limit this. That is, the number of sample images can be one or more.

[0070] The images in the training image set are related to the application scenario of the image processing model; different application scenarios require different training image sets. For example, when the application scenario of the image processing model is to process fundus images to help medical personnel determine whether they are fundus images of the target type, the images in the training image set can be various types of qualified fundus images from multiple medical institutions or image acquisition centers of different levels. For instance, when the target type of fundus image refers to fundus images with the target disease (glaucoma, cataracts, AMD (age-related macular degeneration), diabetic retinopathy, etc.), the training image set can include qualified fundus images suspected of having the target disease or early-stage target disease, fundus images with mid-to-late-stage target disease, normal fundus images, and fundus images with other diseases. The number of fundus images of each type can be the same or different; this application embodiment does not limit this. For example, the number of fundus images of each type can be 2000.

[0071] When the images in the training image set are fundus images, the process of obtaining fundus image annotations can be as follows: multiple ophthalmologists with intermediate or higher professional qualifications annotate the fundus images, and then senior ophthalmologists review the annotations. Once the review is passed, the annotations of the fundus images are obtained. The annotations of fundus images can include various aspects of annotations from ophthalmologists (such as classification annotations, segmentation annotations, etc.) to extract the necessary annotations during model training.

[0072] In one possible implementation, acquiring sample images can refer to randomly extracting a reference number of images from the training image set as sample images. For example, after acquiring the sample images, they can be scaled to have uniform dimensions to facilitate model training. For instance, the sample images can be scaled to a uniform size of 512×512 pixels. It should be noted that if the first annotation includes segmentation annotations, the segmentation annotations need to be scaled to the same dimensions as the sample images. Alternatively, in another possible implementation, the images in the training image set can be made to have uniform dimensions during the construction of the training image set, eliminating the need for scaling when acquiring sample images.

[0073] In one possible implementation, after obtaining sample images with uniform aspect ratios, the sample images can be randomly enhanced to improve the model's generalization ability. The model's generalization ability refers to its adaptability to new samples; the stronger the generalization ability, the stronger the model's adaptability to new samples. In one possible implementation, random enhancement includes one or more of the following: standardization, random rotation, random horizontal flipping, random elastic deformation, and adding noise. Standardization can be achieved by subtracting the image mean from each sample image and then dividing by the image variance. The random rotation angle can range from -30° to +30°. The added noise can be speckle noise, etc.

[0074] 2. Initial Image Processing Model

[0075] The first processing model in the initial image processing model can be any type of network model capable of extracting image features. This application embodiment does not limit the structure of the first processing model or the extracted image features. For example, the first processing model can refer to a U-Net (U-Network) model, a PSPNet (Pyramid Scene Parsing Network) model, a DenseNet (Dense Convolutional Network) model, a ResNet (Residual Network) model, or a MobileNet (Mobile Network) model, etc. The first processing model itself can have initial parameters, which can be parameters pre-trained on the ImageNet dataset. During the first iteration of training, the first processing model is trained based on the initial parameters.

[0076] The second processing model in the initial image processing model is used to perform recognition processing on the sample images. Before training the initial image processing model, the parameters of the second processing model can be initialized. For example, the parameters of the second processing model can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0. Alternatively, the parameters can be initialized using a Gaussian distribution with a variance of 0.001 and a mean of 0. Different model parameters can be initialized in the same way or in different ways; this application embodiment does not limit this. The structure and parameter types of the second processing model can be flexibly set according to requirements; this application embodiment does not limit this. After setting the structure and parameter types and initializing the parameters, the second processing model is obtained.

[0077] In step 202, image features of the sample images are extracted based on the first processing model.

[0078] The first processing model is used to extract features from the sample image. After the sample image is input into the first processing model, the first processing model extracts the features of the sample image and then outputs the image features of the sample image. Different first processing models may extract different image features, and this application embodiment does not limit this.

[0079] It should be noted that when there are multiple sample images, these images are input into the first processing model, which can then output the image features of each sample image. In other words, based on the first processing model, the image features of each sample image can be extracted.

[0080] In step 203, the image features are processed by the second processing model to obtain the recognition result of the sample image and the uncertainty parameter corresponding to the recognition result. The uncertainty parameter is used to indicate the degree of uncertainty of the recognition result.

[0081] The image features output by the first processing model are input into the second processing model. After the second processing model performs recognition processing on the image features, it can output the recognition result of the sample image and the corresponding uncertainty parameter. Since the uncertainty parameter is used to indicate the degree of uncertainty of the recognition result, the reliability of the recognition result can be judged based on the uncertainty parameter corresponding to the recognition result. The greater the uncertainty of the recognition result, the less reliable the recognition result is. It should be noted that the uncertainty parameter can refer to the uncertainty itself, or it can refer to the parameter obtained after converting the uncertainty; this application embodiment does not limit this.

[0082] In this embodiment, the second processing model performs image feature recognition processing including at least one of classification processing and segmentation processing. Accordingly, the second processing model includes at least one of a first sub-model and a second sub-model. The first sub-model is used for image feature classification processing, and the second sub-model is used for image feature segmentation processing. That is, the second processing model has the following three possibilities:

[0083] Scenario 1: The second processing model only includes the first sub-model.

[0084] In this case, the recognition result includes the classification result. The image features are processed by the second processing model to obtain the recognition result of the sample image and the corresponding uncertainty parameter. This includes: the image features are classified by the first sub-model to obtain the classification result of the sample image and the corresponding classification uncertainty parameter. The classification uncertainty parameter indicates the degree of uncertainty in the classification result.

[0085] In one possible implementation, the first sub-model includes at least three fully connected layers. Different fully connected layers are used to obtain different results. For example, the at least three fully connected layers can be used to obtain fully connected features, obtain classification results, and obtain classification uncertainty parameters, respectively. It should be noted that each fully connected layer can be composed of at least one sub-fully connected layer, and this embodiment of the application does not limit this.

[0086] In one possible implementation, the first sub-model includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. The process of classifying image features using the first sub-model to obtain the classification result of the sample image and the corresponding classification uncertainty parameter includes the following three steps:

[0087] Step 1: Input the image features into the first fully connected layer to obtain the fully connected features output by the first fully connected layer.

[0088] The first fully connected layer is used to perform fully connected processing on the image features of the sample image. In one possible implementation, the first fully connected layer may include two sub-fully connected layers. After performing fully connected processing on the image features of the sample image through these two sub-fully connected layers, fully connected features can be output. It should be noted that the two sub-fully connected layers included in the first fully connected layer may have the same initial parameters or different initial parameters; this application embodiment does not limit this.

[0089] It should be noted that when there are multiple sample images, the image features of each sample image can be input into the first fully connected layer to obtain the fully connected features of each sample image.

[0090] Step 2: Input the fully connected features into the second fully connected layer to obtain the classification results of the sample images output by the second fully connected layer.

[0091] The second fully connected layer is used to process the fully connected features to obtain the classification result of the sample image. This application embodiment does not limit the parameter settings of the second fully connected layer, as long as the classification result corresponding to the fully connected features can be obtained.

[0092] It should be noted that, in this embodiment, the classification processing of image features refers to determining whether a sample image is a target type image. The classification result of each sample image can be represented by two probability values, representing the probability that the sample image is a target type image and the probability that the image is a non-target type image, respectively. The target type can be determined based on the type of the sample image and its classification label. For example, when the sample image is a fundus image, and the classification label of the sample image is "having the target disease" or "not having the target disease," the target type is "having the target disease." In this case, the classification result of each sample image can be represented by the probability of having the target disease and the probability of not having the target disease.

[0093] It should be noted that when there are multiple sample images, the fully connected features of each sample image are input into the second fully connected layer to obtain the classification result for each sample image. The classification result of N (an integer not less than 1) sample images can be represented by an N×2 matrix or by N 1×2 matrices; this embodiment does not limit the representation.

[0094] Step 3: Input the fully connected features into the third fully connected layer to obtain the classification uncertainty parameter output by the third fully connected layer, which corresponds to the classification result.

[0095] The third fully connected layer is used to process the fully connected features to obtain the classification uncertainty parameter corresponding to the classification result. This application embodiment does not limit the parameter settings of the third fully connected layer, as long as the classification uncertainty parameter corresponding to the fully connected features can be obtained.

[0096] Since the classification result is represented by two probability values, the classification uncertainty parameter corresponding to each classification result is represented by two parameter values, each of which corresponds to a probability value, used to indicate the degree of uncertainty corresponding to that probability value.

[0097] It should be noted that when there are multiple sample images, inputting the fully connected features of each sample image into the third fully connected layer yields the uncertainty parameter corresponding to the classification result of each sample image. The uncertainty parameter corresponding to the classification result of N sample images can be represented by an N×2 matrix or by N 1×2 matrices; this embodiment does not limit the representation.

[0098] For example, in one case, the initial image processing model can be as follows: Figure 3 As shown. In Figure 3 In the first processing model 31, image features of sample image 30 are extracted and then input into the first fully connected layer 321 of the first sub-model 32. The fully connected features output by the first fully connected layer 321 are then input into the second fully connected layer 322 and the third fully connected layer 323, respectively. After processing the fully connected features, the second fully connected layer 322 outputs the classification result 324 of the sample features. This classification result 324 indicates that the probability of the sample image being a target type image is 0.9, and the probability of it being a non-target type image is 0.1. After processing the fully connected features, the third fully connected layer 323 outputs the classification uncertainty parameter 325. This classification uncertainty parameter 325 indicates that the classification uncertainty parameter for a sample image with a probability of 0.9 being a target type image is 1.9, and the classification uncertainty parameter for a sample image with a probability of 0.1 being a non-target type image is 1.3.

[0099] Scenario 2: The second processing model only includes the second sub-model.

[0100] In this case, the recognition result includes the segmentation result. The image features are processed by a second processing model to obtain the recognition result of the sample image and the corresponding uncertainty parameter. This includes: the image features are segmented by a second sub-model to obtain the segmentation result of the sample image and the corresponding segmentation uncertainty parameter. The segmentation uncertainty parameter indicates the degree of uncertainty in the segmentation result.

[0101] Image feature segmentation refers to dividing a sample image into multiple regions corresponding to different categories by determining the category of each pixel. The segmentation result can be represented by a single sample image with region segmentation labels. The segmentation result can include segmentation results corresponding to different categories, with the segmentation result corresponding to any one category indicating the segmented region in the sample image that corresponds to that category.

[0102] The categories involved in the segmentation result can be consistent with the categories involved in the segmentation annotation of the sample image. For example, assuming the segmentation annotation of the sample image is a sample image segmented into regions corresponding to the first category and regions corresponding to the second category, the segmentation result output by the second sub-model includes the segmentation result corresponding to the first category and the segmentation result corresponding to the second category. The first and second categories can be set according to the application scenario. For example, when the application scenario is to segment fundus images into optic cup and optic disc regions, the first category can be the optic cup, and the second category can be the optic disc. The optic cup and optic disc regions are important indicators used in the medical field to determine whether a fundus image indicates glaucoma.

[0103] Since the segmentation result can be represented by a single sample image with region segmentation labels, the segmentation uncertainty parameter corresponding to the segmentation result can also be represented by a single sample image with a parameter distribution. The sample image with the parameter distribution displays the segmentation uncertainty parameters for each region. It should be noted that when there are multiple sample images, the segmentation result and segmentation uncertainty parameter for each sample image can be obtained.

[0104] In one possible implementation, the second sub-model includes multiple Monte Carlo networks. The process of segmenting image features through the second sub-model to obtain the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result can be as follows: inputting the image features into multiple Monte Carlo networks to obtain multiple prediction results output by the multiple Monte Carlo networks; and determining the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result based on the multiple prediction results.

[0105] A Monte Carlo network can refer to a network obtained by adding a Dropout layer (random deactivation layer) to a trained network. Different Monte Carlo networks have different Dropout layers. This application does not limit the structure of the Monte Carlo network. For example, each Monte Carlo network may sequentially include one convolutional layer, one Dropout layer, three convolutional layers, one deconvolutional layer, and one convolutional layer. The parameters of different convolutional layers can be the same or different, and this application does not limit this. For example, the first four convolutional layers can be 3×3 convolutional layers, and the last convolutional layer can be a 1×1 convolutional layer.

[0106] Image features are input into each Monte Carlo network. By performing T forward operations across T (an integer greater than 1) Monte Carlo networks, T distinct Monte Carlo samples are obtained. These T Monte Carlo samples represent the T prediction results output by the T Monte Carlo networks. Here, the prediction results are segmentation predictions.

[0107] After obtaining T prediction results, the segmentation result of the sample image and the corresponding segmentation uncertainty parameter can be obtained based on these T prediction results. In one possible implementation, assume that the T prediction results are represented as {P1, P2, ..., P...} T Then, the segmentation result can be obtained according to Formula 1, and the segmentation uncertainty parameter can be obtained according to Formula 2:

[0108]

[0109] Where, μ c T represents the segmentation result for category c; T represents the number of prediction results; C represents the number of categories; c represents the category. This represents the prediction result corresponding to category c in the t-th prediction result; u represents the segmentation uncertainty, which is used as the segmentation uncertainty parameter.

[0110] It should be noted that the segmentation uncertainty obtained according to Formula 2 above is the variance uncertainty. Depending on the actual needs, other methods can be used to calculate the uncertainty based on multiple prediction results output by the Monte Carlo network, and the calculated uncertainty can be used as the segmentation uncertainty. For example, the following uncertainty can also be calculated:

[0111] (1) Prediction entropy uncertainty

[0112] Prediction entropy uncertainty is used to measure the amount of information in the prediction density function, and can be calculated according to formula 3.

[0113]

[0114] in, The i-th pixel in the input image is represented by c; the class is represented by C; the number of classes is represented by T; the number of predictions is represented by t; and W represents the t-th prediction. t This represents the parameters of the t-th Monte Carlo network; denoted as the prediction result for the i-th pixel; p represents the probability value; H represents the prediction entropy uncertainty; X represents the distribution of the input image; Y represents the distribution of the annotations of the input image.

[0115] (2) Mutual information uncertainty

[0116] Mutual information uncertainty is used to measure the information entropy of the posterior density function and the predicted density function of the model, and can be calculated according to Equation 4:

[0117]

[0118] Wherein, MI represents mutual information uncertainty; This represents the i-th pixel in the input image; W represents the prediction result for the i-th pixel. t Let X represent the parameters of the t-th Monte Carlo network; X represents the distribution of the input image; Y represents the distribution of the annotations of the input image; and H represents the prediction entropy uncertainty. This represents the expected value of the prediction entropy uncertainty H.

[0119] The specific uncertainty calculation method used in this application embodiment to calculate the segmentation uncertainty of the sample image can be selected according to the actual situation. Furthermore, other methods that can implement the uncertainty calculation in this application embodiment, not mentioned in this application embodiment, can also be applied to this application embodiment. Regardless of the method used, after calculating the uncertainty, this uncertainty can be used as the segmentation uncertainty parameter corresponding to the segmentation result.

[0120] For example, in case two, the initial image processing model can be as follows: Figure 4 As shown. In Figure 4 In the first processing model 41, image features of sample image 40 are extracted, and then the image features are input into Monte Carlo network 421 in the second sub-model 42. Based on the prediction results output by Monte Carlo network 421, segmentation result 422 and segmentation uncertainty parameter 423 corresponding to segmentation result 422 can be obtained respectively. Monte Carlo network 421 includes, in sequence, a 3×3 convolutional layer, a random deactivation layer, three 3×3 convolutional layers, a deconvolutional layer, and a 1×1 convolutional layer.

[0121] Scenario 3: The second processing model includes the first sub-model and the second sub-model.

[0122] In this case, the recognition result includes the classification result and the segmentation result. The image features are processed by the second processing model to obtain the recognition result of the sample image and the corresponding uncertainty parameter, including: classifying the image features using the first sub-model to obtain the classification result of the sample image and the corresponding classification uncertainty parameter; and segmenting the image features using the second sub-model to obtain the segmentation result of the sample image and the corresponding segmentation uncertainty parameter.

[0123] The process of classifying image features using the first sub-model to obtain the classification result of the sample image and the classification uncertainty parameter corresponding to the classification result can be referred to in Case 1 above, and will not be repeated here; the process of segmenting image features using the second sub-model to obtain the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result can be referred to in Case 2 above, and will not be repeated here.

[0124] For example, in case three, the initial image processing model can be as follows: Figure 5As shown. In Figure 5 In the first processing model 51, the image features of the sample image 50 are extracted and input into the first sub-model 52 for classification processing to obtain the classification result 521 and the classification uncertainty parameter 522; the image features are input into the second sub-model 53 for segmentation processing to obtain the segmentation result 531 and the segmentation uncertainty parameter 532.

[0125] In step 204, a first loss function is obtained based on the identification result, uncertainty parameter, and first label.

[0126] The first loss function is used to train the initial image processing model. Corresponding to the three cases of the second processing model in step 203, there are also three ways to implement step 204:

[0127] Case 1: This corresponds to the case where the second processing model only includes the first sub-model. The recognition result includes the classification result, the uncertainty parameter includes the classification uncertainty parameter, and the first label includes the classification label. In this case, the first loss function is obtained based on the recognition result, the uncertainty parameter, and the first label as follows: the first loss function is obtained based on the classification result, the classification uncertainty parameter, and the classification label.

[0128] In one possible implementation, the process of obtaining the first loss function based on the classification result, classification uncertainty parameter, and classification label includes the following three steps:

[0129] Step 1: Based on the classification results, classification certainty parameters, and classification labels, obtain the first sub-loss function and the second sub-loss function respectively.

[0130] In one possible implementation, based on the classification result, classification certainty parameter, and classification label, the first sub-loss function can be obtained according to Formula 5, and the second sub-loss function can be obtained according to Formula 6.

[0131]

[0132] Among them, L KL L represents the first sub-loss function, which is mainly used to train the process of determining the classification uncertainty parameters; cls This represents the second sub-loss function, which uses an attention mechanism to weight the classification uncertainty parameter with the classification result. It is primarily used to train the process of determining the classification result; α e Represents the classification uncertainty parameter; u e This represents the classification uncertainty determined based on the classification uncertainty parameter. x g Indicates category labeling; x e⊙ represents the classification result; ⊙ represents the product operation of corresponding items. Since the classification result can be represented by two probability values, x g x e α e and u e All are two-dimensional vectors.

[0133] Step 2: Set the first weight and the second weight for the first sub-loss function and the second sub-loss function, respectively.

[0134] The first and second weights can be set based on experience or adjusted freely according to the application scenario; this application embodiment does not limit this. For example, the first weight can be set to 0.1, and the second weight can be set to 0.9, etc.

[0135] Step 3: Use the sum of the first product and the second product as the first loss function. The first product is the product of the first sub-loss function and the first weight, and the second product is the product of the second sub-loss function and the second weight.

[0136] The first loss function can be expressed by Equation 7:

[0137]

[0138] in, β1 represents the first loss function; L represents the first weight. KL Denotes the first sub-loss function; β1L KL β1 represents the first product; β2 represents the second weight; L cls Denotes the second sub-loss function; β2L cls This represents the second product.

[0139] For example, in case 1, the initial image processing model can be as follows: Figure 3 As shown. In Figure 3 In the middle, the obtained As the first loss function.

[0140] In obtaining the first loss function using the above method, the first loss function includes a product term of the classification result and the classification uncertainty parameter. The larger the classification uncertainty parameter, the greater its proportion in the first loss function. Therefore, based on the attention mechanism, the system can focus on learning the classification results with higher classification uncertainty parameters, which is beneficial for improving classification performance.

[0141] Case 2: This corresponds to the case where the second processing model only includes the second sub-model. The identification result includes the segmentation result, the uncertainty parameter includes the segmentation uncertainty parameter, and the first label includes the segmentation label. In this case, the first loss function is obtained based on the identification result, the uncertainty parameter, and the first label as follows: the first loss function is obtained based on the segmentation result, the segmentation uncertainty parameter, and the segmentation label.

[0142] In one possible implementation, using Equation 8, the first loss function is obtained based on the segmentation result, the segmentation uncertainty parameter, and the segmentation label:

[0143]

[0144] in, This represents the first loss function for each sample image; N represents the number of pixels; y i μ represents the segmentation label of the i-th pixel. i This represents the segmentation result of the i-th pixel; u i Represents the segmentation uncertainty parameter for the i-th pixel; rescale(u i Let represent the segmentation uncertainty parameter after normalization calculation for the i-th pixel. The first loss function for a sample image can be obtained by summing the calculation results for each pixel.

[0145] When there are multiple sample images, the first loss function corresponding to each sample image can be calculated using Formula 8 above, and then the average value of the first loss functions corresponding to each sample image can be used as the first loss function in this training.

[0146] For example, in case 2, the initial image processing model can be as follows: Figure 4 As shown. In Figure 4 In the middle, the obtained As the first loss function.

[0147] In obtaining the first loss function using the above method, the first loss function includes a product term of the segmentation result and the segmentation uncertainty parameter. The larger the segmentation uncertainty parameter, the greater its proportion in the first loss function. Therefore, based on the attention mechanism, the segmentation result with a higher segmentation uncertainty parameter can be prioritized for learning, which is beneficial for improving the segmentation effect.

[0148] Scenario 3: This corresponds to the second processing model including both a first and a second sub-model. The identification results include classification and segmentation results, the uncertainty parameters include classification uncertainty and segmentation uncertainty parameters, the first labeling includes classification and segmentation labels, and the first loss function includes a second and a third loss function. In this case, based on the identification results, uncertainty parameters, and first labeling, the first loss function is obtained as follows: the second loss function is obtained based on the classification results, classification uncertainty parameters, and classification labels; and the third loss function is obtained based on the segmentation results, segmentation uncertainty parameters, and segmentation labels.

[0149] The process of obtaining the second loss function based on the classification result, classification uncertainty parameter, and classification label can be referred to Case 1 above, and will not be repeated here. The first loss function obtained in Case 1 is used as the second loss function here. The process of obtaining the third loss function based on the segmentation result, segmentation uncertainty parameter, and segmentation label can be referred to Case 2 above, and will not be repeated here. The first loss function obtained in Case 2 is used as the third loss function here.

[0150] For example, in case 3, the initial image processing model can be as follows: Figure 5 As shown. In Figure 5 In the middle, the obtained As the second loss function, the obtained As the third loss function.

[0151] In one possible implementation, under case 3, the initial image processing model may further include a third processing model. After obtaining the second and third loss functions, a fourth loss function can be obtained based on the third processing model. The third processing model is used to jointly process the classification result obtained from the first sub-model and the segmentation result obtained from the second sub-model.

[0152] The classification result includes two probability values, representing the probability that the sample image is a target type image and a non-target type image, respectively. Based on the classification result, it is possible to directly identify whether the sample image is a target type image. The segmentation result can be represented by a sample image with region segmentation labels. Based on the segmentation result, different regions corresponding to different categories can be determined. Then, based on the size of the different regions corresponding to different categories, index parameters that indicate whether the sample image is a target type image can be calculated. Finally, the index parameters are used to identify whether the sample image is a target type image. In other words, the segmentation result can indirectly identify whether the sample image is a target type image. As can be seen from the above analysis, the tasks of the first sub-model and the second sub-model are similar. To avoid contradictions between the outputs of the first and second sub-models, this embodiment uses a third processing model to jointly process the outputs of the first and second sub-models to improve the consistency of their outputs.

[0153] In one possible implementation, the process of obtaining the fourth loss function based on the third processing model includes the following three steps:

[0154] Step 1: Combine the image features and segmentation results to obtain the combined result.

[0155] In one possible implementation, the process of concatenating image features and segmentation results is as follows: the segmentation results are reduced in dimensionality using a pooling layer, and the reduced-dimensional segmentation results are then concatenated with the image features using a concatenation layer to obtain the concatenated result. The pooling layer can be a max-pooling layer, and this embodiment does not limit this to a specific type.

[0156] Step 2: Use the third processing model to jointly process the splicing results and classification results to obtain the target parameters.

[0157] The target parameters are used to characterize the joint parameters of the segmentation and classification results. In one possible implementation, the third processing model includes a third sub-model and a fourth sub-model. The process of using the third processing model to jointly process the concatenation and classification results to obtain the target parameters includes the following steps a to c:

[0158] Step a: Input the splicing result into the third sub-model to obtain the first sub-parameter output by the third sub-model.

[0159] The first sub-parameter refers to the parameters related to the segmentation result. The first sub-parameter may include the size of the region corresponding to each category, determined based on the segmentation result. The size of the region can refer to its vertical diameter, its horizontal diameter, or its area.

[0160] For example, in the case where the segmentation process involves dividing a sample image into regions corresponding to a first category and regions corresponding to a second category, the first sub-parameter may include... and in, This represents the size of the region formed by pixels of the first category (represented by 0) in the segmentation result; This represents the size of the region formed by pixels of the second category (represented by 1) in the segmentation result.

[0161] The size of the region formed by pixels of the first category (represented by 0) in the segmentation result The size of the region formed by the pixels of the second category (represented by 1) in the segmentation result. The ratio of the size of the optic cup to the size of the optic disc is used to characterize whether a sample image is of the target type. For example, in the process of image processing of a fundus image to determine whether it is an image of glaucoma (the target type), the region formed by pixels of the first category (represented by 0) can refer to the optic cup region, and the region formed by pixels of the second category (represented by 1) can refer to the optic disc region. The ratio of the size of the optic cup region to the size of the optic disc region is called the CDR (Cup-Disc Ratio). In the medical field, the CDR is an important parameter for determining whether a fundus image is of glaucoma; the larger the cup-disc ratio, the greater the probability that the fundus image is of glaucoma.

[0162] In one possible implementation, the structure of the third sub-model may sequentially include three convolutional layers and three fully connected layers. This application embodiment does not limit the parameters of the convolutional and fully connected layers. For example, the convolutional layers can be 3×3 convolutional layers. Since the stitching result is obtained by stitching the segmentation result and image features, after inputting the stitching result into the third sub-model, the third sub-model can process the stitching result and output the first sub-parameter related to the segmentation result.

[0163] Step b: Obtain the second sub-parameter from the classification results.

[0164] The classification result includes the probability that the image is the target type and the probability that it is a non-target type. The second sub-parameter refers to the probability that the image in the classification result is the target type.

[0165] For example, the classification result can be represented by x e Indicates that x e It can include and in, This represents the probability that the image is a non-target type. This represents the probability that the image is of the target type. Here, we can... As the second sub-parameter.

[0166] Step c: Input the first and second sub-parameters into the fourth sub-model to obtain the target parameters output by the fourth sub-model.

[0167] The fourth sub-model is used to perform joint operations on the first and second sub-parameters to obtain the target parameters. In one possible implementation, after inputting the first and second sub-parameters into the fourth sub-model, the fourth sub-model can obtain the target parameters using Formula 9:

[0168]

[0169] Where δ represents the target parameter; and Indicates the first sub-parameter; This indicates the second sub-parameter; the 10 in the denominator -6 This helps prevent the denominator from being zero. (Formula 9) This refers to the index parameter obtained from the segmentation results, which characterizes whether a sample image is a target type image. The larger the index parameter, the greater the probability that the sample image is a target type image. This represents an index parameter, obtained from the classification results, characterizing whether a sample image is of the target type. The larger the index parameter, the greater the probability that the sample image is of the target type. Multiplying the two formulas deeply correlates the segmentation and classification processes, enhancing the predictive consistency between the two processes.

[0170] For example, in the process of image processing of fundus images to determine whether the fundus image is a glaucoma image (target type), the target parameter can be represented as CDOS (Optic Cup Disk Overlap Score). CDOS is used to improve the consistency between the classification and segmentation results of fundus images.

[0171] Step 3: Based on the target parameters and classification labels, obtain the fourth loss function corresponding to the target parameters.

[0172] The fourth loss function refers to the loss function corresponding to the target parameters. The classification label includes the probability that the image is a target type and the probability that it is a non-target type. When the sample image is a target type image, the probability of it being a target type image in the classification label is 1, and the probability of it being a non-target type image is 0; when the sample image is not a target type image, the probability of it being a target type image in the classification label is 0, and the probability of it being a non-target type image is 1. In other words, the probability of it being a target type image in the classification label is either 0 or 1.

[0173] In one possible implementation, the fourth loss function corresponding to the target parameters can be obtained based on the target parameters and classification labels by calculating the probability (0 or 1) of the target parameters and the probability of the image being the target type in the classification labels. This application does not limit the method of calculating the fourth loss function based on the probability of the target parameters and the probability of the image being the target type in the classification labels; any method can be used. For example, the cross-entropy loss, calculated from the probability of the target parameters and the probability of the image being the target type in the classification labels, can be used as the fourth loss function. The cross-entropy loss is used to evaluate the difference between the probability of the target parameters and the probability of the image being the target type in the classification labels. The smaller the cross-entropy loss, the smaller the difference between the probability of the target parameters and the probability of the image being the target type in the classification labels, indicating better consistency between the classification and segmentation processing results.

[0174] For example, in case 3, when the initial image processing model also includes a third processing model, the initial image processing model can be as follows: Figure 6 As shown. In Figure 6 In the first processing model 61, image features of sample image 60 are extracted, and the image features are input into the first sub-model 62 for classification processing to obtain classification result 621, classification uncertainty parameter 622, and second loss function. Image features are input into the second sub-model 63 for segmentation processing, resulting in segmentation result 631, segmentation uncertainty parameter 632, and a third loss function. After the segmentation result 631 is processed by the max pooling layer 64, it is concatenated with image features in the concatenation layer 65 to obtain the concatenated result. The concatenated result is input into the third sub-model 661 in the third processing model 66 to obtain the first sub-parameter. The first sub-parameter and the second sub-parameter extracted from the classification result 621 are input into the fourth sub-model 662 to obtain the target parameter δ, and then the fourth loss function L corresponding to the target parameter δ is obtained. δ The third sub-model 661 consists of three 3×3 convolutional layers and three fully connected layers.

[0175] In step 205, a first processing model and a second processing model are trained according to the first loss function to obtain the target image processing model.

[0176] The process of training the first processing model and the second processing model based on the first loss function can be as follows: update the parameters of the first processing model and the parameters of the second processing model in reverse according to the first loss function.

[0177] In this embodiment, the parameters in the model can be solved using gradient descent. For example, the convolutional parameters and bias parameters in the model are solved using Adam's gradient descent method. During iterative training, a learning rate decay method combining warm-up and cosine annealing can be used.

[0178] It should be noted that, corresponding to the three cases of obtaining the first loss function in step 204, the training of the first processing model and the second processing model based on the first loss function in this step 205 includes the following three cases:

[0179] Case A: Corresponding to Case 1 in step 204, the process of training the first processing model and the second processing model according to the first loss function is as follows: update the parameters of the first processing model and the parameters of the first sub-model according to the first loss function.

[0180] Case B: Corresponding to Case 2 in step 204, the process of training the first processing model and the second processing model according to the first loss function is as follows: update the parameters of the first processing model and the parameters of the second sub-model according to the first loss function.

[0181] Case C: Corresponding to Case 3 in step 204, the first loss function includes the second loss function and the third loss function; the process of training the first processing model and the second processing model according to the first loss function is as follows: update the parameters of the first processing model according to the second loss function and the third loss function, update the parameters of the first sub-model according to the second loss function, and update the parameters of the second sub-model according to the third loss function.

[0182] Furthermore, in case 3 of step 204, where the initial image processing model also includes a third processing model, a fourth loss function is obtained in addition to the second and third loss functions. In this case, the process of training the first and second processing models based on the first loss function to obtain the target image processing model is as follows: The first processing model, the first sub-model, the second sub-model, and the third processing model are trained based on the second, third, and fourth loss functions to obtain the target image processing model.

[0183] In one possible implementation, the process of training the first processing model, the first sub-model, the second sub-model, and the third processing model based on the second loss function, the third loss function, and the fourth loss function is as follows: updating the parameters of the first processing model based on the second loss function, the third loss function, and the fourth loss function; updating the parameters of the first sub-model based on the second loss function and the fourth loss function; updating the parameters of the second sub-model based on the third loss function and the fourth loss function; and updating the parameters of the third processing model based on the fourth loss function.

[0184] The process of training the initial image processing model based on the loss function is an iterative process. Each time the model is trained, it is determined whether the training termination condition is met. If the training termination condition is not met, sample images for the next training iteration are acquired, and training continues according to steps 201 to 205 until the training termination condition is met. The image processing model obtained when the training termination condition is met is taken as the target image processing model.

[0185] It should be noted that, in each acquisition of sample images for one iteration of training, assuming the number of sample images acquired each time is m, then m sample images are randomly selected from the training image set without replacement for each iteration of training, until all images in the training image set have been extracted. If the training termination condition is not met at this point, the training image set is initialized, that is, the training image set is restored, and the above process is repeated until the training termination condition is met.

[0186] In one possible implementation, the training termination conditions include, but are not limited to, the following three:

[0187] 1. The number of iterations for training has reached the threshold.

[0188] The number of times threshold can be set based on experience or adjusted flexibly according to the application scenario. This application embodiment does not limit this.

[0189] 2. The loss function is less than the loss threshold.

[0190] When training an initial image processing model using multiple loss functions, the training termination condition is met when each loss function is less than a loss threshold. It should be noted that different loss functions can correspond to the same loss threshold or different loss thresholds; this embodiment does not limit this.

[0191] 3. The loss function converges.

[0192] Loss function convergence means that, as the number of training iterations increases, the fluctuation range of the loss function in the training results of a reference number of iterations remains within the reference range.

[0193] For example, suppose the reference range is -10. -3 ~10 -3 Assume the reference number is 10. If the loss function fluctuates within -10 in all 10 training iterations... -3 ~10 -3 If the loss function is considered to converge, then it is assumed to be convergent.

[0194] When training an initial image processing model using multiple loss functions, the training termination condition is met when each loss function converges. It should be noted that different loss functions may correspond to the same reference range or different reference ranges; this embodiment does not limit this.

[0195] In one possible implementation, after training the target image processing model, the model can be tested using test images to evaluate its performance. This process can be as follows: The target image processing model is used to process the test image to obtain the processing result; based on the processing result, a first evaluation metric and a second evaluation metric of the target image processing model are obtained. The first evaluation metric is used to evaluate the image processing effect of the target image processing model, and the second evaluation metric is used to evaluate the image processing repeatability of the target image processing model.

[0196] The first evaluation metric includes at least one of a first sub-evaluation metric and a second sub-evaluation metric. The first sub-evaluation metric is used to evaluate the classification performance of the target image processing model; the second sub-evaluation metric is used to evaluate the segmentation performance of the target image processing model. The processes for obtaining the first and second sub-evaluation metric are described below:

[0197] First, the process of obtaining the first sub-evaluation metric is introduced: the target image processing model is used to process the test image to obtain the classification result of the test image; based on the classification result of the test image, the first sub-evaluation metric of the target image processing model is obtained. The first sub-evaluation metric includes one or more of the following metrics: sensitivity, specificity, ROC (Receiver-Operator Characteristic) curve, and AUC (Area Under the Curve of ROC).

[0198] Sensitivity and specificity can be calculated based on the classification results and annotations of the test images. Sensitivity measures the proportion of images in the test images that are both labeled as the target type and classified as the target type. Specificity measures the proportion of images in the test images that are both labeled as non-target types and classified as non-target types. Specifically, a classification result indicating the target type means that the probability of an image being the target type is not less than a probability threshold; a classification result indicating non-target type means that the probability of an image being the target type is less than the probability threshold. The probability threshold can be set empirically; for example, it can be set to 0.85.

[0199] In one possible implementation, an image labeled as the target type and also classified as the target type is called a true positive image; an image labeled as a non-target type but classified as the target type is called a false positive image; an image labeled as a non-target type and also classified as a non-target type is called a true negative image; and an image labeled as the target type but classified as a non-target type is called a false negative image. Sensitivity can then be calculated using Formula 10; specificity can be calculated using Formula 11.

[0200]

[0201] The following describes how to determine the ROC curve: Different thresholds are obtained between 0 and 1, with reference values ​​(e.g., 0.01) as intervals. Each threshold corresponds to an operation point. The test image is then binary-classified based on any given threshold. This is done by comparing the probability of the test image being the target type with the probability of it being classified as such in the classification results. After binary classification, the sensitivity and specificity corresponding to the operation point at each threshold are obtained. By plotting the specificity on the horizontal axis and the sensitivity on the vertical axis for each threshold, the ROC curve can be obtained.

[0202] After obtaining the ROC curve, the area between the ROC curve and the horizontal axis (0-1) is taken as the AUC. The larger the AUC, the better the classification effect of the target image processing model.

[0203] Next, the process of obtaining the second sub-evaluation metric is described: The target image processing model is used to process the test image to obtain the segmentation result; based on the segmentation result, the second sub-evaluation metric of the target image processing model is obtained. The second sub-evaluation metric can be one or more of the Dice coefficient and error metrics.

[0204] The Dice coefficient measures the accuracy of segmentation by a target image processing model. Its value ranges from 0 to 1. A Dice coefficient closer to 1 indicates higher segmentation accuracy, meaning a better segmentation effect. The Dice coefficient can be calculated using formula 12:

[0205]

[0206] Where c represents the segmentation category; The size of the region formed by pixels of category c in the segmentation result; This indicates the size of the region formed by pixels of category c in the segmentation annotation. This represents the size of the overlapping region formed by the pixels of category c in the segmentation result and the segmentation annotation. Formula 12 above can calculate the Dice coefficient corresponding to each test image, and the average value of the Dice coefficients corresponding to multiple test images can be used as the second sub-evaluation index for evaluating the segmentation effect of the target image processing model.

[0207] The error metric can be the root mean square error or the R-squared value. In one possible implementation, for the segmentation process that divides the test image into regions corresponding to the first category (first region) and regions corresponding to the second category (second region), the ratio of the size of the first region to the size of the second region in the segmentation result (prediction ratio) can be calculated according to Formula 13, and the ratio of the size of the first region to the size of the second region in the segmentation annotation (calibration ratio) can be calculated according to Formula 14.

[0208]

[0209] in, This indicates the size of the first region in the segmentation result; This indicates the size of the second region in the segmentation result; Indicates the size of the first region in the segmentation annotation; This indicates the size of the second region in the segmentation annotation.

[0210] After obtaining the prediction ratio and calibration ratio for each test image, the average of the calibration ratios for all test images can be used as the average calibration ratio. Then, the root mean square error is calculated according to Formula 15, and the R-squared value is calculated according to Formula 16.

[0211]

[0212] The root mean square error (RMSE) measures the deviation between the prediction ratio and the calibration ratio, reflecting the segmentation accuracy of the target image processing model. The closer the RMSE is to 0, the better the segmentation performance of the target image processing model.

[0213] The R-squared value, also known as the coefficient of determination, is a statistical coefficient that measures the goodness of fit between the regression prediction ratio and the calibration ratio. The R-squared value ranges from 0 to 1. The closer the R-squared value is to 0, the closer the segmentation effect of the target image processing model is to randomness; the closer the R-squared value is to 1, the better the segmentation effect of the target image processing model.

[0214] The second evaluation metric is used to evaluate the image processing repeatability of the target image processing model. In one possible implementation, the second evaluation metric of the target image processing model can be obtained by repeatedly processing the same multiple test images three times using the target image processing model, obtaining three sets of processing results, calculating the Kappa coefficient between any two sets of processing results, and using the average of the three Kappa coefficients as the second evaluation metric. It should be noted that the processing result here can refer to either classification or segmentation results; this embodiment does not limit this. Alternatively, Kappa coefficients can be calculated separately for classification and segmentation results, and then the average of the Kappa coefficients corresponding to the classification and segmentation results can be used as the Kappa coefficient between the two sets of processing results.

[0215] The Kappa coefficient between any two sets of treatment results can be calculated using Formula 17:

[0216]

[0217] Where κ represents the Kappa coefficient; p o The observational consistency rate can be calculated using formula 18; p e The expected consistency rate is used to represent the consistency rate between two treatment results due to chance, and can be calculated according to formula 19.

[0218]

[0219] Where c represents the segmentation category in the segmentation process or the target type in the classification process; M represents the confusion matrix corresponding to the two sets of processing results, which is a c×c matrix; and n represents the number of test images corresponding to each set of processing results.

[0220] The theoretical range of the Kappa coefficient is between -1 and 1, but it is usually between 0 and 1. A Kappa value between 0.0 and 0.2 indicates almost no consistency, between 0.21 and 0.39 indicates very low consistency, between 0.4 and 0.59 indicates weak consistency, between 0.6 and 0.79 indicates moderate consistency, between 0.8 and 0.9 indicates strong consistency, and between 0.9 and 1.0 indicates near-perfect consistency. The image processing repeatability of the target image processing model can be evaluated based on the degree of consistency between any two sets of processing results.

[0221] If the evaluation results based on the first evaluation metric indicate that the image processing performance of the target image processing model is poor, or if the evaluation results based on the second evaluation metric indicate that the image processing repeatability of the target image processing model is poor, then the target image processing model can be further trained to improve its image processing performance and repeatability.

[0222] In one possible implementation, the technical solution of this application embodiment can be implemented using PyTorch and run on an Nvidia Tesla P40 graphics card.

[0223] In this embodiment, during model training, a loss function is obtained based on the recognition result of the sample image, the uncertainty parameter corresponding to the recognition result, and the annotation of the sample image. In other words, the loss function considers not only the recognition result and the annotation, but also the degree of uncertainty of the recognition result. Training the image processing model using the loss function obtained in this way is beneficial to improving the performance of the image processing model, thereby improving the effect of using the trained image processing model to process the target image.

[0224] based on Figure 1 The implementation environment shown in this application provides an image processing method, taking its application to a terminal as an example. For instance... Figure 7 As shown, the method provided in this application embodiment may include the following steps:

[0225] In step 701, the target image to be processed is acquired.

[0226] The target image to be processed refers to an image of the same type as the images in the model training image set. For example, when the images in the model training image set are fundus images, the target image is also a fundus image. It should be noted that the target image is unlabeled.

[0227] In one possible implementation, after acquiring the target image to be processed, the target image is scaled to have the same length and width dimensions (e.g., 512×512 (pixels)) as the sample images used during model training, in order to reduce the adverse effect of image size on image processing results.

[0228] After scaling the target image, it can be further enhanced randomly. One possible implementation includes one or more of the following: normalization, random rotation, random horizontal flipping, random elastic deformation, and noise addition. Normalization can involve subtracting the image mean from the target image and then dividing by the image variance. The random rotation angle can range from -30° to +30°. The added noise can be speckle noise, etc.

[0229] In step 702, the target image is processed based on the trained target image processing model to obtain the target recognition result of the target image and the target uncertainty parameter corresponding to the target recognition result. The target uncertainty parameter is used to indicate the degree of uncertainty of the target recognition result.

[0230] Target image processing model refers to the use of Figure 2 The image processing model trained by the method in the illustrated embodiment. The process of processing the target image based on the trained target image processing model can be found in [reference needed]. Figure 2 The process of processing sample images during model training in the illustrated embodiment will not be described in detail here.

[0231] It should be noted that, depending on the specific circumstances of the second processing model within the target image processing model, the target recognition result and target uncertainty parameter of the target image obtained in step 702 will also differ.

[0232] When the second processing model in the target image processing model only includes the first sub-model for classification processing, the target image can be processed based on the trained target image processing model to obtain the target classification result of the target image and the target classification uncertainty parameter corresponding to the target classification result. The target classification uncertainty parameter is used to indicate the degree of uncertainty of the target classification result.

[0233] When the second processing model in the target image processing model only includes the second sub-model for segmentation processing, the target image can be processed based on the trained target image processing model to obtain the target segmentation result of the target image and the target segmentation uncertainty parameter corresponding to the target segmentation result. The target segmentation uncertainty parameter is used to indicate the degree of uncertainty of the target segmentation result.

[0234] When the second processing model in the target image processing model includes a first sub-model for classification processing and a second sub-model for segmentation processing, the target image can be processed based on the trained target image processing model to obtain the target classification result of the target image and the target classification uncertainty parameter corresponding to the target classification result, as well as the target segmentation result and the target segmentation uncertainty parameter corresponding to the target segmentation result.

[0235] In one possible implementation, the target image processing model further includes a third processing model. Based on the trained target image processing model, the target image is processed. In addition to obtaining the target recognition result of the target image and the target uncertainty parameter corresponding to the target recognition result, a recognition parameter can also be obtained. This recognition parameter is output by the third processing model and can be used as a basis for determining whether the target image is a target type image.

[0236] In one possible implementation, the technical solution of this application embodiment can be implemented using PyTorch and run on an Nvidia Tesla P40 graphics card.

[0237] The target image processing model provided in this application performs image processing on the target image, resulting in highly accurate classification and segmentation results. Furthermore, by predicting the uncertainty parameters of the recognition results (classification and segmentation results), the problem of incorrect judgments caused by directly using the output recognition results as the basis for the final judgment is avoided. The uncertainty parameters of the recognition results provide a quantitative indicator for the judgment of the final result, helping to improve the accuracy of the final judgment. In addition, the recognition parameters, obtained by jointly processing the classification and segmentation results, also provide a quantitative indicator for the judgment of the final result, further improving the accuracy of the final judgment.

[0238] For example, when processing fundus images using a target image processing model, the system can comprehensively consider classification results, classification uncertainty parameters, segmentation results, segmentation uncertainty parameters, and recognition parameters to ultimately determine whether a fundus image has a certain disease, thereby improving the accuracy of fundus image diagnosis and reducing misdiagnosis and medical accidents caused by incorrect judgment.

[0239] In this embodiment, during model training, a loss function is obtained based on the recognition result of the sample image, the uncertainty parameter corresponding to the recognition result, and the annotation of the sample image. That is, the loss function considers not only the recognition result and the annotation, but also the degree of uncertainty of the recognition result. Training the image processing model using the loss function obtained in this way is beneficial to improving the performance of the image processing model. When using the trained image processing model to process the target image, in addition to obtaining the recognition result of the target image, an uncertainty parameter indicating the degree of uncertainty of the target recognition result can also be obtained, resulting in better image processing performance.

[0240] See Figure 8 This application provides a model training apparatus, which includes:

[0241] The acquisition module 801 is used to acquire a sample image and an initial image processing model. The sample image has a first annotation, and the initial image processing model includes a first processing model and a second processing model.

[0242] Extraction module 802 is used to extract image features of sample images based on the first processing model;

[0243] The processing module 803 is used to perform recognition processing on image features through the second processing model to obtain the recognition result of the sample image and the uncertainty parameter corresponding to the recognition result. The uncertainty parameter is used to indicate the degree of uncertainty of the recognition result.

[0244] The acquisition module 801 is also used to acquire a first loss function based on the recognition result, uncertainty parameter and first label;

[0245] Training module 804 is used to train a first processing model and a second processing model based on a first loss function to obtain a target image processing model.

[0246] In one possible implementation, the second processing model includes a first sub-model, which includes at least three fully connected layers.

[0247] In one possible implementation, the first sub-model includes a first fully connected layer, a second fully connected layer, and a third fully connected layer, and the recognition result includes a classification result; the processing module 803 is used to input image features into the first fully connected layer to obtain the fully connected features output by the first fully connected layer; input the fully connected features into the second fully connected layer to obtain the classification result of the sample image output by the second fully connected layer; and input the fully connected features into the third fully connected layer to obtain the classification uncertainty parameter output by the third fully connected layer corresponding to the classification result.

[0248] In one possible implementation, the identification result includes a classification result, the uncertainty parameter includes a classification uncertainty parameter, and the first label includes a classification label. The acquisition module 801 is further configured to acquire a first sub-loss function and a second sub-loss function based on the classification result, the classification uncertainty parameter, and the classification label; set a first weight and a second weight for the first sub-loss function and the second sub-loss function, respectively; and use the sum of the first product and the second product as the first loss function, wherein the first product is the product of the first sub-loss function and the first weight, and the second product is the product of the second sub-loss function and the second weight.

[0249] In one possible implementation, the second processing model includes a second sub-model, which includes multiple Monte Carlo networks, and the recognition result includes a segmentation result; the processing module 803 is used to input image features into multiple Monte Carlo networks to obtain multiple prediction results output by multiple Monte Carlo networks; based on the multiple prediction results, the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result are determined.

[0250] In one possible implementation, the second processing model includes a first sub-model and a second sub-model, and the recognition result includes a classification result and a segmentation result; the processing module 803 is used to classify the image features through the first sub-model to obtain the classification result of the sample image and the classification uncertainty parameter corresponding to the classification result; and to segment the image features through the second sub-model to obtain the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result.

[0251] In one possible implementation, the first labeling includes classification labeling and segmentation labeling, and the first loss function includes a second loss function and a third loss function; the acquisition module 801 is further configured to acquire the second loss function based on the classification result, classification uncertainty parameter and classification labeling; and acquire the third loss function based on the segmentation result, segmentation uncertainty parameter and segmentation labeling.

[0252] In one possible implementation, the initial image processing model further includes a third processing model, processing module 803, which is also used to stitch together image features and segmentation results to obtain stitched results; and to use the third processing model to jointly process the stitched results and classification results to obtain target parameters.

[0253] The acquisition module 801 is also used to acquire the fourth loss function corresponding to the target parameters based on the target parameters and classification labels;

[0254] Training module 804 is used to train the first processing model, the first sub-model, the second sub-model, and the third processing model based on the second loss function, the third loss function, and the fourth loss function to obtain the target image processing model.

[0255] In one possible implementation, the training module 804 is further configured to update the parameters of the first processing model according to the second loss function, the third loss function, and the fourth loss function; update the parameters of the first sub-model according to the second loss function and the fourth loss function; update the parameters of the second sub-model according to the third loss function and the fourth loss function; and update the parameters of the third processing model according to the fourth loss function.

[0256] In one possible implementation, the third processing model includes a third sub-model and a fourth sub-model. The processing module 803 is further configured to input the splicing result into the third sub-model to obtain the first sub-parameter output by the third sub-model; obtain the second sub-parameter from the classification result; and input the first sub-parameter and the second sub-parameter into the fourth sub-model to obtain the target parameter output by the fourth sub-model.

[0257] In one possible implementation, the processing module 803 is further configured to perform image processing on the test image using the target image processing model to obtain the processing result of the test image;

[0258] The acquisition module 801 is also used to acquire a first evaluation index and a second evaluation index of the target image processing model based on the processing results of the test image. The first evaluation index is used to evaluate the image processing effect of the target image processing model, and the second evaluation index is used to evaluate the image processing repeatability of the target image processing model.

[0259] In this embodiment, during model training, a loss function is obtained based on the recognition result of the sample image, the uncertainty parameter corresponding to the recognition result, and the annotation of the sample image. That is, the loss function considers not only the recognition result and the annotation, but also the degree of uncertainty of the recognition result. Training the image processing model using the loss function obtained in this way is beneficial to improving the performance of the image processing model. When using the trained image processing model to process the target image, in addition to obtaining the recognition result of the target image, an uncertainty parameter indicating the degree of uncertainty of the target recognition result can also be obtained, resulting in better image processing performance.

[0260] See Figure 9 This application provides an image processing apparatus, which includes:

[0261] The acquisition module 901 is used to acquire the target image to be processed;

[0262] The processing module 902 is used to process the target image based on the trained target image processing model to obtain the target recognition result of the target image and the target uncertainty parameter corresponding to the target recognition result. The target uncertainty parameter is used to indicate the degree of uncertainty of the target recognition result.

[0263] In this embodiment, during model training, a loss function is obtained based on the recognition result of the sample image, the uncertainty parameter corresponding to the recognition result, and the annotation of the sample image. That is, the loss function considers not only the recognition result and the annotation, but also the degree of uncertainty of the recognition result. Training the image processing model using the loss function obtained in this way is beneficial to improving the performance of the image processing model. When using the trained image processing model to process the target image, in addition to obtaining the recognition result of the target image, an uncertainty parameter indicating the degree of uncertainty of the target recognition result can also be obtained, resulting in better image processing performance.

[0264] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0265] Figure 10 This is a schematic diagram of a server structure provided in an embodiment of this application. The server can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1001 and one or more memories 1002. Each memory 1002 stores at least one line of program code, which is loaded and executed by the processors 1001 to implement the model training method or image processing method provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0266] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The device can be a terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0267] Typically, a terminal includes a processor 1101 and a memory 1102.

[0268] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0269] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one instruction, which is executed by the processor 1101 to implement the model training method or image processing method provided in the method embodiments of this application.

[0270] In some embodiments, the terminal may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1104, a touch display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1109.

[0271] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0272] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0273] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 1105 may be a single screen, disposed on the front panel of the terminal; in other embodiments, display screen 1105 may be at least two screens, disposed on different surfaces of the terminal or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of the terminal. Furthermore, display screen 1105 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 1105 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0274] The camera assembly 1106 is used to acquire images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0275] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 to achieve voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0276] Power supply 1109 is used to power the various components in the terminal. Power supply 1109 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1109 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0277] In some embodiments, the terminal further includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to: an acceleration sensor 1111, a gyroscope sensor 1112, a pressure sensor 1113, an optical sensor 1115, and a proximity sensor 1116.

[0278] Accelerometer 1111 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, accelerometer 1111 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1101 can control touchscreen 1105 to display the user interface in landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1111. Accelerometer 1111 can also be used for games or for acquiring user motion data.

[0279] The gyroscope sensor 1112 can detect the terminal's orientation and rotation angle. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal. Based on the data collected by the gyroscope sensor 1112, the processor 1101 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0280] The pressure sensor 1113 can be disposed on the side bezel of the terminal and / or on the lower layer of the touch display screen 1105. When the pressure sensor 1113 is disposed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the touch display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the touch display screen 1105. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0281] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the touch screen 1105 based on the ambient light intensity collected by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the touch screen 1105 is increased; when the ambient light intensity is low, the display brightness of the touch screen 1105 is decreased. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 based on the ambient light intensity collected by the optical sensor 1115.

[0282] The proximity sensor 1116, also known as a distance sensor, is typically installed on the front panel of the terminal. The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 1101 controls the touch display screen 1105 to switch from a screen-on state to a screen-off state; when the proximity sensor 1116 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 1101 controls the touch display screen 1105 to switch from a screen-off state to a screen-on state.

[0283] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0284] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory, the memory storing at least one line of program code. This at least one line of program code is loaded and executed by one or more processors to implement any of the model training methods or image processing methods described above.

[0285] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one piece of program code, which is loaded and executed by a processor of a computer device to implement any of the above-described model training methods or image processing methods.

[0286] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0287] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0288] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

Claims

1. A model training method, characterized in that, The method includes: Obtain sample images and an initial image processing model. The sample images have classification and segmentation labels, and the initial image processing model includes a first processing model, a first sub-model, a second sub-model, a third sub-model, and a fourth sub-model. Based on the first processing model, the image features of the sample image are extracted; The image features are classified using the first sub-model to obtain the classification result of the sample image and the classification uncertainty parameter corresponding to the classification result; the image features are segmented using the second sub-model to obtain the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result, wherein the segmentation result is used to indicate the segmentation region in the sample image corresponding to each category respectively; Based on the classification result, the classification uncertainty parameter, and the classification label, a second loss function is obtained; based on the segmentation result, the segmentation uncertainty parameter, and the segmentation label, a third loss function is obtained. The image features and the segmentation results are concatenated to obtain a concatenated result; the concatenated result is input into the third sub-model to obtain the first sub-parameter output by the third sub-model, the first sub-parameter including the size of the segmentation region corresponding to each category respectively; When the sample image is an image of the target type, the probability that the sample image in the classification result is an image of the target type is used as the second sub-parameter; the first sub-parameter and the second sub-parameter are input into the fourth sub-model to obtain the target parameter output by the fourth sub-model. The target parameter is the product of the index parameter and the second sub-parameter. The index parameter is a parameter determined based on the first sub-parameter to indicate the probability that the sample image is an image of the target type. Based on the difference between the target parameters and the probability that the sample image in the classification label is an image of the target type, a fourth loss function is obtained, which is used to measure the consistency between the classification result and the segmentation result; The parameters of the first processing model are updated according to the second loss function, the third loss function, and the fourth loss function; the parameters of the first sub-model are updated according to the second loss function and the fourth loss function; the parameters of the second sub-model are updated according to the third loss function and the fourth loss function; and the parameters of the third sub-model and the fourth sub-model are updated according to the fourth loss function to obtain the target image processing model.

2. The method according to claim 1, characterized in that, The first sub-model includes at least three fully connected layers.

3. The method according to claim 2, characterized in that, The first sub-model includes a first fully connected layer, a second fully connected layer, and a third fully connected layer; the step of classifying the image features using the first sub-model to obtain the classification result of the sample image and the classification uncertainty parameter corresponding to the classification result includes: The image features are input into the first fully connected layer to obtain the fully connected features output by the first fully connected layer; The fully connected features are input into the second fully connected layer to obtain the classification result of the sample image output by the second fully connected layer; The fully connected features are input into the third fully connected layer to obtain the classification uncertainty parameter output by the third fully connected layer, which corresponds to the classification result.

4. The method according to claim 1, characterized in that, The step of obtaining the second loss function based on the classification result, the classification uncertainty parameter, and the classification label includes: Based on the classification result, the classification uncertainty parameter, and the classification label, the first sub-loss function and the second sub-loss function are obtained respectively. Set a first weight and a second weight for the first sub-loss function and the second sub-loss function, respectively; The sum of the first product and the second product is used as the second loss function. The first product is the product of the first sub-loss function and the first weight, and the second product is the product of the second sub-loss function and the second weight.

5. The method according to claim 1, characterized in that, The second sub-model includes multiple Monte Carlo networks; the step of segmenting the image features using the second sub-model to obtain the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result includes: The image features are input into the multiple Monte Carlo networks to obtain multiple prediction results output by the multiple Monte Carlo networks; Based on the multiple prediction results, the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result are determined.

6. The method according to claim 1, characterized in that, After obtaining the target image processing model, the method further includes: The target image processing model is used to process the test image to obtain the processing result of the test image; Based on the processing results of the test image, a first evaluation index and a second evaluation index of the target image processing model are obtained. The first evaluation index is used to evaluate the image processing effect of the target image processing model, and the second evaluation index is used to evaluate the image processing repeatability of the target image processing model.

7. An image processing method, characterized in that, The method includes: Obtain the target image to be processed; The target image is processed by the target image processing model trained based on the method described in any one of claims 1-6 to obtain the target recognition result of the target image and the target uncertainty parameter corresponding to the target recognition result. The target uncertainty parameter is used to indicate the degree of uncertainty of the target recognition result.

8. A model training device, characterized in that, The device includes: The acquisition module is used to acquire sample images and an initial image processing model. The sample images have classification and segmentation labels, and the initial image processing model includes a first processing model, a first sub-model, a second sub-model, a third sub-model, and a fourth sub-model. The extraction module is used to extract image features from the sample image based on the first processing model; The processing module is used to classify the image features using the first sub-model to obtain the classification result of the sample image and the classification uncertainty parameter corresponding to the classification result; and to segment the image features using the second sub-model to obtain the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result, wherein the segmentation result is used to indicate the segmentation region in the sample image corresponding to each category respectively. The acquisition module is further configured to: acquire a second loss function based on the classification result, the classification uncertainty parameter, and the classification label; acquire a third loss function based on the segmentation result, the segmentation uncertainty parameter, and the segmentation label; concatenate the image features and the segmentation result to obtain a concatenated result; input the concatenated result into the third sub-model to obtain a first sub-parameter output by the third sub-model, the first sub-parameter including the size of the segmentation region corresponding to each category; when the sample image is an image of the target type, use the probability that the sample image in the classification result is an image of the target type as a second sub-parameter; input the first sub-parameter and the second sub-parameter into the fourth sub-model to obtain a target parameter output by the fourth sub-model, the target parameter being the product of an index parameter and the second sub-parameter, the index parameter being a parameter determined based on the first sub-parameter to indicate the probability that the sample image is an image of the target type; and acquire a fourth loss function based on the difference between the target parameter and the probability that the sample image in the classification label is an image of the target type, the fourth loss function being used to measure the consistency between the classification result and the segmentation result. The training module is used to update the parameters of the first processing model according to the second loss function, the third loss function, and the fourth loss function; update the parameters of the first sub-model according to the second loss function and the fourth loss function; update the parameters of the second sub-model according to the third loss function and the fourth loss function; and update the parameters of the third sub-model and the fourth sub-model according to the fourth loss function to obtain the target image processing model.

9. The apparatus according to claim 8, characterized in that, The first sub-model includes at least three fully connected layers.

10. The apparatus according to claim 9, characterized in that, The first sub-model includes a first fully connected layer, a second fully connected layer, and a third fully connected layer; the processing module is used to input the image features into the first fully connected layer to obtain the fully connected features output by the first fully connected layer; input the fully connected features into the second fully connected layer to obtain the classification result of the sample image output by the second fully connected layer; and input the fully connected features into the third fully connected layer to obtain the classification uncertainty parameter output by the third fully connected layer corresponding to the classification result.

11. The apparatus according to claim 8, characterized in that, The acquisition module is used to acquire a first sub-loss function and a second sub-loss function based on the classification result, the classification uncertainty parameter, and the classification label; set a first weight and a second weight for the first sub-loss function and the second sub-loss function respectively; and use the sum of the first product and the second product as the second loss function, wherein the first product is the product of the first sub-loss function and the first weight, and the second product is the product of the second sub-loss function and the second weight.

12. The apparatus according to claim 8, characterized in that, The second sub-model includes multiple Monte Carlo networks; the processing module is used to input the image features into the multiple Monte Carlo networks to obtain multiple prediction results output by the multiple Monte Carlo networks; based on the multiple prediction results, to determine the segmentation result of the sample image and the segmentation uncertainty parameter corresponding to the segmentation result.

13. The apparatus according to claim 8, characterized in that, The processing module is further configured to perform image processing on the test image using the target image processing model to obtain the processing result of the test image; The acquisition module is further configured to acquire a first evaluation index and a second evaluation index of the target image processing model based on the processing result of the test image. The first evaluation index is used to evaluate the image processing effect of the target image processing model, and the second evaluation index is used to evaluate the image processing repeatability of the target image processing model.

14. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the target image to be processed; The processing module is used to process the target image based on the target image processing model trained by any one of the methods described in claims 1-6, to obtain the target recognition result of the target image and the target uncertainty parameter corresponding to the target recognition result, wherein the target uncertainty parameter is used to indicate the degree of uncertainty of the target recognition result.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one piece of program code, which is loaded and executed by the processor to implement the model training method as described in any one of claims 1 to 6, or the image processing method as described in claim 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the model training method as described in any one of claims 1 to 6, or the image processing method as described in claim 7.

Images