Data processing method, device, program product, computer equipment and medium
By acquiring images of different feature regions and combining prediction and auxiliary network optimization, the accuracy problem of machine learning models under the influence of sample quality differences during training is solved, and efficient segmentation of image feature regions is achieved.
Patent Information
- Application Number
- CN202210466331.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-04-29
AI Technical Summary
Existing machine learning models require a large amount of sample data during training, and the sample quality varies greatly, resulting in insufficient model training accuracy. Especially in image segmentation tasks, how to effectively utilize high-quality and low-quality annotation data to improve model performance is A challenge.
By acquiring images with different feature areas, calling the prediction network for prediction, combining the auxiliary network to classify and predict image pixel points, and optimizing network parameters to improve the accuracy of the model's segmentation of image feature areas.
Effectively utilizing high-quality and low-quality annotation data improves the training accuracy of the model and the accuracy of image feature area segmentation, and solves the model performance problems under the influence of sample quality differences.
Smart Images

Figure CN115115828B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, program product, computer equipment and medium. Background Art
[0002] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.
[0003] In existing applications, a large amount of sample data is often required when performing machine learning on a model. However, this large amount of sample data often has differences in sample quality. Therefore, how to train the model more accurately through sample data of varying quality has become an urgent problem to be solved. Summary of the invention
[0004] The present application provides a data processing method, apparatus, program product, computer equipment and medium, which can improve the accuracy of a trained prediction network, and subsequently accurately segment feature areas in an image through the trained prediction network.
[0005] On the one hand, the present application provides a data processing method, the method comprising:
[0006] Acquire a first image having a first characteristic region and a second image having a second characteristic region;
[0007] Calling the prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel point of the first image belongs to a pixel point of the first feature area;
[0008] Calling the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel point of the second image belongs to a pixel point of the second feature area;
[0009] Performing classification prediction on the pixels of the second image through the auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image;
[0010] Network parameter optimization is performed based on the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information. The network parameter optimization includes any one or a combination of two of parameter optimization of the prediction network and parameter optimization of the auxiliary network.
[0011] On one hand, the present application provides a data processing device, the device comprising:
[0012] An acquisition module, used for acquiring a first image having a first characteristic region and a second image having a second characteristic region;
[0013] A first calling module is used to call the prediction network to predict the first image and obtain a first prediction result; the first prediction result includes: first predicted pixel information for determining whether each pixel point of the first image belongs to a pixel point of the first feature area;
[0014] A second calling module is used to call the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel point of the second image belongs to a pixel point of the second feature area;
[0015] A classification module, used to perform classification prediction on the pixels of the second image through the auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image;
[0016] The optimization module is used to optimize network parameters according to the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information, wherein the network parameter optimization includes any one or a combination of optimizing the parameters of the prediction network and optimizing the parameters of the auxiliary network.
[0017] In one aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes a method in one aspect of the present application.
[0018] In one aspect, the present application provides a computer-readable storage medium storing a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method in the above aspect.
[0019] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional modes such as the above-mentioned one aspect.
[0020] The present application can obtain a first image with a first feature area and a second image with a second feature area; call a prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel of the first image belongs to a pixel of the first feature area; call the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area; classify and predict the pixels of the second image through an auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image; according to the first prediction pixel information, the predicted pixel information of the pixels belonging to the first classification in the second prediction pixel information, and the predicted pixel information of the pixels belonging to the second classification in the second prediction pixel information, network parameter optimization is performed, and the network parameter optimization includes any one or a combination of parameter optimization of the prediction network and parameter optimization of the auxiliary network. It can be seen that the method proposed in the present application can classify the pixels in the second image through the auxiliary network (such as classification of correct or incorrect markings), and then the classification results of each pixel in the second image can be used by the auxiliary network to optimize the parameters of the prediction network or the auxiliary network, which can improve the accuracy of parameter optimization of the prediction network. Subsequently, the prediction network after parameter optimization can also accurately segment the feature areas in the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0022] Figure 1 It is a structural diagram of a network architecture provided by this application;
[0023] Figure 2 This is a schematic diagram of a model training scenario provided by this application;
[0024] Figure 3 It is a flowchart of a data processing method provided by this application;
[0025] Figure 4 It is a schematic diagram of a network training scenario provided by this application;
[0026] Figure 5 It is a flow chart of determining classification prediction results provided by the present application;
[0027] Figure 6 It is a schematic diagram of a scenario for determining a classification result provided by the present application;
[0028] Figure 7 It is a schematic diagram of a process for determining prediction differences provided by the present application;
[0029] Figure 8 This is a schematic diagram of a model training scenario provided by this application;
[0030] Fig. 9 It is a structural schematic diagram of a data processing device provided by the present application;
[0031] Fig.10 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0033] This application involves artificial intelligence related technologies. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0035] This application mainly involves machine learning in artificial intelligence. Among them, machine learning (ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines, specializing in how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.
[0036] The machine learning involved in this application mainly refers to how to train a prediction network, and then use the trained prediction network to accurately segment the feature area in the image. For details, please refer to the following Figure 3 Description in the corresponding embodiment.
[0037] See also Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided by this application. Figure 1 As shown, the network architecture may include a server 200 and a terminal device cluster, and the terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include a terminal device 100a, a terminal device 101a, a terminal device 102a, ..., a terminal device 103a; Figure 1 As shown, the terminal device 100a, the terminal device 101a, the terminal device 102a, ..., the terminal device 103a can all be connected to the server 200 through a network, so that each terminal device can exchange data with the server 200 through the network connection.
[0038] like Figure 1The server 200 shown can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smart terminal such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc. The following takes the communication between the terminal device 100a and the server 200 as an example to describe the specific embodiments of the present application.
[0039] See also Figure 2 , Figure 2 This is a schematic diagram of a model training scenario provided by this application. Figure 2 As shown, the server 200 can be used to train the student model, and the process can be: the server 200 can obtain sample data for model training, and the sample data can include a small number of samples annotated by experts and a large number of samples annotated by non-experts. Any sample data can be image data, and any image data can contain a number of pixels. Any pixel in any image data has a label information, and any sample data can contain a target object. The label information of any pixel is used to indicate whether the pixel belongs to the target object in the image. The image area where the target object is located in the image can be called a feature area. The server 200 can input the sample data annotated by experts and the sample data annotated by non-experts into the student model, and input the sample data annotated by non-experts into the teacher model.
[0040] Among them, the student model can generate a mask for each pixel point in the sample data annotated by an expert, and based on the mask, generate predicted pixel information for determining whether each pixel point in the sample data annotated by the expert belongs to the target object. The student model can also generate a mask for each pixel point in the sample data annotated by non-experts, and based on the mask, generate predicted pixel information for determining whether each pixel point in the sample data annotated by non-experts belongs to the target object.
[0041] Furthermore, the teacher model can generate a mask for each pixel in the sample data annotated by non-experts, and then obtain the feature distance between each pixel in the sample data annotated by non-experts and the target prototype / background prototype based on the mask. The target prototype can be used to represent the features of the target object in the sample data, and the background prototype can be used to represent the features of the background image of the target object in the sample data. Then, the teacher model can determine whether the marking information of each pixel in the sample data annotated by non-experts is correctly marked or incorrectly marked based on the feature distance and the marking information of each pixel in the sample data annotated by non-experts, and give the judgment result to the student model.
[0042] Next, the student model can generate a prediction deviation based on the judgment results of the teacher model (i.e., the correctly labeled pixels and incorrectly labeled pixels in the sample data labeled by non-experts) and the predicted pixel information for the sample data (including the predicted pixel information for each pixel in the sample data labeled by experts, and the predicted pixel information for each pixel in the sample data labeled by non-experts), and correct the network parameters of the student model based on the prediction deviation to obtain a trained student model.
[0043] Subsequently, the server 200 may use the trained student model to segment the target object in the image, and the server 200 may provide the segmentation result to the terminal device 100a, and the terminal device 100a may display the segmentation result on the terminal interface and provide it to relevant technical personnel for analysis.
[0044] Among them, the sample data annotated by the expert can be the following first image, the sample data annotated by the non-expert can be the following second image, the mask can be the following mask area, the student model can be the following prediction network, the teacher network can be the following auxiliary network, the predicted pixel information of the student model for each pixel in the sample data can be included in the following first predicted pixel information and second predicted pixel information, the target prototype can be the following target center feature, and the background prototype can be the following background center feature. Therefore, the specific process of how to train the student model through the teacher model can be referred to as follows Figure 3 and Figure 5 Corresponding to the description in the embodiment.
[0045] By adopting the method provided in the present application, the teacher model judges whether the marking information of each pixel in the sample data annotated by non-experts is correctly or incorrectly annotated based on the target prototype and background prototype of the target object, and then based on the judgment result, the student model is trained to distinguish the pixels in the sample data annotated by non-experts, and the student model is supervised and trained with the sample data annotated by experts to improve the training accuracy of the student model, so that an accurate student model can be trained.
[0046] See also Figure 3 , Figure 3 1 is a flowchart of a data processing method provided by the present application. The execution subject in the embodiment of the present application can be a computer device or a computer device cluster composed of multiple computer devices. The computer device can be a server or a terminal device. In the following, the execution subject in the embodiment of the present application is collectively referred to as a computer device for illustration. Figure 3 As shown, the method may include:
[0047] Step S101 , acquiring a first image having a first characteristic region and a second image having a second characteristic region.
[0048] Optionally, the computer device can obtain a first image and a second image. The number of the first image and the second image is determined according to an actual application scenario and is not limited thereto. The first image and the second image are sample data for training a prediction network.
[0049] Among them, both the first image and the second image may contain a target object, and the target object may be any object that needs to be segmented from the image data, and the target object may be determined according to the actual application scenario. The method provided in the present application can be applied to any image segmentation scenario, and the image segmentation scenario may be a two-dimensional segmentation scene or a three-dimensional segmentation scene. The display form of the target object in the first image (such as different object categories, or the same object categories but different postures, or different environments where the objects are located, etc.) and the display form of the target object in the second image may be different. For example, the target object may be the left ventricle, and both the first image and the second image may contain images of the left ventricle, but the image of the left ventricle contained in the first image and the image of the left ventricle contained in the second image may be different.
[0050] Among them, the image area where the target object in the image is located can be called a feature area. Further, the image area where the target object in the first image is located can be called a first feature area, and the image area where the target object in the second image is located can be called a second feature area.
[0051] The first image and the second image may both include a number of pixels.
[0052] For example, if the method provided in the present application is applied to a two-dimensional image segmentation scenario, the first image and the second image may be two-dimensional images, the pixels in the first image and the second image may be two-dimensional, and the target object may be any object that needs to be segmented from the two-dimensional image. The object can be specifically determined according to the actual application scenario. The target object may be an object whose local structural features are highly correlated or similar. For example, the target object may be a plant whose overall texture structural features are relatively similar in a two-dimensional image.
[0053] For another example, if the method provided in the present application is applied to a three-dimensional image segmentation scenario, the first image and the second image may be three-dimensional images, the pixels in the first image and the second image may be three-dimensional (in this case, the pixels in the first image and the second image may also be called voxels), and the target object may be any object that needs to be segmented from the three-dimensional image. The specific object may also be determined according to the actual application scenario. For example, the present application may be applied to a medical image segmentation scenario, and the target object may be an object whose local structural features are relatively related or similar. For example, the target object may be a human organ (which may be called a part) that needs to be segmented in the three-dimensional image data, and the organ (part) may be any organ such as the left ventricle.
[0054] Furthermore, the supervisory data set for the first image may be referred to as first supervisory data, and the first supervisory data is used to indicate whether each pixel in the first image belongs to a pixel in the first feature area. In other words, the first supervisory data is used to indicate whether each pixel in the first image belongs to a target object. The first supervisory data may include: labeling information of each pixel in the first image. The labeling information of each pixel in the first image is used to indicate whether each pixel belongs to the target object in the first image or to the background image of the target object in the first image. In other words, the labeling information of each pixel in the first image is used to indicate whether each pixel belongs to the first feature area or to a pixel in an area other than the first feature area in the first image (i.e., an area of the background image of the target object in the first image). The background image of the target object in the first image may also be referred to as the background image of the first feature area in the first image.
[0055] Similarly, the supervisory data set for the above-mentioned second image can be referred to as second supervisory data, and the second supervisory data is used to indicate whether each pixel in the second image belongs to a pixel in the second feature area. In other words, the second supervisory data is used to indicate whether each pixel in the second image belongs to the target object. The second supervisory data may include: labeling information of each pixel in the second image. Among them, the labeling information of each pixel in the second image is used to indicate whether each pixel belongs to the target object in the second image or to the background image of the target object in the second image. In other words, the labeling information of each pixel in the second image is used to indicate whether each pixel belongs to the pixel in the second feature area or to the pixel in the area other than the second feature area in the second image (i.e., the area of the background image of the target object in the second image). The background image of the target object in the second image can also be referred to as the background image of the second feature area in the second image.
[0056] In other words, the marking information of any pixel point is used to indicate the relationship between the pixel point and the target object in the image where it is located. The relationship can be a relationship in which the pixel point belongs to the target object (that is, the pixel point is a pixel point contained in the image of the target object in the image), or the relationship can be a relationship in which the pixel point does not belong to the target object (that is, the pixel point does not belong to the pixel point contained in the image of the target object in the image).
[0057] It should be noted that the first image may be sample data with high-quality annotations, and the second image may be sample data with low-quality annotations, which can be reflected in that the accuracy of the first supervisory data set for the first image (i.e., the accuracy of the labeling information of each pixel in the first image) is higher than the accuracy of the second supervisory data set for the second image (i.e., the accuracy of the labeling information of each pixel in the second image), and the accuracy is usually in a subjective sense. For example, the first image may be sample data labeled by an expert, i.e., the first supervisory data of the first image may be labeled by a professional in the technical field; and the second image may be sample data labeled by a non-expert, i.e., the second supervisory data of the second image may be labeled by a person in a non-technical field (such as an outsourced person or a netizen, etc.).
[0058] For example, if the method provided in the present application is applied to a medical segmentation scenario, the first image and the second image may be image data containing organs to be segmented, the marking information of the pixels in the first image may be marked by professionals in the medical field, and the marking information of the pixels in the second image may be marked by amateurs. Therefore, the accuracy of the marking information of the pixels in the first image is usually higher than that of the marking information of the pixels in the second image.
[0059] In addition, it should be noted that the cost of obtaining a large number of high-quality annotated samples (such as the first image) is very high, especially in the field of medical imaging that relies on expert knowledge. Therefore, in order to save the cost of obtaining samples, the first image in this application can be a small number, and the second image can be a large number. This application can effectively use a small amount of high-quality annotated data (such as the first image) and a large amount of low-quality annotated data (such as the second image) to accurately train a model (such as a prediction network).
[0060] Optionally, the marking information of a pixel point (such as the marking information of a pixel point in the first image or the marking information of a pixel point in the second image) can be recorded as 0 or 1. If the marking information of a pixel point is 0, it indicates that the pixel point does not belong to the target object in the image. Conversely, if the marking information of a pixel point is 1, it indicates that the pixel point belongs to the target object in the image.
[0061] For example, if the marking information of a pixel point in the first image is 0, it can indicate that the pixel point does not belong to the pixel point of the first feature area in the first image. Conversely, if the marking information of a pixel point in the first image is 1, it can indicate that the pixel point belongs to the pixel point of the first feature area in the first image.
[0062] For another example, if the marking information of a pixel point in the second image is 0, it may indicate that the pixel point does not belong to the pixel point of the second feature area in the second image. Conversely, if the marking information of a pixel point in the second image is 1, it may indicate that the pixel point belongs to the pixel point of the second feature area in the second image.
[0063] Step S102, calling the prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first predicted pixel information for determining whether each pixel of the first image belongs to a pixel of the first feature area.
[0064] Optionally, the computer device may call the prediction network to predict the first image, that is, predict the relationship between each pixel in the first image and the target object, and the prediction result predicted by the prediction network for each pixel in the first image may be referred to as the first prediction result. The first prediction result includes: first prediction pixel information for determining whether each pixel in the first image belongs to the first feature area, and the first prediction pixel information may include prediction pixel information for each pixel in the first image. The prediction pixel information of any pixel in the first prediction pixel information may include: the predicted probability that the pixel belongs to the target object in the first image (which may be referred to as the target probability), and the predicted probability that the pixel does not belong to the target object in the second image (that is, the background image belonging to the target object in the second image) (which may be referred to as the background probability), and the sum of the target probability and the background probability corresponding to any pixel in the first image may be 1.
[0065] Among them, the process of generating first predicted pixel information of the first image through the prediction network may include: the computer device may call the prediction network to generate a mask area of each pixel point in the first image in the first image, wherein the mask area of the pixel point may refer to an area used to select the main features of the pixel point.
[0066] Then, the computer device can call the prediction network to predict the target probability that each pixel in the first image belongs to the target object and the background probability that it does not belong to the target object according to the features of each pixel in the first image in the corresponding mask area, so as to obtain the predicted pixel information of each pixel in the first image. The predicted pixel information of any pixel in the first image includes the target probability that the pixel belongs to the target object (such as the probability that the image feature in the mask area of the pixel is the feature belonging to the target object) and the background probability that the pixel does not belong to the target object but belongs to the background image of the target object (such as the probability that the image feature in the mask area of the pixel is the feature belonging to the target object).
[0067] In summary, the predicted pixel information of each pixel point in the first image may constitute the first predicted pixel information.
[0068] Step S103, calling the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second predicted pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area.
[0069] Similarly, the computer device can call the prediction network to predict the second image, that is, predict the relationship between each pixel in the second image and the target object, and the prediction result predicted by the prediction network for each pixel in the second image can be called the second prediction result. The second prediction result includes: second prediction pixel information for determining whether each pixel in the second image belongs to the second feature area, and the second prediction pixel information may include prediction pixel information for each pixel in the second image. The prediction pixel information of any pixel in the second prediction pixel information may include: the predicted target probability that the pixel belongs to the target object in the second image, and the predicted background probability that the pixel does not belong to the target object in the second image (that is, the background image of the target object in the second image). The sum of the target probability and the background probability corresponding to any pixel in the second image can be 1.
[0070] Among them, the process of generating second predicted pixel information of the second image through the prediction network may include: the computer device may call the prediction network to generate a mask area of each pixel point in the second image in the second image, wherein the mask area of the pixel point may refer to the area used to select the main features of the pixel point.
[0071] Then, the computer device can call the prediction network to predict the target probability that each pixel in the second image belongs to the target object and the background probability that it does not belong to the target object according to the features of each pixel in the corresponding mask area in the second image, so as to obtain the predicted pixel information of each pixel in the second image. The predicted pixel information of any pixel in the second image includes the target probability that the pixel belongs to the target object (such as the probability that the image feature in the mask area of the pixel is the feature belonging to the target object) and the background probability that the pixel does not belong to the target object but belongs to the background image of the target object (such as the probability that the image feature in the mask area of the pixel is the feature belonging to the target object).
[0072] The predicted pixel information of each pixel point in the second image may constitute the second predicted pixel information.
[0073] Among them, the process of the above-mentioned prediction network predicting the predicted pixel information of the pixel points in the first image or the second image is the same as the process of the following auxiliary network predicting the predicted pixel information of the pixel points in the second image.
[0074] Step S104, classifying and predicting the pixels of the second image through the auxiliary network to obtain classification prediction results; the classification prediction results are used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image.
[0075] Optionally, in an embodiment of the present application, an auxiliary network can be used to determine which pixels in the second image are correctly labeled (i.e., have accurate labeling information) and which pixels are incorrectly labeled (i.e., have inaccurate labeling information), and then the prediction network can be trained to distinguish between the correctly labeled pixels and the incorrectly labeled pixels in the second image. For details, please refer to the following description.
[0076] Furthermore, the pixels with correct labels in the second image predicted by the auxiliary network can be referred to as pixels of the first classification, and the pixels of the first classification include the pixels with correct labels in the second image predicted by the auxiliary network. Similarly, the pixels with incorrect labels in the second image predicted by the auxiliary network can be referred to as pixels of the second classification, and the pixels of the second classification include the pixels with incorrect labels in the second image predicted by the auxiliary network.
[0077] The process of the auxiliary network predicting whether each pixel in the second image is correctly labeled or incorrectly labeled may include:
[0078] The computer device can call the auxiliary network to generate a regional center feature of the second feature area (which can be understood as an object center feature of the target object in the second image) and pixel features of each pixel point in the second image based on the second image.
[0079] Among them, the regional center feature may include a target center feature of the second feature region and a background center feature of the second feature region. The target center feature is used to characterize the structural features of the target object in the second image (that is, the structural features of the image in the second feature region in the second image, such as the texture structure, color structure and edge structure of the target object in the second image). In other words, the target center feature can be used to represent the features of the target object in the second image. The background center feature is used to characterize the structural features of the background image of the target object in the second image (that is, the structural features of the background image in the second feature region in the second image, such as the texture structure, color structure and edge structure of the background image of the target object in the second image). In other words, the background center feature can be used to represent the features of the background image of the target object in the second image.
[0080] Among them, the target center feature of the target object is obtained by integrating the pixel features of the pixel points belonging to the target object in the predicted second image, and the background center feature of the target object is obtained by integrating the pixel features of the pixel points of the background image belonging to the target object in the predicted second image; the pixel features of each pixel point in the second image generated by the auxiliary network can be the relatively accurate (can be determined based on experimental experience) intermediate features of each pixel point in the second image generated by the auxiliary network, and the pixel features of any pixel point in the second image generated by the auxiliary network can be used to represent the structural features of the pixel point. The specific generation process of the pixel features of each pixel point in the second image and the regional center features of the second feature area (including the target center features and the background center features) can also be found in the following Figure 5 Corresponding to the description in the embodiment.
[0081] Furthermore, the computer device can obtain the classification prediction results for the pixels in the second image through the above-generated regional center features, the pixel features of each pixel in the second image, and the marking information of each pixel in the second image. The classification prediction results are used to indicate the pixels on the second image that belong to the first classification (i.e., correctly marked pixels) and the pixels that belong to the second classification (i.e., incorrectly marked pixels).
[0082] The specific process of obtaining the classification prediction result for the pixel points in the second image by using the regional center feature, the pixel feature of each pixel point in the second image, and the label information of each pixel point in the second image can also be referred to as follows Figure 5 Corresponding to the description in the embodiment.
[0083] From the above, it can be known that the pixels in the second image can be divided into two categories through the auxiliary network, one is the pixels of the first category, and the other is the pixels of the second category. Among them, the pixels of the first category include the pixels with accurate label information (i.e., correct labels) in the second image predicted by the auxiliary network, and the pixels of the second category include the pixels without accurate label information (i.e., wrong labels) in the second image predicted by the auxiliary network.
[0084] In addition, it should be noted that the present application can continuously iteratively train the prediction network through a plurality of first images and a plurality of second images. During the iterative training of the prediction network, the network parameters of the auxiliary network will also be iteratively updated through the network parameters updated (i.e., optimized) by the prediction network. The specific principle of iteratively updating the network parameters of the auxiliary network through the network parameters after iterative update (i.e., iterative optimization) of the prediction network can be referred to the description in the following step S105.
[0085] Step S105, performing network parameter optimization based on the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information, wherein the network parameter optimization includes any one or a combination of performing parameter optimization on the prediction network and performing parameter optimization on the auxiliary network.
[0086] Optionally, the computer device can generate a final prediction deviation (which may be called a prediction difference) of the predicted pixel information of the prediction network for pixels (including pixels in the first image and pixels in the second image) based on the first predicted pixel information, the predicted pixel information of pixels belonging to the first category in the second predicted pixel information, and the predicted pixel information of pixels belonging to the second category in the second predicted pixel information. The prediction difference is used to represent the deviation between the predicted pixel information of the pixel predicted by the prediction network and the marking information of the pixel. The prediction difference can also be understood as the prediction loss of the prediction network.
[0087] The specific process of generating the prediction network's prediction difference of the predicted pixel information of the pixel point can be seen in the following Figure 8 Corresponding to the relevant description in the embodiment.
[0088] The computer device may perform network parameter optimization based on the above-mentioned prediction difference, and the network parameter optimization includes any one of optimizing parameters of the prediction network and optimizing parameters of the auxiliary network, or a combination of the two.
[0089] Among them, since the prediction network can be continuously iteratively updated (that is, the network parameters of the prediction network are iteratively optimized), each iterative update of the prediction network will generate a corresponding prediction difference through the above process, and the network parameters of the prediction network are continuously iteratively updated (that is, iteratively corrected or optimized) through the prediction differences generated in each iterative training process of the prediction network, until the network parameters of the prediction network are updated, a trained prediction network (which can be called a prediction network after parameter optimization) can be obtained, and the trained prediction network includes the network parameters that have been corrected (that is, optimized).
[0090] Optionally, completion of updating the network parameters of the prediction network may refer to updating the network parameters of the prediction network to a converged state, or to the number of iterative updates of the network parameters of the prediction network (i.e., the number of iterative trainings) reaching a certain number threshold, which may be set according to the actual application scenario.
[0091] See also Figure 4 , Figure 4 This is a schematic diagram of a network training scenario provided by this application. Figure 4 As shown, the first image includes multiple pixels, the first image has first supervisory data, and the first supervisory data includes labeling information of each pixel in the first image; similarly, the second image also includes multiple pixels, the second image has second supervisory data, and the second supervisory data includes labeling information of each pixel in the second image.
[0092] The computer device can call the auxiliary network to generate classification prediction results for each pixel in the second image. The classification prediction results include the classification results of each pixel in the second image. The classification result of any pixel in the second image indicates whether the labeling information of the pixel is correctly labeled or incorrectly labeled, that is, whether the pixel belongs to the first classification or the second classification.
[0093] Therefore, the correctly marked pixels in the second image can be used as pixels of the first classification, and the incorrectly marked pixels in the second image can be used as pixels of the second classification. Therefore, the prediction network can perform differentiated training on the pixels in the first image, the pixels of the first classification in the second image, and the pixels of the second classification in the second image, thereby obtaining a trained prediction network. Moreover, during the training of the prediction network, the network parameters of the prediction network will also be passed to the auxiliary network, so that during the training of the prediction network, the auxiliary network that continuously optimizes the parameters can also make more accurate judgments on the classification results of each pixel in the second image. By making more accurate judgments on the classification results of each pixel in the second image, more accurate training of the prediction network can also be achieved.
[0094] From the above, it can be known that in the process of iteratively optimizing the network parameters of the prediction network, the network parameters of the auxiliary network will also be iteratively updated through the network parameters of the prediction network. The prediction network can be understood as a student network (i.e., a student model), and the auxiliary network can be understood as a teacher network (i.e., a teacher model). This application can adopt a design similar to the mean teacher (MT) architecture to update the network parameters of the auxiliary network through the network parameters of the prediction network, because the weighted average self-integration strategy of MT can effectively improve the stability and smoothness of the intermediate feature representation and the final prediction, which is very suitable for the annotation separation strategy based on the feature prototype (the above target center feature can characterize the feature prototype of the target object) (i.e., distinguishing whether the labeling information of each pixel in the second image is a correct label or an incorrect label), because this can obtain a more stable and smooth feature space (such as a feature space composed of pixel features of each pixel in the second image), and the process can be shown in the following formula (1):
[0095]
[0096] Wherein, t and t-1 both represent the number of iterative training of the prediction network (which can also be understood as the number of iterative optimization of the network parameters of the auxiliary network), t represents the tth iterative training, and t-1 represents the t-1th iterative training. represents the network parameters of the auxiliary network after the tth iteration training, represents the network parameters of the auxiliary network after the t-1th iteration training, θ t Represents the network parameters of the prediction network after the tth iteration training.
[0097] Furthermore, α represents the EMA (exponential moving average) decay rate, and α can be set to 0.99. By adopting the exponential moving average decay rate, the prediction results of the features and the network can be smoothed in adjacent iterative training processes, which is beneficial to the generation of feature prototypes of the target object (such as target center features) to achieve robust annotation separation.
[0098] Through the above process, it can be understood that, in the process of iteratively updating the network parameters of the prediction network, the network parameters of the auxiliary network can also be iteratively updated through the network parameters after each iterative update of the prediction network, so that the features learned by the prediction network from the first image and the second image can be passed to the auxiliary network, and then the auxiliary network can also make a more accurate judgment on the classification results of the pixels in the second image during each iterative training process.
[0099] In the present application, the trained prediction network can be used to segment the target object in the image data. For example, the trained prediction network can identify the pixels in the image belonging to the target object, and the image area (i.e., the feature area) where the target object is located can be segmented from the image through the identified pixels belonging to the target object.
[0100] For example, a computer device may acquire a target image, the target image may include a target object, and an image region where the target object is located in the target image may be referred to as a target feature region. The target image may be any image that requires segmentation of the target object.
[0101] Therefore, the computer device can call the trained prediction network (i.e., the prediction network after parameter optimization) to predict the target image, that is, predict the relationship between each pixel in the target image and the target object, and obtain the target prediction result of the target image.
[0102] The target prediction result includes: target prediction pixel information of the pixel points used to determine whether each pixel point of the target image belongs to the target feature area, and the target prediction pixel information includes the prediction pixel information of each pixel point in the target image. The prediction pixel information of any pixel point in the target image may include the target probability that the pixel point belongs to the target object in the target image and the background probability that the pixel point does not belong to the target object in the target image, and the background probability is the probability that the pixel point belongs to the background image of the target object in the target image.
[0103] Therefore, the computer device can regard the pixel points in the target image whose corresponding target probability is greater than the background probability as the identified pixel points belonging to the target object (that is, the identified pixel points belonging to the target feature area). Therefore, the identified pixel points belonging to the target object can be segmented out from the target image, thereby realizing the segmentation of the target object in the target image, and also realizing the segmentation of the image of the target feature area in the target image.
[0104] Among them, in some scenarios, high-quality labeled data (such as the first image) is difficult to obtain and usually requires experts to label, while low-quality labeled data (such as the second image) is relatively easy to obtain. Therefore, the amount of high-quality labeled data is very small, while the amount of low-quality labeled data is very large. Therefore, the method provided in the present application can solve the problem of inaccurate model (network) training in such scenarios. The method provided in the present application can be used to separate and learn samples in a mixed sample consisting of a small amount of high-quality labeled data and a large amount of low-quality labeled data, thereby accurately learning the accurate features of the samples and training an accurate model (such as a prediction network).
[0105] In the present application, since the same type of segmented areas (i.e., the areas where objects of the target type are located) often have highly correlated structural features, the classification results of the pixels in the low-quality second image can be accurately determined by the feature prototype of the target object (such as the target center feature). Subsequently, the prediction network is differentiated and trained through the pixels of different classifications in the second image, and the prediction network is trained together with the high-quality first image as supervised training data. This can improve the training effect of the prediction network, and further train a more accurate prediction network. The accurate prediction network trained can also achieve accurate segmentation of the target object in the image.
[0106] The present application can obtain a first image with a first feature area and a second image with a second feature area; call a prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel of the first image belongs to a pixel of the first feature area; call the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area; classify and predict the pixels of the second image through an auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image; according to the first prediction pixel information, the predicted pixel information of the pixels belonging to the first classification in the second prediction pixel information, and the predicted pixel information of the pixels belonging to the second classification in the second prediction pixel information, network parameter optimization is performed, and the network parameter optimization includes any one or a combination of parameter optimization of the prediction network and parameter optimization of the auxiliary network. It can be seen that the method proposed in the present application can classify the pixels in the second image through the auxiliary network, and then the classification results of each pixel in the second image can be used to optimize the parameters of the prediction network or the auxiliary network, which can improve the accuracy of parameter optimization of the prediction network. Subsequently, the prediction network after parameter optimization can also accurately segment the feature areas in the image.
[0107] See also Figure 5 , Figure 5 is a flow chart of determining a classification prediction result provided by the present application. The execution subject in the embodiment of the present application can be the same as the above Figure 3 The execution subject is the same as Figure 5 As shown, the method may include:
[0108] Step S201, calling the auxiliary network to generate a regional center feature of a second feature region and a pixel feature of each pixel in the second image based on the second image.
[0109] Optionally, the computer device can input the second image into the auxiliary network to call the second image to perform feature learning on the second image, thereby generating pixel features for each pixel in the second image.
[0110] Optionally, the auxiliary network may include multiple convolutional layers for performing feature learning on the second image. Therefore, the pixel feature of each pixel may be the smoothed feature of each pixel generated by the penultimate convolutional layer among the multiple convolutional layers, because experiments have shown that the smoothed feature of the pixel generated by the penultimate convolutional layer is a feature with better effect.
[0111] Furthermore, the computer device can also call the auxiliary network to predict the mask area of each pixel in the second image, and the mask area is the area used to select the main features of each pixel in the second image. The computer device can also generate a prediction accuracy index of the mask area of each pixel in the second image, and the prediction accuracy index reflects the uncertainty of the mask area of each pixel in the generated second image. As the name implies, the prediction accuracy index of the mask area of any pixel in the second image represents the accuracy of the mask area of the pixel.
[0112] Optionally, the present application may perform Bayesian approximation through Monte Carlo dropout to generate a prediction accuracy index of the mask area of each pixel in the second image. The process may be:
[0113] The computer device can perform K random discards (i.e., dropout) on the network parameters of the auxiliary network (which can be called neurons) respectively, and K deformed networks of the auxiliary network can be obtained. K is a positive integer. The specific value of K can be determined according to the actual application scenario. Any discard of the network parameters is performed on the auxiliary network with complete network parameters. Any deformed network is obtained by randomly discarding the network parameters of the auxiliary network. Randomly discarding the network parameters of the auxiliary network can refer to randomly setting some network parameters of the auxiliary network to 0. The network parameters set to 0 are the discarded network parameters. The network parameters set to 0 do not play a role in the subsequent prediction process.
[0114] Among them, it can be understood that here, the network parameters of the auxiliary network are randomly discarded to obtain the deformation network of the auxiliary network, mainly for the subsequent generation of the prediction accuracy index of the mask area of the pixel point through the deformation network, and the pixel features of each pixel point and the mask area of each pixel point are all predicted by the auxiliary network without discarding the network parameters.
[0115] Among them, any pixel point in the second image can be represented as a target pixel point. Since the process of obtaining the prediction accuracy index of the mask area of each pixel point is the same, the process of obtaining the prediction accuracy index of the mask area of the target pixel point is used as an example for explanation here.
[0116] Therefore, the computer device can call each deformation network to predict the predicted pixel information for the target pixel point according to the mask area of the target pixel point (that is, obtain the predicted pixel information according to the image features at the mask area of the target pixel point in the second image), and the predicted pixel information of the pixel point predicted by the deformation network can be called the deformation predicted pixel information, and any deformation network can predict a deformation predicted pixel information for the target pixel point. The above process can be understood as performing K forward random inferences on the target pixel point, and letting K deformation networks perform K softmax (logistic regression) predictions on the target pixel point to obtain each deformation predicted pixel information of the target pixel point.
[0117] Among them, any deformation prediction pixel information may include the target probability that the target pixel predicted by the corresponding deformation network belongs to the pixel in the second feature area in the second image (which may be called the first prediction probability, that is, the target probability of belonging to the target object in the second image), and the background probability that the target pixel predicted by the corresponding deformation network does not belong to the pixel in the second feature area in the second image (which may be called the second prediction probability, that is, the probability of the pixel in the background image belonging to the target object in the second image). Among them, the sum of the first prediction probability and the second prediction probability may be 1. The background image of the target object in the second image refers to the image other than the image of the target object in the second image.
[0118] Therefore, the computer device can determine the prediction accuracy index of the mask area of the target pixel point according to the K deformation prediction pixel information obtained by the above K deformation networks. For details, please refer to the following description.
[0119] Since any deformed prediction pixel information includes a first prediction probability, the K deformed prediction pixel information contains a total of K first prediction probabilities. The computer device can obtain the standard deviation between the K first prediction probabilities and use the standard deviation as the target prediction accuracy index for the target pixel point. The target prediction accuracy index also indicates the accuracy of predicting the target pixel point as belonging to the target object.
[0120] Furthermore, any deformed predicted pixel information includes a second prediction probability. Therefore, K deformed predicted pixel information contain K second prediction probabilities in total. The computer device can obtain the standard deviation between the K second prediction probabilities and use the standard deviation as the background prediction accuracy index for the target pixel point. The background prediction accuracy index indicates the accuracy of predicting the target pixel point as the background image belonging to the target object.
[0121] Therefore, the target prediction accuracy index and the background prediction accuracy index for the target pixel can be used as the prediction accuracy index of the mask area of the target pixel. The computer device can obtain the prediction accuracy index of the mask area of each pixel in the second image in the same way as the prediction accuracy index of the mask area of the target pixel.
[0122] Furthermore, the computer device may also call the auxiliary network (the auxiliary network here refers to a network with complete network parameters, and the K deformation networks mentioned above may be obtained by randomly discarding the network parameters of the auxiliary network) to predict a third prediction result for the second image based on the mask area of each pixel in the generated second image. The third prediction result may include: third predicted pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area, and the third predicted pixel information includes predicted pixel information of each pixel in the second image predicted by the auxiliary network.
[0123] The predicted pixel information of any pixel in the third predicted pixel information includes the probability predicted by the auxiliary network that the pixel belongs to the pixel in the second feature area in the second image (which can be called the target probability), and the probability predicted by the auxiliary network that the pixel does not belong to the pixel in the second feature area in the second image (which can be called the background probability).
[0124] Therefore, the computer device can generate the regional center feature of the target object based on the pixel features of each pixel in the above-mentioned second image generated by the auxiliary network, the prediction accuracy index of the mask area of each pixel in the second image, and the third prediction result. The process is described as follows.
[0125] The computer device may obtain, from among the plurality of pixels included in the second image, a pixel whose prediction accuracy index of the corresponding mask area is greater than an index threshold (the index threshold may be set according to an actual application scenario), as an evaluation pixel, and the evaluation pixel refers to a pixel whose prediction accuracy index of the corresponding mask area in the second image is greater than the index threshold. The number of the evaluation pixel may be at least one.
[0126] Here, the target pixel is also used as an example to illustrate the evaluation pixel. Since the prediction accuracy index of the mask area corresponding to the target pixel includes the target prediction accuracy index and the background prediction accuracy index, the prediction accuracy index of the mask area of the target pixel is greater than the index threshold, which means that the target prediction accuracy index and the background prediction accuracy index of the mask area of the target pixel are both greater than the index threshold, that is, when the target prediction accuracy index of the mask area of the target pixel is greater than the index threshold, and the background prediction accuracy index is also greater than the index threshold, the target pixel can be used as an evaluation pixel. Through this principle, the computer device can obtain several pixels that can be used as evaluation pixels from the pixels contained in the second image, and obtain the above-mentioned at least one evaluation pixel.
[0127] Therefore, the computer device may generate a regional center feature of the target object according to the pixel feature of at least one evaluation pixel and the predicted pixel information belonging to the at least one evaluation pixel in the third predicted pixel information:
[0128] The computer device may use the predicted pixel information in the third predicted pixel information among at least one evaluation pixel point to indicate the evaluation pixel point of the pixel point belonging to the second feature area (i.e., the evaluation pixel point whose target probability is greater than the background probability in the predicted pixel information in the third predicted pixel information) as the target evaluation pixel point of the second feature area (i.e., the target evaluation pixel point of the target object), that is, the target evaluation pixel point is the pixel point in the second image predicted by the auxiliary network that belongs to the target object and whose prediction accuracy index of the corresponding mask area is greater than the index threshold.
[0129] Similarly, the computer device can also use the predicted pixel information in the third predicted pixel information among at least one evaluation pixel to indicate the evaluation pixel point that does not belong to the second feature area (that is, the evaluation pixel point whose target probability is less than the background probability in the predicted pixel information in the third predicted pixel information) as the background evaluation pixel point of the second feature area (that is, the background evaluation pixel point of the target object), that is, the background evaluation pixel point is the pixel point in the second image predicted by the auxiliary network that does not belong to the target object and whose prediction accuracy index of the corresponding mask area is greater than the index threshold.
[0130] Therefore, the computer device can generate a target center feature of the second feature area based on the pixel features of the target evaluation pixel point and the target probability (i.e., the probability of belonging to the target object) in the predicted pixel information of the target evaluation pixel point in the third predicted pixel information. The target center feature is used to represent the structural features of the target object in the second image, i.e., to represent the structural features of the image of the second feature area in the second image.
[0131] Among them, the target center feature can be recorded as q objTherefore, as shown in the following formula (2), the target center feature q obj Can be:
[0132]
[0133] It should be noted that, usually, the marking information of each pixel in the second image (0 or 1, 0 means it does not belong to the target object, 1 means it belongs to the target object) is represented by the same label vector, and the pixel features of each pixel in the second image generated in the auxiliary network can also be contained in the same feature matrix, and a row in the feature matrix can represent the pixel features of one pixel. Therefore, when the auxiliary network generates the target center feature, it can be generated based on the operation of the feature matrix and the label vector, and the dimension of the pixel features of each pixel generated by the auxiliary network is usually different from the dimension of the above-mentioned label vector. Therefore, when calculating the target center feature through the pixel features of the target evaluation pixel, the pixel features of the target evaluation pixel can be upsampled by linear interpolation (if the pixel is three-dimensional, the linear interpolation method can be trilinear interpolation) to increase the dimension of the pixel features of the target evaluation pixel to the same dimension as the above-mentioned label vector.
[0134] Therefore, A represents the total number of all target evaluation pixels, a is less than or equal to A, e a represents the pixel feature obtained after the dimension of the pixel feature of the a-th target evaluation pixel point generated by the auxiliary network is increased to the same dimension as the above label vector (i.e., the pixel feature of the a-th target evaluation pixel point after the dimension increase). The probability that the a-th target evaluation pixel in the predicted pixel information of the a-th target evaluation pixel in the third predicted pixel information belongs to the target object in the second image (i.e., the target probability) is represented. By introducing the target probability of each target evaluation pixel belonging to the target object when obtaining the target center feature, the different contributions of each target evaluation pixel to the target center feature can be reflected. That is, the greater the target probability of the target evaluation pixel, the greater the contribution weight of the target evaluation pixel to generating the target center feature.
[0135] Similarly, the computer device can generate a background center feature of the second feature area based on the pixel features of the background evaluation pixel and the background probability (i.e., the probability of not belonging to the target object) in the predicted pixel information of the background evaluation pixel in the third predicted pixel information, and the background center feature is used to represent the structural features of the background image of the target object in the second image, that is, to represent the structural features of the image in the second image other than the image of the second feature area.
[0136] Among them, the background center feature can be recorded as q bgTherefore, as shown in the following formula (3), the target center feature q bg Can be:
[0137]
[0138] Where B represents the total number of all background evaluation pixels, b is less than or equal to B. Similarly, e b represents the pixel feature obtained after the dimension of the pixel feature of the b-th background evaluation pixel point generated by the auxiliary network is increased to the same dimension as the above label vector (i.e., the pixel feature of the b-th background evaluation pixel point after the dimension increase), The probability that the bth background evaluation pixel in the predicted pixel information of the bth background evaluation pixel in the third predicted pixel information belongs to the background image of the target object in the second image (i.e., the background probability) is represented. By introducing the background probability of each background evaluation pixel belonging to the background image of the target object when obtaining the background center feature, the different contributions of each background evaluation pixel to the background center feature can be reflected. That is, the greater the background probability of the background evaluation pixel, the greater the contribution weight of the background evaluation pixel to generating the background center feature.
[0139] Therefore, the computer device can use the above-mentioned target center feature (which can be understood as the target center feature of the target object in the second image) and the background center feature (which can be understood as the background center feature of the target object in the second image) as the regional center feature of the second feature area, and the regional center feature can be understood as the object center feature of the target object.
[0140] Step S202, determining a classification prediction result based on the regional center feature, the pixel feature of each pixel in the second image and the second supervisory data.
[0141] Optionally, the computer device can obtain the classification result of each pixel in the second image based on the regional center features of the target object generated above, the pixel features of each pixel in the second image, and the marking information of each pixel in the second image (that is, the marking information of each pixel in the second supervisory data). The classification result of any pixel in the second image can be the result that the pixel belongs to the first classification (that is, the marking information of the pixel is correctly marked) or the result that the pixel belongs to the second classification (that is, the marking information of the pixel is incorrectly marked).
[0142] Since the process of determining the classification result of each pixel in the second image is the same, the classification result of determining the target pixel (any pixel in the second image) is taken as an example for description.
[0143] The computer device can obtain the characteristic distance between the pixel feature of the target pixel point and the target center feature, which can be called the first characteristic distance; the computer device can also obtain the characteristic distance between the pixel feature of the target pixel point and the background center feature, which can be called the second characteristic distance.
[0144] Among them, the first characteristic distance can be recorded as The second characteristic distance is recorded as Therefore, as shown in the following formula, the first characteristic distance and the second characteristic distance It can be:
[0145]
[0146]
[0147] Among them, similar to the above, e m It can also represent the pixel features obtained after the dimension of the pixel features of the target pixel generated by the auxiliary network is increased to the same dimension as the above label vector (i.e., the pixel features of the target pixel after the dimension is increased). obj represents the above target center feature, q bg represents the background center feature mentioned above. ‖·‖2 represents the second norm.
[0148] Therefore, if the first feature distance is greater than the second feature distance (indicating that the target pixel is more inclined to belong to the background image of the target object in the second image), and the marking information of the target pixel in the second supervision data is used to indicate that the target pixel belongs to the target object in the second image (that is, the pixel belongs to the second feature area), then it can be determined that the classification result of the target pixel is a result used to indicate that the target pixel belongs to the second classification.
[0149] If the first feature distance is greater than the second feature distance (indicating that the target pixel is more inclined to belong to the background image of the target object in the second image), and the marking information of the target pixel in the second supervision data is used to indicate that the target pixel does not belong to the target object in the second image (that is, it does not belong to the pixel in the second feature area, that is, it belongs to the background image of the target object in the second image), then it can be determined that the classification result of the target pixel is a result indicating that the target pixel is a pixel belonging to the first classification.
[0150] If the first feature distance is smaller than the second feature distance (indicating that the target pixel is more likely to belong to the target object in the second image), and the marking information of the target pixel in the second supervisory data is used to indicate that the target pixel belongs to the target object in the second image, then it can be determined that the classification result of the target pixel is a result indicating that the target pixel belongs to the first classification.
[0151] If the first feature distance is smaller than the second feature distance (indicating that the target pixel is more inclined to belong to the target object in the second image), and the marking information of the target pixel in the second supervisory data is used to indicate that the target pixel does not belong to the target object in the second image (i.e., the background image belonging to the target object in the second image), then it can be determined that the classification result of the target pixel is a result indicating that the target pixel is a pixel belonging to the second classification.
[0152] In summary, it can be understood that if the feature type that the pixel feature of the target pixel tends to tend to (such as the feature type of the target object or the feature type of the background image of the target object) is inconsistent with the feature type indicated by the marking information of the target pixel (such as the feature type of the target object or the feature type of the background image of the target object) (such as one is the feature type of the target object and the other is the feature type of the background image of the target object), then the marking information of the target pixel can be considered to be incorrectly marked, that is, the classification result of the target pixel is used to indicate that the target pixel belongs to the second category of pixels in the second image; conversely, if the feature type that the pixel feature of the target pixel tends to tend to is consistent with the feature type indicated by the marking information of the target pixel (such as both are the feature types of the target object, or both are the feature types of the background image of the target object), then the marking information of the target pixel can be considered to be correctly marked, that is, the classification result of the target pixel is used to indicate that the target pixel belongs to the first category of pixels in the second image.
[0153] The classification prediction result for the second image can be obtained through the classification result of each pixel point in the second image, and the classification prediction result includes the classification result of each pixel point in the second image.
[0154] See also Figure 6 , Figure 6 This is a schematic diagram of a scenario for determining a classification result provided by this application. Figure 6As shown, the pixels in the second image may include pixel 1 to pixel W, where W is a positive integer and the specific value of W is determined according to the actual application scenario. The computer device may obtain the characteristic distance between the pixel features of each pixel in the second image and the target center feature, including the characteristic distance between the pixel features of pixel 1 and the target center feature (i.e., the first characteristic distance 1), the characteristic distance between the pixel features of pixel 2 and the target center feature (i.e., the first characteristic distance 2), the characteristic distance between the pixel features of pixel 3 and the target center feature (i.e., the first characteristic distance 3), ... and the characteristic distance between the pixel features of pixel W and the target center feature (i.e., the first characteristic distance W), and may obtain the characteristic distance between each pixel and the background center feature, including the characteristic distance between the pixel features of pixel 1 and the background center feature (i.e., the second characteristic distance 1), the characteristic distance between the pixel features of pixel 2 and the background center feature (i.e., the second characteristic distance 2), the characteristic distance between the pixel features of pixel 3 and the background center feature (i.e., the second characteristic distance 3), ... and the characteristic distance between the pixel features of pixel W and the background center feature (i.e., the second characteristic distance W).
[0155] Therefore, the computer device can obtain the classification result of pixel 1 based on the first feature distance 1, marking information and second feature distance 1 of pixel 1, and can obtain the classification result of pixel 2 based on the first feature distance 2, marking information and second feature distance 2 of pixel 2, and can obtain the classification result of pixel 3 based on the first feature distance 3, marking information and second feature distance 3 of pixel 3,… and can obtain the classification result of pixel W based on the first feature distance W, marking information and second feature distance W of pixel W.
[0156] Through the above process, for scenarios where the structural features between local areas in the segmented area (such as areas belonging to each pixel point in the image area where the target object is located) are highly correlated (such as highly similar) and the noise tolerance is high, the present application can accurately judge the classification results of each pixel point in the second image through the regional center feature.
[0157] This application focuses on label separation by using the feature prototype (which can be reflected by the regional center feature) to train more robust noise labels with the assistance of the mean teacher model (i.e. the above-mentioned auxiliary network). In the model framework of this application, networks such as V-Net (an image segmentation network), U-Net (a semantic segmentation network), DenseNet (a densely connected network), or ResNet (a residual network) can be used for training and prediction.
[0158] See also Figure 7 , Figure 7is a schematic diagram of a process for determining a prediction difference provided by the present application. The execution subject in the embodiment of the present application can be the same as the above Figure 3 The execution subject is the same as Figure 7 As shown, the method may include:
[0159] Step S301, generating a first prediction difference of a prediction network according to first prediction pixel information and first supervision data.
[0160] Optionally, the computer device can generate a cross entropy loss and an image segmentation loss (Dice loss) of the prediction network for the first image based on the predicted pixel information of each pixel of the first image in the first predicted pixel information and the labeling information of each pixel of the first image in the first supervisory data, and then obtain the prediction loss of the prediction network for the first image through the cross entropy loss and the image segmentation loss. The prediction loss can be referred to as the first prediction difference.
[0161] The cross entropy loss of the prediction network for the first image can be recorded as L s1 , as shown in the following formula (6), the cross entropy loss L s1 for:
[0162]
[0163] Among them, y true.i Represents the label information of the i-th pixel in the first image, that is, y true.i Indicates the true label of the i-th pixel, i is less than or equal to N, and N can be the total number of all pixels in the first image. If the label information of the i-th pixel is used to truly indicate that the i-th pixel belongs to the target object in the first image, then y true.i can be equal to 1, otherwise, that is, if the label information of the i-th pixel is used to truly indicate that the i-th pixel does not belong to the target object in the first image, then y true.i Can be equal to 0. pred.i It represents the probability that the i-th pixel in the predicted pixel information of the i-th pixel in the first predicted pixel information predicted by the prediction network belongs to the target object (ie, the target probability).
[0164] The image segmentation loss of the prediction network for the first image can be recorded as L Dice1 , as shown in the following formula (7), the image segmentation loss L Dice1 for:
[0165]
[0166] Same as above, here y true.i Indicates the label information of the i-th pixel in the first image, y true.i is 1 or 0, ypred.i Indicates the probability that the i-th pixel point in the predicted pixel information of the i-th pixel point in the first predicted pixel information belongs to the target object (ie, the target probability).
[0167] Therefore, the first prediction difference of the above prediction network can be recorded as L HQ , as shown in the following formula (8), the first prediction difference L HQ It is the cross entropy loss L s1 and image segmentation loss L Dice1 sum:
[0168] L HQ =L s1 +L Dice1 (8)
[0169] Among them, since the pixels in the first image have accurate label information, the first prediction difference L of the first image is obtained by the prediction network. HQ It can play the role of positive supervision training for the prediction network.
[0170] Step S302: generating a second prediction difference of the prediction network according to the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information and the first label information in the second supervisory data.
[0171] Optionally, the marking information of the pixel points of the first classification in the second supervisory data can be referred to as the first marking information, that is, the first marking information includes: the marking information of the pixel points of the second supervisory data that is preset to mark whether the pixel points belonging to the first classification on the second image belong to the second feature area. The computer device can generate the prediction loss of the prediction network for the pixel points of the first classification based on the predicted pixel information of the pixel points belonging to the first classification in the second predicted pixel information and the first marking information, and the prediction loss can be referred to as the second prediction difference.
[0172] Specifically, the computer device can generate the cross entropy loss and image segmentation loss (Dice loss) of the prediction network for the pixel points of the first category based on the predicted pixel information of each pixel point of the first category in the second predicted pixel information and the labeling information of each pixel point of the first category, and then obtain the second prediction difference of the prediction network for the pixel points of the first category through the cross entropy loss and the image segmentation loss.
[0173] The cross entropy loss of the prediction network for the first classification of pixels can be recorded as L s2 , as shown in the following formula (9), the cross entropy loss L s2 for:
[0174]
[0175] Among them, y true.j Represents the label information of the jth pixel in the first category, that is, y true.j represents the true label of the jth pixel, j is less than or equal to M, and M can be the total number of all pixels belonging to the first category. If the label information of the jth pixel is used to truly represent that the jth pixel belongs to the target object in the second image, then y true.j can be equal to 1, otherwise, that is, if the label information of the j-th pixel is used to truly indicate that the j-th pixel does not belong to the target object in the second image, then y true.j Can be equal to 0. pred.j Represents the probability that the j-th pixel point in the predicted pixel information of the j-th pixel point in the second predicted pixel information belongs to the target object (ie, the target probability).
[0176] The image segmentation loss of the prediction network for the first classification of pixels can be recorded as L Dice2 , as shown in the following formula (10), the image segmentation loss L Dice2 for:
[0177]
[0178] Same as above, here y true.j Indicates the label information of the jth pixel, y true.j is 1 or 0, y pred.j Represents the probability that the j-th pixel point in the predicted pixel information of the j-th pixel point in the second predicted pixel information belongs to the target object (ie, the target probability).
[0179] Therefore, the second prediction difference of the above prediction network can be recorded as L ls , as shown in the following formula (11), the second prediction difference L ls It is the cross entropy loss L s2 and image segmentation loss L Dice2 sum:
[0180] L ls =L s2 +L Dice2 (11)
[0181] The second prediction difference L obtained above ls It is the prediction loss of the prediction network for the correctly labeled pixels (i.e. the pixels of the first category).
[0182] Step S303: Generate a third prediction difference of the prediction network according to the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information and the second label information in the second supervisory data.
[0183] Optionally, the marking information of the pixel points of the second classification in the second supervisory data can be referred to as the second marking information, that is, the second marking information includes: the marking information of the pixel points of the second classification on the second image that is used to mark whether the pixel points belong to the second feature area in the pre-set second supervisory data. The computer device can generate the prediction loss of the prediction network for the pixel points of the second classification based on the predicted pixel information of the pixel points belonging to the second classification in the second predicted pixel information and the second marking information, and the prediction loss can be referred to as the third prediction difference.
[0184] In the present application, since the pixels of the second category are predicted incorrectly labeled pixels, the third prediction difference of the prediction network for the pixels of the second category can be obtained by minimizing the entropy loss, that is, the third prediction difference with smaller influence (smaller entropy) can be used to train the pixels of the second category for the prediction network.
[0185] The third prediction difference can be recorded as L ent , as shown in the following formula (12), the third prediction difference L ent Can be:
[0186]
[0187] Among them, F obj.g represents the probability (i.e., target probability) that the g-th pixel in the second classification belongs to the target object in the second image in the predicted pixel information in the second predicted pixel information, F bg.g represents the probability (i.e., background probability) that the g-th pixel point in the predicted pixel information in the second predicted pixel information does not belong to the target object in the second image, G is the total number of all pixels in the second classification, and g is less than or equal to G.
[0188] It can be seen from the above that the second prediction difference and the third prediction difference are calculated in different ways, thereby achieving the purpose of distinguishing the pixels of the first category and the pixels of the second category in the second image.
[0189] Step S304: generating a prediction difference of the prediction network according to the first prediction difference, the second prediction difference and the third prediction difference.
[0190] Optionally, the computer device can generate the final prediction loss of the prediction network based on the first prediction difference, the second prediction difference and the third prediction difference obtained above, and the prediction loss is also the prediction deviation of the prediction network (that is, the final prediction difference of the prediction network). The prediction difference refers to the deviation of the prediction network's predicted pixel information for pixels (including pixels of the first image and pixels of the second image).
[0191] Among them, the computer device can obtain the final prediction loss of the prediction network for the second image based on the second prediction difference of the prediction network for the pixel points of the first category and the third prediction difference of the prediction network for the pixel points of the second category obtained above. This prediction loss can be called the comprehensive prediction difference of the prediction network for the second image.
[0192] The comprehensive prediction difference can be recorded as L LQ , as shown in the following formula (13), the comprehensive prediction difference L LQ The second prediction difference L ls and the third prediction difference L ent sum:
[0193] L LQ =L ls +L ent (13)
[0194] Furthermore, the computer device also obtains a weighting coefficient for the comprehensive prediction difference, and can then weight the comprehensive prediction difference according to the weighting coefficient to obtain a weighted comprehensive prediction difference. The computer device can then generate the final prediction deviation (i.e., the final prediction loss value) of the prediction network based on the first prediction difference and the weighted comprehensive prediction difference.
[0195] Among them, the weight coefficient of the comprehensive prediction difference can be composed of a rising L-S function with increasing training time (number of times). Since the prediction network can be trained for multiple iterations, the weight coefficient of the comprehensive prediction difference in the process of the t-th iteration training of the prediction network can be recorded as λ(t), as shown in the following formula (14). The weight coefficient λ(t) can be:
[0196]
[0197] Among them, t max It represents the preset maximum number of iterative training for the prediction network, which can be called the maximum number of iterations, and e represents a natural constant.
[0198] For example, in the present application, the method for obtaining the weighting coefficient for the comprehensive prediction difference in the current iterative training process of the prediction network can be: the computer can obtain the number of iterations (which can be referred to as the current number of iterations) of the iterative correction (i.e., the current iterative training) of the network parameters of the prediction network, and can obtain the preset maximum number of iterations for iterative correction of the network parameters of the prediction network. Then, the computer device can substitute the current number of iterations into t in the above formula (14), and substitute the maximum number of iterations into t in the above formula (14). max , the weighted coefficient for the comprehensive prediction difference in the current iterative training process can be obtained.
[0199] Therefore, the prediction loss (i.e., prediction difference) of the prediction network can be recorded as L z , as shown in the following formula (15), the predicted difference L z Can be:
[0200] L z =L HQ +λ(t)L LQ (15)
[0201] It can be understood that each iterative training process of the prediction network will have a corresponding prediction loss. If the current training is the tth iterative training, then L in formula (14) HQ is the first prediction difference during the t-th iteration of training, L LQ It is the comprehensive prediction difference during the t-th iteration training process, and the obtained L z That is, the prediction loss during the t-th iteration of training. The prediction difference L obtained during each training process can be z The network parameters of the prediction network are iteratively optimized to obtain the prediction network with the final parameters optimized (i.e., the trained prediction network).
[0202] Through the above process, it can be understood that the smaller the number of iterations t, the smaller the weighting coefficient of the comprehensive prediction difference will be. Conversely, the larger the number of iterations t, the larger the weighting coefficient of the comprehensive prediction difference will be. This is to reduce the interference of the second image on the training of the prediction network when training the prediction network at the beginning (such as when the number of iterations t is relatively small). As the number of iterations t increases, the prediction network becomes more and more accurate. Therefore, a larger weighting coefficient can be used to increase the training effect of the second image on the prediction network, which can improve the training accuracy of the prediction network.
[0203] See also Figure 8 , Figure 8 This is a schematic diagram of a model training scenario provided by this application. Figure 8 As shown, the computer device can call the prediction network to generate a first prediction difference based on the first image. The computer device can also call the auxiliary network to label and separate the pixels in the second image, that is, to distinguish the pixels in the second image into pixels of the first category and pixels of the second category.
[0204] Furthermore, the computer device may call the prediction network to generate a second prediction difference based on the pixels of the first classification, and generate a third prediction difference based on the pixels of the second classification. The computer device may generate a comprehensive prediction difference for the second image based on the second prediction difference and the third prediction difference, and may weight the comprehensive prediction difference based on the weighting coefficient to obtain a weighted comprehensive prediction difference.
[0205] Finally, the computer device can obtain the final prediction loss of the prediction network (i.e., the prediction deviation or prediction difference) based on the above-mentioned first prediction difference and the weighted comprehensive prediction difference, and the prediction network can correct the network parameters according to the prediction deviation to obtain a trained prediction network (i.e., the prediction network after parameter optimization).
[0206] In the present application, a first image with high-quality labeling information can be used to supervise the training of the prediction network. For a second image with low-quality labeling information, both the predicted correctly labeled pixels (i.e., pixels of the first classification) and the predicted incorrectly labeled pixels (i.e., pixels of the second classification) can participate in the training of the prediction network. However, the correctly labeled pixels can produce a greater training effect on the prediction network, while the incorrectly labeled pixels can produce a smaller training effect on the prediction network. This allows full utilization of the second image to train the prediction network. Therefore, a very accurate prediction network can be trained.
[0207] Furthermore, the present application can be used to perform differentiated learning on sample data of mixed quality (including the first image and the second image), that is, to achieve labeled isolation learning of sample data of mixed quality, so as to fully learn the correct features of the sample data, and then train an accurate prediction network.
[0208] In addition, the present application also conducts precise experiments on the provided method. The experiment uses a left atrium (LA) segmentation dataset for the experiment. The left atrium segmentation dataset provides 100 3D magnetic resonance images (which can be understood as the three-dimensional first image) with expert labels (which can be understood as the labeling information of the pixels in the first image). The resolution of the image can be 0.625*0.625*0.625m^3. All images are cropped to the center of the heart area and normalized to zero mean and unit variance. In order to simulate actual scenarios, this application studies extreme settings and common soft settings.
[0209] Among them, in the extreme setting, only 2 samples (the minimum number of HQ label batch size in the code implementation) are used as HQ label information (i.e., samples with high-quality label information, which can be understood as the first image), while the soft setting uses 8 (10%) samples as HQ label information. The remaining samples are regarded as non-expert low-quality labeled data (i.e., samples with low-quality label information, which can be understood as the second image). These samples are processed by the commonly used simulation label damage scheme, including random corrosion and expansion of 3-15 voxels.
[0210] The experimental framework uses an NVIDIA GeForce RTX (graphics processing platform) 3090 GPU (graphics processing unit) with 24GB (computer storage unit) memory, implemented in Python (a computer programming language) and PyTorch (an open source machine learning library). In all experiments, the same 3D V-Net (a 3D (3-dimensional) medical image segmentation network designed based on a fully convolutional neural network) is used as the backbone for fair comparison. In addition, the network is trained using SGD (an optimizer) (weight decay = 0.0001, momentum = 0.9). The batch size is set to 4, including 2 high-quality annotated images (which can be understood as the first image) and 2 low-quality annotated images (which can be understood as the second image). The maximum training steps are all set to 8000. The learning rate is initialized to 0.01 and decays by a power of 0.9 after each step. This application randomly crops 112×112×80 voxel blocks as network input, applies standard data augmentation including random cropping, flipping and rotation, and uses a sliding window strategy with a step size of 18×18×4 voxels for the testing phase.
[0211] On this basis, this application uses four indicators for comprehensive evaluation, including: Dice (an image segmentation evaluation indicator), Jaccard (a data mining indicator), ASD (average surface distance), 95HD (a medical image segmentation evaluation indicator). The experimental data of this application in the medical scene are shown in Table 1 below:
[0212] Table 1
[0213]
[0214]
[0215] Among them, the present application conducted two experiments, rows 2 to 9 in Table 1 are experimental data of one experiment, and rows 10 to 17 in Table 1 are experimental data of another experiment. Set-HQ represents the number of high-quality labeled data used for training, and Set-LQ represents the number of low-quality labeled data used for training. "HQ-LQ separation?" indicates whether the corresponding method performs separate training on low-quality labeled data and high-quality labeled data. The higher the Dice and Jaccard, the better the effect, while the smaller the ASD and 95HD, the better the effect. The values in brackets in Table 1 represent the standard deviation of the indicators under the corresponding method.
[0216] H-Sup means supervised training with only high-quality labeled data, HL-Sup means supervised training with mixed high-quality and low-quality labeled data, TriNet means using a joint learning framework consisting of three networks, which supervises the third network by integrating predictions from two networks, 2RnT means a two-stage method to improve the labeling quality by estimating the confusion matrix for label correction, PNL means introducing an image-level label quality assessment module to identify images with clean labels to adjust the network, KDEM means introducing knowledge distillation technology and entropy minimization optimization terms to train the network, and Decoupled means using two separate decoders (one corresponding to high-quality labeled data and one corresponding to low-quality labeled data) to implicitly decouple the network.
[0217] It can be seen from the above experimental data that for the two experiments conducted above, the method provided in this application has the best comprehensive training effect, which also proves the superiority and robustness of the method provided in this application.
[0218] See also Fig. 9 , Fig. 9 is a structural diagram of a data processing device provided by the present application. The data processing device may be a computer program (including program code) running in a computer device. For example, the data processing device is an application software. The data processing device may be used to execute the corresponding steps of the method provided in the embodiment of the present application. Fig. 9 As shown, the data processing device 1 may include: an acquisition module 11, a first calling module 12, a second calling module 13, a classification module 14 and an optimization module 15.
[0219] An acquisition module 11, used for acquiring a first image having a first characteristic region and a second image having a second characteristic region;
[0220] The first calling module 12 is used to call the prediction network to predict the first image and obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel point of the first image belongs to a pixel point of the first feature area;
[0221] The second calling module 13 is used to call the prediction network to predict the second image and obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel point of the second image belongs to a pixel point of the second feature area;
[0222] The classification module 14 is used to perform classification prediction on the pixels of the second image through the auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image;
[0223] The optimization module 15 is used to optimize network parameters according to the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information, wherein the network parameter optimization includes any one or a combination of optimizing the parameters of the prediction network and optimizing the parameters of the auxiliary network.
[0224] Optionally, first supervision data is set for the first image, and second supervision data is set for the second image;
[0225] When optimizing the network parameters, the parameters of the prediction network are optimized by using a first prediction difference between the first predicted pixel information and the first supervisory data, a second prediction difference between the predicted pixel information of the pixel points belonging to the first classification in the second predicted pixel information and the first label information in the second supervisory data, and a third prediction difference between the predicted pixel information of the pixel points belonging to the second classification in the second predicted pixel information and the second label information in the second supervisory data;
[0226] The first marking information includes: information in the preset second supervisory data used to mark whether the pixel points belonging to the first classification on the second image belong to the second feature area;
[0227] The second marking information includes: information in the preset second supervisory data used to mark whether the pixel points belonging to the second classification on the second image belong to the second feature area;
[0228] The second prediction difference and the third prediction difference are calculated differently.
[0229] Optionally, the process of optimizing the parameters of the prediction network by the optimization module 15 through the first prediction difference, the second prediction difference and the third prediction difference includes:
[0230] Determine a comprehensive prediction difference of the prediction network for the second image according to the second prediction difference and the third prediction difference;
[0231] Obtaining a weighting coefficient for the comprehensive prediction difference, and weighting the comprehensive prediction difference according to the weighting coefficient to obtain a weighted comprehensive prediction difference;
[0232] The parameters of the prediction network are optimized according to the first prediction difference and the weighted comprehensive prediction difference.
[0233] Optionally, second supervision data is provided for the second image; the classification module 14 performs classification prediction on the pixels of the second image through the auxiliary network to obtain the classification prediction result, including:
[0234] Calling the auxiliary network to generate a regional center feature of the second feature region and a pixel feature of each pixel point in the second image based on the second image;
[0235] A classification prediction result is determined based on the regional center feature, the pixel feature of each pixel in the second image and the second supervisory data.
[0236] Optionally, the classification module 14 calls the auxiliary network to generate the regional center feature of the second feature area and the pixel feature of each pixel in the second image based on the second image, including:
[0237] Calling the auxiliary network to generate pixel features of each pixel in the second image;
[0238] Calling the auxiliary network to predict the mask area of each pixel in the second image, and determining the prediction accuracy index of the mask area of each pixel in the second image;
[0239] The auxiliary network is called to predict the second image based on the mask area of each pixel in the second image to obtain a third prediction result; the third prediction result includes: third prediction pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area;
[0240] A region center feature is generated according to the pixel feature of each pixel in the second image, the prediction accuracy index of the mask region of each pixel in the second image, and the third predicted pixel information.
[0241] Optionally, the classification module 14 generates a method of generating a regional center feature according to the pixel feature of each pixel in the second image, the prediction accuracy index of the mask area of each pixel in the second image, and the third predicted pixel information, including:
[0242] Determine the pixel points whose prediction accuracy index corresponding to the mask area in the second image is greater than the index threshold as evaluation pixel points;
[0243] A region center feature is generated according to the pixel feature of at least one evaluation pixel and the predicted pixel information belonging to the at least one evaluation pixel in the third predicted pixel information.
[0244] Optionally, the target evaluation pixel point of the second feature area includes: among at least one evaluation pixel point, the predicted pixel information in the third predicted pixel information is used to indicate the evaluation pixel point of the pixel point belonging to the second feature area;
[0245] The background evaluation pixel points of the second feature area include: among the at least one evaluation pixel points, the predicted pixel information in the third predicted pixel information is used to indicate the evaluation pixel point of the pixel point that does not belong to the second feature area;
[0246] The classification module 14 generates a region center feature according to the pixel feature of at least one evaluation pixel and the predicted pixel information belonging to at least one evaluation pixel in the third predicted pixel information, including:
[0247] Generate a target center feature of the second feature area according to the pixel feature of the target evaluation pixel and the predicted pixel information of the target evaluation pixel in the third predicted pixel information; the target center feature is used to represent the structural feature of the image of the second feature area;
[0248] generating a background center feature of the second feature area according to the pixel feature of the background evaluation pixel and the predicted pixel information of the background evaluation pixel in the third predicted pixel information; the background center feature is used to represent the structural feature of the background image of the second feature area in the second image;
[0249] The target center feature and the background center feature are determined as the regional center feature.
[0250] Optionally, any pixel point in the second image is represented as a target pixel point; the classification module 14 determines the classification prediction result based on the regional center feature, the pixel feature of each pixel point in the second image and the second supervisory data, including:
[0251] Obtaining a first characteristic distance between a pixel feature of a target pixel and a target center feature, and obtaining a second characteristic distance between a pixel feature of a target pixel and a background center feature;
[0252] If the first feature distance is greater than the second feature distance, and the label information of the target pixel in the second supervisory data is used to indicate that the target pixel does not belong to a pixel of the second feature area, then determining that the target pixel belongs to a pixel of the first classification;
[0253] If the first feature distance is greater than the second feature distance, and the label information of the target pixel in the second supervisory data is used to indicate that the target pixel belongs to a pixel of the second feature area, then determining that the target pixel belongs to a pixel of the second classification;
[0254] If the first feature distance is less than the second feature distance, and the label information of the target pixel in the second supervisory data is used to indicate that the target pixel belongs to a pixel of the second feature area, then determining that the target pixel belongs to a pixel of the first classification;
[0255] If the first feature distance is less than the second feature distance, and the marking information of the target pixel in the second supervisory data is used to indicate that the target pixel does not belong to a pixel of the second feature area, then a pixel of the second classification of the target pixel is determined.
[0256] Optionally, any pixel point in the second image is represented as a target pixel point; the classification module 14 determines the prediction accuracy index of the mask area of each pixel point in the second image in a manner including:
[0257] The network parameters of the auxiliary network are randomly discarded K times respectively to obtain K deformed networks of the auxiliary network; K is a positive integer;
[0258] Calling each deformation network to obtain deformation prediction pixel information for the target pixel based on the mask area of the target pixel respectively; any deformation prediction pixel information includes: information indicating whether the target pixel belongs to a pixel of the second feature area;
[0259] The prediction accuracy index of the mask area of the target pixel point is determined according to the K deformation prediction pixel information obtained by the K deformation networks.
[0260] Optionally, any deformation prediction pixel information includes a first prediction probability that the predicted target pixel belongs to a pixel of the second feature area, and a second prediction probability that the target pixel belongs to a pixel of the background image of the second feature area in the second image;
[0261] The classification module 14 determines the prediction accuracy index of the mask area of the target pixel point according to the K deformation prediction pixel information obtained by the K deformation networks, including:
[0262] Obtaining the standard deviation between the first prediction probabilities in the K deformed prediction pixel information as the target prediction accuracy index for the target pixel point;
[0263] Obtaining the standard deviation between the second prediction probabilities in the K deformed prediction pixel information as the background prediction accuracy index for the target pixel point;
[0264] The target prediction accuracy index and the background prediction accuracy index are determined as the prediction accuracy index of the mask area of the target pixel point.
[0265] Optionally, the classification module 14 determines the pixel points whose prediction accuracy index of the corresponding mask area in the second image is greater than the index threshold as the evaluation pixel points, including:
[0266] If the target prediction accuracy index and the background prediction accuracy index are both greater than the index threshold, the target pixel is determined as the evaluation pixel.
[0267] Optionally, the device 1 is also used for:
[0268] Acquire a target image; the target image includes a target feature area;
[0269] The prediction network after parameter optimization is called to predict the target image to obtain a target prediction result; the target prediction result includes: target prediction pixel information of the pixel points used to determine whether each pixel point of the target image belongs to the target feature area;
[0270] The image of the target feature area is segmented in the target image based on the target predicted pixel information.
[0271] According to one embodiment of the present application, Figure 3 The steps involved in the data processing method shown can be represented by Fig. 9 The data processing device 1 shown in the figure is executed by each module. For example, Figure 3 The step S101 shown in FIG. 1 can be performed by Fig. 9 The acquisition module 11 in is executed, Figure 3 The step S102 shown in FIG. 1 can be performed by Fig. 9 The first calling module 12 is executed; Figure 3 The step S103 shown in FIG. 1 can be performed by Fig. 9 The second calling module 13 is executed, Figure 3 The step S104 shown in FIG. 1 can be performed by Fig. 9 The classification module 14 is used to perform, Figure 3 The step S105 shown in FIG. 1 can be performed by Fig. 9 The optimization module 15 is used to execute.
[0272] The present application can obtain a first image with a first feature area and a second image with a second feature area; call a prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel of the first image belongs to a pixel of the first feature area; call the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area; classify and predict the pixels of the second image through an auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image; according to the first prediction pixel information, the predicted pixel information of the pixels belonging to the first classification in the second prediction pixel information, and the predicted pixel information of the pixels belonging to the second classification in the second prediction pixel information, network parameter optimization is performed, and the network parameter optimization includes any one or a combination of parameter optimization of the prediction network and parameter optimization of the auxiliary network. It can be seen that the device proposed in the present application can classify the pixels in the second image through the auxiliary network, and then the classification results of each pixel in the second image can be used to optimize the parameters of the prediction network or the auxiliary network, which can improve the accuracy of parameter optimization of the prediction network. Subsequently, the prediction network after parameter optimization can also accurately segment the feature areas in the image.
[0273] According to one embodiment of the present application, Fig. 9 The various modules in the data processing device 1 shown can be separately or all combined into one or several units to constitute, or one (some) of the units can be further divided into multiple functionally smaller sub-units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of a module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the data processing device 1 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.
[0274] According to one embodiment of the present application, the program can be executed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in Fig. 9The data processing device 1 shown in the figure is used to implement the data processing method of the embodiment of the present application. The above computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above computing device through the computer-readable recording medium and run therein.
[0275] See also Fig.10 , Fig.10 Schematic diagram of the structure of a computer device provided by this application. Fig.10 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.
[0276] exist Fig.10 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:
[0277] Acquire a first image having a first characteristic region and a second image having a second characteristic region;
[0278] Calling the prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel point of the first image belongs to a pixel point of the first feature area;
[0279] Calling the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel point of the second image belongs to a pixel point of the second feature area;
[0280] Performing classification prediction on the pixels of the second image through the auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image;
[0281] Network parameter optimization is performed based on the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information. The network parameter optimization includes any one or a combination of two of parameter optimization of the prediction network and parameter optimization of the auxiliary network.
[0282] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 The description of the above data processing method in the corresponding embodiment can also be performed Fig. 9 The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.
[0283] In addition, it should be pointed out that the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 The description of the data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.
[0284] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.
[0285] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0286] The present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above Figure 3 The description of the above data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.
[0287] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.
[0288] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0289] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.
[0290] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A data processing method, It is characterized in that The method comprises: Acquire a first image having a first feature area and a second image having a second feature area; the first image is provided with first supervisory data, and the second image is provided with second supervisory data; Calling a prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first prediction pixel information for determining whether each pixel point of the first image belongs to a pixel point of the first feature area; Calling the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel point of the second image belongs to a pixel point of the second feature area; Performing classification prediction on the pixels of the second image through the auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image; Performing network parameter optimization according to the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information, wherein the network parameter optimization includes any one of performing parameter optimization on the prediction network and performing parameter optimization on the auxiliary network, or a combination of both; Wherein, when performing the network parameter optimization, the prediction network is parameter optimized through the first prediction difference between the first predicted pixel information and the first supervisory data, the second prediction difference between the predicted pixel information of the pixel points belonging to the first classification in the second predicted pixel information and the first label information in the second supervisory data, and the third prediction difference between the predicted pixel information of the pixel points belonging to the second classification in the second predicted pixel information and the second label information in the second supervisory data; the first label information includes: information in the second supervisory data used to mark whether the pixel points belonging to the first classification on the second image belong to the second feature area; the second label information includes: information in the second supervisory data used to mark whether the pixel points belonging to the second classification on the second image belong to the second feature area; The second prediction difference and the third prediction difference are calculated in different ways.
2. The method according to claim 1, It is characterized in that The process of performing parameter optimization on the prediction network through the first prediction difference, the second prediction difference and the third prediction difference includes: Determine a comprehensive prediction difference of the prediction network for the second image according to the second prediction difference and the third prediction difference; Obtaining a weighting coefficient for the comprehensive prediction difference, and weighting the comprehensive prediction difference according to the weighting coefficient to obtain a weighted comprehensive prediction difference; Parameters of the prediction network are optimized according to the first prediction difference and the weighted comprehensive prediction difference.
3. The method according to claim 1, It is characterized in that The performing classification prediction on the pixels of the second image by using the auxiliary network to obtain the classification prediction result includes: Calling the auxiliary network to generate a regional center feature of the second feature region and a pixel feature of each pixel point in the second image based on the second image; The classification prediction result is determined based on the region center feature, the pixel feature of each pixel in the second image and the second supervision data.
4. The method according to claim 3, It is characterized in that The calling of the auxiliary network to generate a regional center feature of the second feature region and a pixel feature of each pixel in the second image based on the second image includes: Calling the auxiliary network to generate pixel features of each pixel in the second image; Calling the auxiliary network to predict a mask area of each pixel in the second image, and determining a prediction accuracy index of the mask area of each pixel in the second image; Calling the auxiliary network to predict the second image based on the mask area of each pixel in the second image to obtain a third prediction result; the third prediction result includes: third prediction pixel information for determining whether each pixel of the second image belongs to a pixel of the second feature area; The region center feature is generated according to the pixel feature of each pixel in the second image, the prediction accuracy index of the mask area of each pixel in the second image and the third predicted pixel information.
5. The method according to claim 4, It is characterized in that The generating the region center feature according to the pixel feature of each pixel in the second image, the prediction accuracy index of the mask region of each pixel in the second image and the third predicted pixel information includes: Determine the pixel points whose prediction accuracy index corresponding to the mask area in the second image is greater than the index threshold as evaluation pixel points; The region center feature is generated according to the pixel feature of at least one evaluation pixel point and the predicted pixel information belonging to the at least one evaluation pixel point in the third predicted pixel information.
6. The method according to claim 5, It is characterized in that The target evaluation pixel point of the second feature area includes: among the at least one evaluation pixel point, the predicted pixel information in the third predicted pixel information is used to indicate the evaluation pixel point of the pixel point belonging to the second feature area; The background evaluation pixel points of the second feature area include: among the at least one evaluation pixel point, the predicted pixel information in the third predicted pixel information is used to indicate an evaluation pixel point that does not belong to the second feature area; The generating the region center feature according to the pixel feature of at least one evaluation pixel and the predicted pixel information belonging to the at least one evaluation pixel in the third predicted pixel information comprises: generating a target center feature of the second feature area according to the pixel feature of the target evaluation pixel and the predicted pixel information of the target evaluation pixel in the third predicted pixel information; the target center feature is used to represent the structural feature of the image of the second feature area; generating a background center feature of the second feature area according to the pixel feature of the background evaluation pixel and the predicted pixel information of the background evaluation pixel in the third predicted pixel information; the background center feature is used to represent the structural feature of the background image of the second feature area in the second image; The target center feature and the background center feature are determined as the region center feature.
7. The method according to claim 6, It is characterized in that Any pixel point in the second image is represented as a target pixel point; and determining the classification prediction result based on the region center feature, the pixel features of each pixel point in the second image and the second supervisory data includes: Acquire a first characteristic distance between a pixel characteristic of the target pixel point and a target center characteristic, and acquire a second characteristic distance between a pixel characteristic of the target pixel point and a background center characteristic; If the first feature distance is greater than the second feature distance, and the label information of the target pixel in the second supervisory data is used to indicate that the target pixel does not belong to a pixel of the second feature area, then determining that the target pixel belongs to a pixel of the first classification; If the first feature distance is greater than the second feature distance, and the label information of the target pixel in the second supervisory data is used to indicate that the target pixel belongs to a pixel of the second feature area, then determining that the target pixel belongs to a pixel of the second classification; If the first feature distance is less than the second feature distance, and the label information of the target pixel in the second supervisory data is used to indicate that the target pixel belongs to a pixel of the second feature area, then determining that the target pixel belongs to a pixel of the first classification; If the first feature distance is smaller than the second feature distance, and the marking information of the target pixel in the second supervisory data is used to indicate that the target pixel does not belong to the pixel of the second feature area, the pixel of the second classification of the target pixel is determined.
8. The method according to claim 4, It is characterized in that Any pixel point in the second image is represented as a target pixel point; and determining the prediction accuracy index of the mask area of each pixel point in the second image includes: The network parameters of the auxiliary network are randomly discarded K times respectively to obtain K deformed networks of the auxiliary network; K is a positive integer; Calling each deformation network to obtain deformation prediction pixel information for the target pixel based on the mask area of the target pixel respectively; any deformation prediction pixel information includes: information indicating whether the target pixel belongs to a pixel of the second feature area; Determine a prediction accuracy index of the mask area of the target pixel point according to the K deformation prediction pixel information obtained by the K deformation networks.
9. The method according to claim 8, It is characterized in that Any deformed predicted pixel information includes a first predicted probability that the target pixel belongs to a pixel of the second feature area, and a second predicted probability that the target pixel belongs to a pixel of the background image of the second feature area in the second image; The determining the prediction accuracy index of the mask area of the target pixel point according to the K deformation prediction pixel information obtained by the K deformation networks includes: Obtaining a standard deviation between first prediction probabilities in the K pieces of deformed prediction pixel information as a target prediction accuracy index for the target pixel point; Obtaining a standard deviation between the second prediction probabilities in the K deformed prediction pixel information as a background prediction accuracy index for the target pixel point; The target prediction accuracy index and the background prediction accuracy index are determined as the prediction accuracy index of the mask area of the target pixel point.
10. The method according to claim 9, It is characterized in that The step of determining the pixel points whose prediction accuracy index corresponding to the mask area in the second image is greater than the index threshold as evaluation pixel points includes: If the target prediction accuracy index and the background prediction accuracy index are both greater than the index threshold, the target pixel point is determined as an evaluation pixel point.
11. The method according to claim 1, It is characterized in that The method further comprises: Acquire a target image; the target image includes the target feature area; Calling the prediction network after parameter optimization to predict the target image to obtain a target prediction result; the target prediction result includes: target prediction pixel information for determining whether each pixel point of the target image belongs to a pixel point of the target feature area; The image of the target feature area is segmented in the target image based on the target predicted pixel information.
12. A data processing device, It is characterized in that The device comprises: An acquisition module, used to acquire a first image having a first characteristic region and a second image having a second characteristic region; the first image is provided with first supervision data, and the second image is provided with second supervision data; A first calling module, used for calling a prediction network to predict the first image to obtain a first prediction result; the first prediction result includes: first predicted pixel information for determining whether each pixel of the first image belongs to a pixel of the first feature area; A second calling module, used for calling the prediction network to predict the second image to obtain a second prediction result; the second prediction result includes: second prediction pixel information for determining whether each pixel point of the second image belongs to a pixel point of the second feature area; A classification module, used to perform classification prediction on the pixels of the second image through an auxiliary network to obtain a classification prediction result; the classification prediction result is used to indicate the pixels belonging to the first classification and the pixels belonging to the second classification on the second image; an optimization module, configured to optimize network parameters according to the first predicted pixel information, the predicted pixel information of the pixel points belonging to the first category in the second predicted pixel information, and the predicted pixel information of the pixel points belonging to the second category in the second predicted pixel information, wherein the network parameter optimization includes any one or a combination of optimizing parameters of the prediction network and optimizing parameters of the auxiliary network; Wherein, when performing the network parameter optimization, the prediction network is parameter optimized through the first prediction difference between the first predicted pixel information and the first supervisory data, the second prediction difference between the predicted pixel information of the pixel points belonging to the first classification in the second predicted pixel information and the first label information in the second supervisory data, and the third prediction difference between the predicted pixel information of the pixel points belonging to the second classification in the second predicted pixel information and the second label information in the second supervisory data; the first label information includes: information in the second supervisory data used to mark whether the pixel points belonging to the first classification on the second image belong to the second feature area; the second label information includes: information in the second supervisory data used to mark whether the pixel points belonging to the second classification on the second image belong to the second feature area; The second prediction difference and the third prediction difference are calculated in different ways.
13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.
14. A computer device, It is characterized in that The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 11.
15. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Self-distillation training method and scalable dynamic prediction method of convolutional neural network
CN110472730A
Three-dimensional human body posture estimation method based on feature fusion and sample enhancement
CN111428586A
Neural network training method and device and image segmentation method
CN113822428A
Image segmentation method and device, equipment and storage medium
CN113822851A
Classification model training method and device, image classification method and device, equipment and medium
CN113902010A
